Show HN: Improve LLM Performance by Maximizing Iterative Development

104 points

2 years ago

I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process that let’s you quickly test and evaluate these different combinations.

I have had the chance to talk with many companies working on AI products. The biggest mistake I see is a lack of standard process that allows them to rapidly iterate towards their performance goal. Using my learnings, I’m working on an Open Source Framework that structures your application development for rapid iteration so you can easily test different combination of your LLM application components and quickly iterate towards your accuracy goals.

You can checkout the project at https://github.com/palico-ai/palico-ai

You can locally setup a complete LLM Chat App with us with a single command. Stars are always appreciated!

Would love any feedback or your thoughts around LLM Development.

22 comments