When using LLMs in production, our prompts are often long and complex, involving multi-shot reasoning, ReAct, CoT and other prompting techniques. It’s really painful to experiment and evaluate these prompts, we either use the web interfaces like chatgpt (Really hard to edit long prompts, needs a lot of copy pasting), or write a python script to test each prompt (Too many scripts in the end).

So I developed a VSCode extension that makes it really easy to experiment with prompts. I also designed a file format (or programming language) called Prompt File that encapsules the common prompt operations like user inputs, import files, web browsing, multi role, etc. When executing prompt files, the extension will deal with all the tedious manual works for us.

It also supports prompt chaining, i.e. including the result of one prompt run in another prompt, so it’s actually possible to implement a complete AI Agent workflow purely with Prompt Files. There’s some examples in the git repo.

I also plan to add testing syntax like Rust’s [#cfg(tests)], so it’s possible to manage the whole lifecycle of prompt development using this file format alone.

The whole project is written over the weekend so many things are still missing. But I would love to hear your thoughts! Thanks!