We’re excited to share PySpur, an open-source tool that provides a graph-based interface for building, debugging, and evaluating LLM workflows.

Why we built this:

Before this, we built several LLM-powered applications that collectively served thousands of users. The biggest challenge we faced was ensuring reliability: making sure the workflows were robust enough to handle edge cases and deliver consistent results.

In practice, achieving this reliability meant repeatedly:

1. Breaking down complex goals into simpler steps: Composing prompts, tool calls, parsing steps, and branching logic. 2. Debugging failures: Identifying which part of the workflow broke and why. 3. Measuring performance: Assessing changes against real metrics to confirm actual improvement.

We tried some existing observability tools or agent frameworks and they fell short on at least one of these three dimensions. We wanted something that allowed us to iterate quickly and stay focused on improvement rather than wrestling with multiple disconnected tools or code scripts.

We eventually arrived at three principles upon which we built PySpur :

1. Graph-based interface: We can lay out an LLM workflow as a node graph. A node can be an LLM call, a function call, a parsing step, or any logic component. The visual structure provides an instant overview, making complex workflows more intuitive. 2. Integrated debugging: When something fails, we can pinpoint the problematic node, tweak it, and re-run it on some test cases right in the UI. 3. Evaluate at the node level: We can assess how node changes affect performance downstream.

We hope it's useful for other LLM developers out there, enjoy!