Show HN: Realistic Synthetic Conversations for Testing LLMs

2 points

a year ago

Testing multi-turn conversational AI is tough, especially when you lack large volumes of real user data. Existing synthetic data tools often generate conversations that lack diversity and are not statistically representative, leading models to overfit synthetic patterns.

To help with this problem, I'm open-sourcing a synthetic conversation generation library. This library generates more realistic multi-conversations than other synthetic data libraries by using the following techniques:

  1. Decoupling Persona & Conversation Generation: This library first create diverse user personas, ensuring each new persona differs from the last. This builds a wide range of user types before generating conversations, tackling bias and improving coverage.

  2. Modeling Realistic Stopping Points: Instead of arbitrary turn limits, the library dynamically assesses if the user's goal is met or if they're frustrated, ending conversations naturally like real users would.

You can generate user personas tailored to your AI's specs and then simulate user messages using those personas. The library calls your AI endpoint (via a configurable HTTP definition) for responses during the simulation.

I built this because I needed a better way to test conversational agents for my clients, and found existing tools lacking in generating high-fidelity dialogues. Would love to hear your feedback and any suggestions!

1 comment