Show HN: PolyCouncil: Multi-Model Deliberation Engine for LMStudio (Open Source)

1 point

7 months ago

I built PolyCouncil to solve a problem I kept running into when testing local LLMs, I never knew if a single model's answer was actually good, or just confidently wrong. Instead of querying one model, PolyCouncil runs multiple local LLMs in parallel through LM Studio, has them evaluate each other's responses using a shared rubric, then conducts weighted voting to reach consensus on the best answer. Think of it like a panel of experts that score each other's work, then vote democratically on the winner. Key features:

Parallel execution across multiple models Rubric-based cross-evaluation (accuracy, clarity, completeness, etc.) Weighted voting system based on peer scores Custom personas (fact-checker, engineer, risk assessor, etc.) Built-in leaderboard to track model performance Single-voter "judge" mode for controlled experiments Clean Python/PySide6 GUI with pre-built Windows executable

Why it's useful:

Compare models systematically without manual evaluation Get more reliable answers through ensemble methods Explore emergent behavior from model deliberation Test how different personas affect model responses

It's free for personal/research use (Polyform Noncommercial). Built for anyone tinkering with local models who wants to see how collective intelligence emerges from LLM orchestration. Would love feedback from folks experimenting with local models! GitHub: https://github.com/TrentPierce/PolyCouncil

1 comment