Show HN: LLM Router – Open-source prompt router for multi-LLM deployments

2 points

a year ago

We just open-sourced LLM Router, a framework that lets you route user prompts to the most suitable large language model (LLM) automatically optimizing for speed, cost, and accuracy. If you have multiple LLMs (open or proprietary) and want to use the right one for each task, LLM Router acts as a drop-in OpenAI API-compatible proxy. It classifies each prompt (e.g., code generation, QA, rewrite) and sends it to the best backend, all configurable via YAML.

Built for production: Rust backend, NVIDIA Triton integration, metrics for monitoring.

Flexible policies: Use built-in classifiers or plug in your own PyTorch models.

Easy integration: No major code changes needed, point your client at the router.

Example: Route complex coding questions to a powerful model, and simple rewrites to a smaller, cheaper one.

Repo: github.com/NVIDIA-AI-Blueprints/llm-router

Would love feedback, ideas, and to hear how others are handling multi-LLM workflows!