No matter what I do, I run into bad requests with claude, day in, day out.
Its making up arguments, misunderstands required types, and misses fields in the requests. And when it catches its issues, the then inititated web search usually ends fuzzy scraped information, that yields even more issues.
Context7 helps. Its better than starting only with the LLM's vague (mis)understanding from pretraining. But it only does semantic search. And often times, semantic search is not precise enough for hyper-precision needed for API requests: CC runs into the same misunderstanding issues as above. And burns tons of tokens in the process.
2. What about Determistic Search in OpenAPI Specs?
In my opinion agents need 1) understanding the damn thing holistically, and 2) ability to do some type of agentic search within the docs.
Thankfully, we do have magnificiently standardized formats for API schemas, most notably OpenAPI/Swagger. Why is no one (to my best knowledge) making use of it?
As I need to work a lot with APIs, I started to build something myself few months ago. In the end its a simple python script that splits the JSON/YAML/RAML/etc files into a) a holistic overview ("manifest"), and b) indexed chunks (by endpoints, tags, and schemas) md files. Agents can access via MCP. It takes a) convert local files, or b) community-converted files, and give the agent the capability to do agentic search on the specs. You can check it out out here, and hook up the MCP server: https://github.com/mohidbt/api-ingest
3. Should we benchmark this? // Feel free to contribute!
WDYT? I am thinking about quantifively corroborating my assumptions, by doing some type of evals. And yes, this by endpoint indexing approach also has many limitations. I.e. when the individual chunks are themselves way too big to load fully into context.
Geniunely curious about all your thoughts
PS: Yes, for many - especially AI-tech - companies, we already have agent optimized API doc formats, like llms.txt in the docs, or skills built for using the APIs; and thats wonderful! But whats with, i.e. Semantic Scholar Graph APIs? What do you do if core CC & Context7 fail? Check out this example: https://github.com/mohidbt/api-ingest/tree/main#opus-47-exam...