We built Index - new SOTA Open Source browser agent.

It reached 92% on WebVoyager with Claude 3.7 (extended thinking). o1 was used as a judge, also we manually double checked the judge.

At the core is same old idea - run simple JS script in the browser to identify interactable elements -> draw bounding boxes around them on a screenshot of a browser window -> feed it to the LLM.

What made Index so good:

1. We essentially created browser agent observability. We patched Playwright to record the entire browser session while the agent operates, simultaneously tracing all agent steps and LLM calls. Then we synchronized everything in the UI, creating an unparalleled debugging experience. This allowed us to pinpoint exactly where the agent fails by seeing what it "sees" in session replay alongside execution traces.

2. Our detection script is simple but extremely good. It's carefully crafted via trial and error. We also employed CV and OCR.

3. Agent is very simple, literally just a while loop. All power comes from carefully crafted prompt and ton of eval runs.

Index is a simple python package. It also comes with a beautiful CLI.

pip install lmnr-index

playwright install chromium

index run

We've recently added o4-mini, Gemini 2.5 Pro and Flash. Pro is extremely good and fast. Give it a try via CLI.

You can also use index via serverless API. (https://docs.lmnr.ai/index-agent/api/getting-started)

Or via chat UI - https://lmnr.ai/chat.

To learn more about browser agent observability and evals check out open-source repo (https://github.com/lmnr-ai/lmnr) and our docs (https://docs.lmnr.ai/tracing/browser-agent-observability).