How this benchmark was run

Claude Sonnet 5 at Medium effort in the claude.ai web app, on one account, 19 September 2026. Web search on, memory on at the account default, no project, extended thinking at the account default. Every question went into a fresh chat as a single message, with no follow-ups, and the first reply is what was graded. For the web arm the OpenData connector was switched off at the account level and the question was pasted in as written. For the connector arm it was switched on and the question carried one extra line in front of it, "Use the opendata MCP". Each chat was run by a separate agent driving Claude in Chrome, which recorded the chat's address, its timestamps, the line naming which tools the reply used, and any banners. Pulling the transcript afterwards supplied the app's own timestamps and every tool call with its input.

1. One question per chat, a new chat every time, no follow-ups. If the app asks a clarifying question instead of answering, that is where the chat ends and the row counts as no answer.
2. The question is pasted in exactly as written, with nothing before or after it. No "answer with a single number" at the end.
3. The same model and settings for every chat in a pass. The model and settings are read back off each conversation and recorded on its row.
4. The connector is switched on or off for the whole account rather than per chat. A connector that is switched on and never called counts as a miss.
5. Alternate between the two arms rather than running all of one and then all of the other. This run did not: each arm ran as a block, which is in the caveats.
6. No editing, retrying or regenerating a message. A chat that has to be redone starts fresh, and the abandoned chat is reported.
7. Nothing else heavy running on the OpenData account during a pass, so no arm is measured while it is being rate-limited.
8. Ten minutes per chat on the app's own clock. A reply that arrives after that counts as a miss whatever it says.
9. For the web arm, switch OpenData off for the whole account: the toggle in the message box leaves it callable. Every web chat is checked for an OpenData tool name in its trace, and a chat that shows one is thrown out.
10. Memory off for both arms. It was on for every run so far, and one web reply offered to run the query "if you connect OpenData", so carry-over between chats is something we have actually seen rather than a worry in principle.
11. The agents check the page about every five minutes rather than continuously, in under a dozen browser calls between sending a question and capturing the reply.

Every question has a gold answer, the SQL query that produced it, and a note naming the wrong answers someone could reasonably land on. Each gold answer was computed twice against the live OpenData API before the run. Every row was then graded twice: once by a model handed the gold answer and the note, and once by hand. The hand grade is the one of record and is what every figure here counts. The model grader agreed on 58 of the 60 rows, and both of the hand corrections narrowed the gap between the arms. Where the two disagree, the row shows both.

There is no cost column because there is no honest one to publish. Both arms ran on a paid subscription, so each extra chat cost nothing at the margin and any dollar figure would be made up. Token counts cannot stand in for one either. They are estimated from the transcript at four characters per token, and the app records web search and fetch results as zero characters, so the arm that searched the most comes out looking the cheapest. What can be measured honestly is the time, the number of tool calls and sandbox commands, how often a chat ran past the cap, and how often the answer came back as a job for the reader to finish, so that is what the page reports instead.

• The questions were written to be hard to answer without the underlying dataset: a web search should not land on a news article that already states the answer, though a few did. Answering them meant loading small, large, and messy datasets, some needing assembly across several years or cleaning before use.
• One run per arm. Thirty questions, one chat each, no repeats, so we cannot say how much of this is chance. Whether the web arm attempts a question or declines it varies visibly, so the no-answer count in particular would come out differently on a second pass.
• Claude's memory was on for both arms, so a chat could in principle draw on an earlier one. Across all 32 transcripts, one web chat read memory, and it then declined the question and pointed at OpenData. No connector chat read memory.
• The connector arm's question had one extra line in front of it telling Claude to use the connector. The web arm's did not.
• The two arms ran one after the other rather than alternating, so the connector arm ran on an account that had already seen the questions once.
• The ten-minute cap was decided partway through and then applied to every chat. No grade changed because of it, though some web chats that ended with no answer might have produced an answer given longer. A wrong answer is a different failure, not a better result.
• Our catalog and the live web are not always on the same release of a dataset. Two web misses are correct answers computed on a different release, and two questions rest on sites that were unreachable that day, so part of that arm's result is the state of the web on 19 September.
• Four of the 30 questions rest on one dataset, whose enrolment column we repaired before the run. The gold answers were computed after that repair.
• The times include the app's own thinking and rendering, not just the data work. The model that graded each row comes from the same family as the model being tested, and the person who graded by hand is the person who wrote the questions.

An earlier run, in September 2026, outside Claude Desktop. A program called one cheap model, DeepSeek V4 Flash, through the OpenRouter API and sent it the same questions three times over: once with no tools, once with web search and a Python sandbox, once with the OpenData connector. Nothing changed between the three except which tools the model could call.

• No tools
• Web search and python: An Exa search plugin that injects snippets, rather than a search tool the model calls; A bare python sandbox with no requests, pandas or pypdf
• OpenData MCP: The production OpenData MCP server

The table below is a projection rather than a measurement. It re-reads the same transcripts on the assumption that the faulty questions and gold answers had been fixed first. It becomes a real number only if those are fixed and all three arms are run again.

• One run per arm, one cheap model (DeepSeek V4 Flash through OpenRouter) at temperature 0, the setting that makes a model's output as repeatable as it can be, on 30 questions written by us and graded once by a single model.
• This web arm is a plain stand-in for the competition rather than a polished product: a plugin that pastes search snippets into the prompt instead of a search tool the model calls, and a bare Python sandbox with no requests, pandas or pypdf. Most of its 12 chats that ran out of turns were the model writing its own download-and-parse code. Nothing differs between the arms except the tools, which is what makes the comparison fair, but this is not a finished app with built-in tools and DeepSeek is not a frontier model.
• Each task was capped at 30 turns and at 25 cents. Almost every no-answer in the web arm is a task that ran out of turns.
• These were the first questions we wrote, and some of them are ambiguous. Seven of the nine connector misses come down to the question or its gold answer rather than to the model, so this arm's score understates what it actually found.
• This run was only ever meant to check that attaching the connector does not make a model worse. It says that a cheap model with the connector is not worse than the same cheap model with generic web tools, and that the harness, the questions and the grading work. It says nothing about how the frontier models compare.
• The table above is a projection read off the same transcripts, not a measurement. It becomes a real number only if the faulty questions and gold answers are fixed and all three arms are run again.

Every chat in this run has a stored transcript with each tool call and its input, alongside the log the automation wrote while the chat ran. Those files are not published, because some of the conversations quote the account back to itself.

To see a transcript for a particular row, get in touch.