On 30 questions that need a data catalog, Claude with OpenData answered 25. Without it, 6.

Each question ran twice in the Claude Desktop app, Sonnet 5, September 2026, one fresh chat each time. One run had only web search and a sandbox, the scratch computer Claude can run code on. The other also had the OpenData connector: an MCP server, which is how a chat app plugs into an outside tool, so Claude could query our data catalog directly. Those two setups are the arms.

The questions were written to be hard to answer without the underlying dataset: a web search should not land on a news article that already states the answer, though a few did. Answering them meant loading small, large, and messy datasets, some needing assembly across several years or cleaning before use.

With OpenData
Web search only

of 30

With OpenDataWeb search only

Every question, one cell each

Easiest first, one row per setup.

Hover a cell for its detail; click to jump to the question.

When did low-income students become a majority in Illinois schools?

The answer needs one field pulled out of 29 annual files, each with its own layout. Both columns are the same question on the same day, and every chat in the run was cut off at ten minutes.

Web search only

0m
5m
10m
15m
20m
25m
30m
  1. 0 sQuestion sent
  2. 4 sSearched for the ISBE report card archive
  3. 24 sScraped the download links off the archive page
  4. 49 sStarted downloading the 29 annual files, with retries
  5. 10m 11s28 of 29 files down; one year still missing
  6. 10m 22sDownloaded ISBE's per-year record layout specs
  7. 16m 37sLocated the low-income field position in each layout
  8. 22m 22sVerified field positions against known values
  9. 24m 08sFound and fetched the missing 2001 file
  10. 25m 00sStopped; the extraction was never run
  11. 25m 09sRan out of room with the extraction still to run, past the 10-minute cap.

With OpenData

0m
5m
10m
15m
20m
25m
30m
  1. 0 sQuestion sent
  2. 3 sSearched the catalog
  3. 9 sRead the dataset's schema and coverage
  4. 13 sCounted majority-low-income schools per year
  5. 19 sA column name missed; corrected on the next call
  6. 24 sChecked which years carry the figure
  7. 31 sInspected raw enrolment values across eras
  8. 44 sFound the marker that separates schools from aggregates
  9. 49 sRan the final per-year count
  10. 56 sCharted the 29-year series
  11. 62 sAnswered
  12. 63 sFirst crossing in 2015, a majority in four of the 29 years.

On an earlier pass of the same 30 questions, the connector was 3.3x faster on the 14 both setups got right.

Each line runs between the two arms' times on one question.

Web search onlyWith OpenData

Across those 14, the connector's median time, the middle one with half its chats faster and half slower, was 3.3x quicker. Across all 30 in that pass the medians were 73 s and 29 s.

When web search did find the dataset, it usually got to an answer by downloading the raw file into the sandbox and parsing it as text. It did that in 17 of the 30 chats, over 392 commands, and it worked more often than we expected: 4 of its 6 right answers came that way. It is still slow and heavy on tool calls, and reading numbers out of a text file will always go wrong more often than a query against a table whose columns are already known.

Where the time went

Web search only
63 s median · 5 past the cap
With OpenData
35 s median · 0 past the cap
Web search onlyWith OpenData

Total chat time

Web search only, Total chat time: 180 min
With OpenData, Total chat time: 29 min

Tool calls

Web search only, Tool calls: 541
With OpenData, Tool calls: 228

Sandbox commands

Web search only, Sandbox commands: 392
With OpenData, Sandbox commands: 0

Web searches

Web search only, Web searches: 94
With OpenData, Web searches: 3

Chats past the 10-minute cap

Web search only, Chats past the 10-minute cap: 5
With OpenData, Chats past the 10-minute cap: 0

No answer

Web search only, No answer: 16
With OpenData, No answer: 0

Ended by asking the user to fetch the data

Web search only, Ended by asking the user to fetch the data: 3
With OpenData, Ended by asking the user to fetch the data: 0

Where the misses came from

On its 5 misses the connector found the right rows and then counted the wrong set of them. The web arm mostly never got at the data in the first place.

Web search only

declined (9)
Found the data and stopped, asking what to do next instead of computing an answer. p1-003, p1-004, p1-009, p1-011, p1-012, p1-013, p1-019, p1-021, p1-023
wrong slice (5)
It computed an answer, but from a different release of the data or a different filter. p1-002, p1-014, p1-017, p1-018, p1-022
blocked (4)
The site that publishes the data refused the download, or was down that day. p1-006, p1-007, p1-008, p1-010
out of turn (3)
The chat ran out of tool calls or out of time with the work unfinished. p1-015, p1-016, p1-020
discovery (1)
Said no such dataset exists when one does. p1-028
fabricated (1)
Quoted a figure off a web page for something no dataset measures. p1-029
wrong universe (1)
The right rows, counted over the wrong set. The one miss both arms share. p1-026

With OpenData

wrong universe (5)
It queried the right rows, then totalled a set of them that is not the one the question asks about. p1-007, p1-009, p1-013, p1-021, p1-026

The same gap shows up on a cheap model outside Claude Desktop, where we paid per question.

Everything above ran inside Claude Desktop, where a person types a question and Claude decides what to do next. This run was a program instead. It called DeepSeek V4 Flash, a cheap model, through the OpenRouter API, and sent it the same 30 questions three times over: once with no tools, once with web search and a Python sandbox, once with the OpenData connector.

Nothing changed between the three except which tools the model could call.

No tools
$0.002
per correct answer, on 5 of 30
Web search and python
$0.45
per correct answer, on 8 of 30
OpenData MCP
$0.0042
per correct answer, on 21 of 30

One run per arm, one cheap model. The setup and its caveats are on the methodology page.

All 30 questions, with the gold answer and both replies

Ordered by difficulty. Open a row for the question as asked, both arms' replies in full, and the gold answer: the answer we worked out ourselves before the run, with the query that produced it.

Run the query behind this answer

It runs in OpenData's SQL sandbox, against the same tables the connector queried.

SELECT year, COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE)>=50) maj, COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE) IS NOT NULL) reporting, ROUND(100.0*COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE)>=50)/COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE) IS NOT NULL),2) pct FROM "isbe/report-card:all" WHERE COALESCE(type,type_1)='School' GROUP BY 1 HAVING ROUND(100.0*COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE)>=50)/COUNT(*) FILTER (WHERE TRY_CAST(student_enrollment_low_income AS DOUBLE) IS NOT NULL),2)>50 ORDER BY year
Open in the sandboxAttach the OpenData connectorHow this was run and graded

What this does not show

  • The questions were written to be hard to answer without the underlying dataset: a web search should not land on a news article that already states the answer, though a few did. Answering them meant loading small, large, and messy datasets, some needing assembly across several years or cleaning before use.
  • One run per arm. Thirty questions, one chat each, no repeats, so we cannot say how much of this is chance. Whether the web arm attempts a question or declines it varies visibly, so the no-answer count in particular would come out differently on a second pass.
  • Claude's memory was on for both arms, so a chat could in principle draw on an earlier one. Across all 32 transcripts, one web chat read memory, and it then declined the question and pointed at OpenData. No connector chat read memory.
  • The connector arm's question had one extra line in front of it telling Claude to use the connector. The web arm's did not.
  • The two arms ran one after the other rather than alternating, so the connector arm ran on an account that had already seen the questions once.
  • The ten-minute cap was decided partway through and then applied to every chat. No grade changed because of it, though some web chats that ended with no answer might have produced an answer given longer. A wrong answer is a different failure, not a better result.
  • Our catalog and the live web are not always on the same release of a dataset. Two web misses are correct answers computed on a different release, and two questions rest on sites that were unreachable that day, so part of that arm's result is the state of the web on 19 September.
  • Four of the 30 questions rest on one dataset, whose enrolment column we repaired before the run. The gold answers were computed after that repair.
  • The times include the app's own thinking and rendering, not just the data work. The model that graded each row comes from the same family as the model being tested, and the person who graded by hand is the person who wrote the questions.

The full protocol, grading and caveats

Search datasets

Search for datasets or run quick actions