Text-to-ES|QL benchmark: results and analysis

The 2026 picture

Single-shot, zero-shot text-to-ES|QL on BIRD Mini-Dev (500 questions), run against Elasticsearch 9.5. The headline: GPT-5.5 writes correct ES|QL 59% of the time under lenient scoring with nothing but a schema, comfortably above what GPT-4 scored writing SQL on the same benchmark (~48%). For the best model the syntax barrier is effectively gone; what is left is semantic correctness.

Lenient EX. Strict BIRD EX for the same cells: gpt-5.5 45.6 / 33.2 / 32.8, opus 32.6 / 34.8 / 34.0, sonnet 16.2 / 21.2 / 21.4, mini 13.0 / 18.6 / 19.0. Flip the switch above.

Calibration reference: single-call zero-shot text-to-SQL (GPT-4) is ~48% on the same questions. BIRD's public leaderboard tops out higher, but those entries are engineered pipelines with schema linking and self-correction, not one call with one prompt.

Two findings about the skill doc

Why the weaker numbers are low

The models are not bad at the logic of these questions. They trip on a handful of ES|QL-specific mechanics and on the strictness of execution-accuracy scoring. The single biggest cause is one join rule.

Failure taxonomy (no-skill, claude-opus-4-8)

Field-grounding (won't execute) vs logic-but-valid is almost exactly 50/50. The grounding half throws precise errors, so an execution-feedback loop would fix most of it. The other half runs silently wrong, so a loop alone gives no signal.

The skill doc buys validity, not correctness

Adding Elastic's ES|QL reference to the prompt slashes syntax errors, but those queries then fail one stage later on schema grounding or semantics. The failure mix shifts from "doesn't run" to "runs but wrong."

"Runs-but-wrong" share of all failures climbs from 52%→72% (Opus) and 52%→76% (Sonnet). The skill all but eliminates field-grounding errors (Opus 143→4, Sonnet 21→9), and for Sonnet it halves the syntax errors its SQL habits produce (single = instead of ==, CASE WHEN, single quotes, COUNT(DISTINCT)). Opus already had the syntax mostly right, so for it the trade shows up as a rise in syntax errors instead: those are the ambiguous-reference failures the join edit introduced.

wrong_result: right idea, wrong tuple

The hardest bucket: the query executes cleanly but returns the wrong rows. Main reasons:

• Result-shape / projection mismatch (29.5%, measured, and already priced in). BIRD compares the full row tuple, so an otherwise-correct query that KEEPs the wrong columns (or adds the sort key) scores zero. Lenient scoring rescues 830 of the 2,814 clean-running strict misses, 88 of them on column split alone. That is why this cause contributes 0% to the bucket below: the lenient metric has already forgiven every one of them.
• Fan-out over one-to-many joins (3–7%, measured). The LOOKUP JOIN emits a row per match, inflating counts vs gold's DISTINCT. A third of this bucket joins and then aggregates, but that only bounds the exposure: 6.5% joins and aggregates where the gold uses DISTINCT, and 2.8% returns a single number larger than gold's.
• LOOKUP JOIN does not fan out. "How many comments on the top post?" After the join, COUNT() returns 1, because LOOKUP JOIN enriches the single left row rather than expanding it.
• Value / column grounding (10–15%, measured). rank vs position, the exact stored code, which of two near-identical tables holds the value, knowledge the model can't get from the schema alone. 9.9% of the bucket returns nothing while filtering on a string literal, the signature of a guessed value; 14.6% returns nothing for any reason.
• Evidence formula mis-applied (7%, measured). The right formula over the wrong population/denominator. 27.3% of the bucket is questions whose gold computes a ratio or a percentage, which bounds the exposure; in 7.0% the prediction is a single number within 50% of gold's, the shape of a correct formula fed the wrong population.
• Dates stored as keyword strings (measured). TO_DATETIME/DATE_PARSE misparse, or "now" arithmetic drifts. 24.9% of this bucket is questions whose gold query does date arithmetic, yet only 5.3% of the failing queries call an ES|QL date function at all.

What would move the number

• An execution-feedback loop (generate → run → read error → retry) directly fixes the ~50% grounding half: RENAME the join key, replace invented functions, qualify ambiguous columns. This is exactly what Elastic's Agent Builder does out of the box.
• Sample rows / a value-lookup tool addresses the value-grounding third (exact codes, rank vs position) that no amount of prompting can solve blind.
• ES|QL-aware guidance on the two things the language does differently from SQL: LOOKUP JOIN key naming, and that it enriches rather than fans out.

Scope: the headline table, the per-category counts, the skill findings and the GPT-5.5 control are all re-derived from this run (Elasticsearch 9.5, 6,000 scored queries, plus a 500-query control for GPT-5.5 on the unmodified skill), as is every wrong_result share marked "measured", computed over the runs-but-wrong bucket (executed cleanly, wrong data under lenient scoring, n=1,984). Where a share bounds the exposure rather than establishing the cause, both figures are given. The failure taxonomy table above is the one section still carried over from a manual read of the earlier 9.1 run; its underlying buckets barely moved between runs (wrong_result 2,414 to 2,525, empty_result 283 to 289), so the structure holds, but treat its counts as indicative.