# EthIQ

> EthIQ is a benchmark of how well AI models know the Ethereum protocol (consensus specs, execution specs, EIPs, fork mechanics, constants). The website is a JavaScript app, so fetching a chart page returns an empty shell. The same data is available as plain JSON from the read-only API below, and the leaderboard pages (/performance, /rankings, /models) also carry the full leaderboard as an HTML table in the page source.

## Answering "where does model X rank?"

Fetch the leaderboard for the dataset and prompt you care about. Rows are sorted by pass rate (best first), so a model's rank is its position in `model_scores`. Each model appears once per reasoning-effort variant, e.g. `kimi-k3-high` and `kimi-k3-max` are separate rows (`display_name` is `kimi-k3` for both).

- https://ethiq.ethpandaops.io/api/overview?dataset=fusaka&prompt=bare&refusals=fallback

Always pass `refusals=fallback` to match the website. The website applies it by default, but the API does not, and ranks differ without it (some models refuse questions; with `fallback` a refused answer is scored using the model's refusal fallback instead of as a failure). Omit the param to get raw results where refusals count as failures.

With jq, rank of every kimi-k3 row:

```
curl -s 'https://ethiq.ethpandaops.io/api/overview?dataset=fusaka&prompt=bare&refusals=fallback' \
  | jq -r '.model_scores | to_entries[] | select(.value.model_id | test("kimi-k3")) | "\(.key+1) \(.value.model_id) \(.value.pass_rate)"'
```

Each page URL on the site maps to an API call with the same query string, and every page lists its API call in a `<link rel="alternate" type="application/json">` tag.

## API

All endpoints are `GET`, return JSON, need no auth, and take the filter params below. Base URL: https://ethiq.ethpandaops.io

- [/api/filters](https://ethiq.ethpandaops.io/api/filters): every valid filter value (datasets, prompts, labs, models, categories, forks, tags). Start here to find the exact `dataset`, `prompt` and model ids.
- [/api/overview](https://ethiq.ethpandaops.io/api/overview?dataset=fusaka&prompt=bare&refusals=fallback): leaderboard. Per model: `pass_rate`, 95% Wilson interval (`ci_lower`, `ci_upper`), `total_questions`, `cost_avg`, `cost_total`, `latency_p50`, per-category pass rates. Plus per-category totals. Page: /performance, /rankings, /models.
- [/api/heatmap](https://ethiq.ethpandaops.io/api/heatmap?dataset=fusaka&prompt=bare&refusals=fallback): pass rate per model and group (`group_by` selects category, fork, question, ...). Page: /capabilities.
- [/api/efficiency](https://ethiq.ethpandaops.io/api/efficiency?dataset=fusaka&prompt=bare&refusals=fallback): pass rate against cost and latency. Page: /efficiency.
- [/api/costs](https://ethiq.ethpandaops.io/api/costs?dataset=fusaka&prompt=bare&refusals=fallback): benchmark spend per day and model. Page: /costs.
- [/api/effectiveness](https://ethiq.ethpandaops.io/api/effectiveness?dataset=fusaka&prompt=bare&refusals=fallback): per-run pass, cost, tokens and latency for each model and reasoning effort. Page: /effectiveness.
- [/api/labs](https://ethiq.ethpandaops.io/api/labs?dataset=fusaka&prompt=bare&refusals=fallback) and `/api/labs/{lab}`: per-lab summaries. Pages: /labs, /lab/{lab}.
- `/api/models/{model_id}`: one model's per-category, per-question and per-fork results. Page: /model/{model_id}.
- [/api/questions](https://ethiq.ethpandaops.io/api/questions?dataset=fusaka&prompt=bare&refusals=fallback): question list (id, category, difficulty, fork, tags, description). Page: /questions.
- `/api/questions/{question_id}` and `/api/questions/{question_id}/runs?model_id={id}&prompt={prompt}`: one question, and the raw model answers for it. Page: /question/{question_id}.
- [/api/refusals](https://ethiq.ethpandaops.io/api/refusals?dataset=fusaka&prompt=bare): which models refused to answer, and how their fallback model did.
- [/api/question-count](https://ethiq.ethpandaops.io/api/question-count?dataset=fusaka): number of questions matching the filters.
- [/api/version](https://ethiq.ethpandaops.io/api/version): build commit and time.

## Filter params

Comma-separated where plural. All optional.

- `dataset`: question set, e.g. `fusaka`, `fusaka-heldout`, `fork-choice-compound-hard`. Without one, all questions are in scope.
- `prompt`: how the model was run: `bare` (plain API call), `claude-code` or `codex` (agentic harnesses). Without one, every prompt is returned, so the same model can appear several times.
- `models` (ids from /api/filters), `lab`, `provider`, `precisions`
- `categories`, `difficulties`, `forks`, `tags`, `search` (question text)
- `open_weights=true`, `min_params`, `max_params` (billions), `params_type=active|total`, `include_unknown_params=true`
- `include_partial=true`: also list models that have not answered every question in scope (hidden by default so scores are comparable).
- `refusals=fallback`: score a refused answer with the model's refusal fallback instead of as a failure (the website's default).

## Notes

- Scores are the share of questions answered correctly, graded automatically against ground truth. Intervals are 95% Wilson intervals.
- Models with different reasoning efforts, providers or precisions are different rows. Use `model_id` to tell them apart.
- Source code and run data: https://github.com/ethpandaops/ai-evals
- Humans: https://ethiq.ethpandaops.io/about