Methodology

Same inputs for every provider, list-price settings as documented, three runs averaged, raw results and harness code published, every run timestamped. One default configuration per provider plus at most one premium configuration, named as in its docs.

Test sets (set-v1, frozen per quarter)

• Search: 50 factual, news set of 25–50 fresh queries (24–72 h old, regenerated weekly).
• Scrape (150 URLs): 50 static, 50 JS-heavy, 30 PDFs/docs, 20 hard (bot-protected, public).
• Browser (20 tasks): scripted Playwright tasks over CDP.

Metrics

• Latency p50/p95 from a US-East GitHub runner
• Success rate (2xx and content check passes)
• Freshness
• Cost per 1k and cost per 1k successful calls, at that day's checked price
• Output size (tokens)

Cadence

Price check daily; smoke run (10 items) daily; full benchmark weekly; new test set quarterly.

Cost columns

Two columns per provider: entry tier (cheapest paid tier, list price) and volume tier (cheapest published volume tier), each per 1k calls attempted and per 1k successful calls (∞ when none pass). Free credits never lower a cost; the free quota is shown as a note. A tier whose price isn't on the vendor's page shows "n/a". Browser-hour prices whose per-session minimum or rounding isn't published are marked "lower bound" (≥).

Freshness

fresh_hit@10 on the weekly news set, median lag (re-probes of misses at +6, +24 and +72 h) and the censored share, published with n, the week and the set sha256.

Terms of service

Where a provider's terms forbid a test class, that pair is marked "ToS-skipped". It counts as neither a pass nor a fail and is left out of every denominator.

Traceability

Every number in a table is generated from a raw run log line (runs/YYYY-MM-DD/<provider>/<set>.jsonl).