Qaf · model evaluation · 4 November 2026No free OpenRouter model can serve tier 3
The free tier allows under 1.5% of tier 3 traffic. The closest model on quality, Nemotron 3.5 Super VL, is less accurate than Gemma and may train on user prompts.
My recommendation is not to use the free tier for tier 3, even as overflow. If we want a cheaper tier 3 model, paid Gemma 4 31B is still the one to beat, at about $0.0025 per answer in this bench.
It can't handle the load
answers a day. At peak traffic, the daily quota would last about 11 minutes.
| Limit | Free tier | Tier 3 needs |
|---|---|---|
| Requests per day | 1,000 | about 75,000 |
| Requests per minute | 20 | about 90 at peak |
- We reached the daily cap in testing. Soon after usage hit 1,000, requests failed with
free-models-per-day-high-balance: three got through in the next 15 seconds, then none. The limit header gives the reset as 00:00 UTC. - OpenRouter's docs say extra accounts or keys don't raise these limits. The caps are the same whichever free model we pick.
- Tier 3 volume comes from Langfuse for the 24 hours to 09:00 UTC on 4 Nov: 36,106 traces, peaking at 2,543 an hour. About 9% of sampled traces had no Gemini call logged, so the real answer count may be slightly lower.
The closest model is less accurate than Gemma
Each model answered the same 30 questions using real retrieval at tier 3 settings. A Claude judge compared each answer blind against Gemma and graded it against the passages it cited. Gemma used its own tuned prompt.
| Model | Preferred over Gemma | Correctness (of 5) | Major or critical issues | Fabricated or misattributed text | Cites invented IDs | Median answer |
|---|---|---|---|---|---|---|
| Gemma 4 31B (paid) | – | 4.17 | 9 | 0 | 1 | 40 s |
| Super VL, free | 13–17 | 3.53 | 17 | 2 | 0 | 43 s |
| Super VL, stricter prompt* | 13–17 | 3.63 | 18 | 0 | 0 | 33 s |
| Nemotron 3 Ultra, free | 12–18 | 3.20 | 23 | 2 | 2 | 41 s |
| North Mini, free | 2–27 | 2.67 | 28 | 7 | 13 | 50 s |
*Run on paid Super VL hosts because the free quota had run out. The first prompt scored the same correctness (3.53) on the free and paid hosts, so the host doesn't explain the differences. The paid runs cost $0.005–0.006 per answer, two to two and a half times Gemma.
- Preference looks close, accuracy doesn't. Super VL's 13–17 is within noise on 30 questions, but its correctness was lower than Gemma's on 15 questions and higher on none.
- The errors matter for religious rulings. Super VL told someone who forgot a prayer that it was invalid, when the cited passages say it is valid. It also labelled a Companion's report as a hadith of the Prophet ﷺ. Most of its major issues were views attributed to the wrong scholar or body, or rulings the passages don't support.
Prompt changes fixed format, not accuracy
- First pass, on format: a final checklist covering search, citation format and single-language answers, plus a required search call. Super VL and Ultra then searched on all 30 questions and cited on 28.
- Second pass, on accuracy: strict rules on attribution, no verdicts of its own, nothing imported from memory, and correct hadith tags. For Super VL, fabricated or misattributed religious text fell from 2 to 0 and critical issues from 2 to 1. Major issues rose from 15 to 17, and blind preference stayed the same.
Privacy rules out the best performers
All four Nemotron models and LFM only route if we allow providers that may train on prompts. Serving them would send real user questions, often personal religious ones, to those providers. Our eval rules forbid this for production data, so I tested with questions I wrote, and no user text was sent. The system prompt did reach those providers.
The working models that pass OpenRouter's no-training filter are North Mini, Dots, Flint and Xing, and all four were much weaker than Gemma.
Every free model
The peak test ran about 105 requests a minute for 30 seconds per model and stopped early after 15 straight failures. Preference and correctness come from the 8-question screen unless marked, so those numbers are rough. Screen models ran on Gemma's prompt without the tuning or required search that the 30-question finalists and Gemma had.
| Model | Peak test success | Preferred over Gemma | Correctness | Median answer | No-training filter | Result |
|---|---|---|---|---|---|---|
| Nemotron 3.5 Super VL | 50/52 | 13–17 (30 Q) | 3.53 | 43 s | No | Closest, but less accurate |
| Nemotron 3 Ultra | 43/52 | 12–18 (30 Q) | 3.20 | 41 s | No | More unsupported claims |
| North Mini | 44/52 | 2–27 (30 Q) | 2.67 | 50 s | Yes | Invents citation IDs |
| Nemotron 3.5 Lightning | 52/52 | 1–7 | 2.75 | 17 s | No | Fast; broken citations, Chinese in answers |
| Nemotron 3 Nano Omni | 45/52 | 0–8 | 3.12 | 93 s | No | Too slow |
| Flint | 46/52 | 2–6 | 3.25 | 166 s | Yes | Far too slow |
| Dots 3 Note | 41/52 | 2–6 | 2.62 | 28 s | Yes | Mixes languages, rarely cites |
| LFM 2.5 | 13/52 | 0–6 | 2.50 | 13 s | No | No valid citations; limited upstream |
| Xing 4.0 | 3/25 | 0–7 | 2.00 | 39 s | Yes | Limited upstream, poor quality |
| Gemma 4 31B (free) | 0/15 | – | – | – | Yes | Limited upstream; no answers |
| Gemma 4 26B (free) | 0/15 | – | – | – | Yes | Limited upstream; no answers |
| Crouton Alpha | 0/15 | – | – | – | Unclear | Every request returned 502 |
openrouter/free | 44/52 | – | – | – | – | Picks a random model each request, so not benchmarked |
| Inkling, Inkling Small | – | – | – | – | – | 403: agentic harnesses only; free period ends today |
| Lyria 3 (2), Content Safety | – | – | – | – | – | No tool support, so can't run our chat |
What this doesn't show
- Authored questions. The 30 questions match tier 3's measured language split and include follow-ups, but they aren't real traffic. Real questions couldn't go to the training endpoints.
- Small sample, one judge. Thirty questions and a single judge model can show a large gap, not a small one.
- Short load tests. Most ran for 30 seconds, and one Super VL run lasted 3 minutes and hit the daily cap after about 47 seconds. Short bursts got past the 20-a-minute limit, so sustained behaviour under it is untested. Upstream limits also vary by hour.
- The Gemma cost is a bench estimate. About $90 a day at 36,000 answers. Longer real conversations could raise it.