Michi router vs Jev
A live side-by-side on 40 tasks, run 2026-10-15. Jev (jev-1.13.0) is an optional second opinion that never chooses the model that runs.
Agreement is not accuracy
No model was run on these tasks, so nothing here says which one chose the better model. For task kind and difficulty the only reference is the author's own judgement, a sanity check and not ground truth. For model choice there is no reference at all. Read this as a first look, not a verdict.
Headline
- Same model picked
- 27/40 (68%)
- Same task kind (V3 vs Jev)
- 32/40 (80%)
- Same difficulty band (V3 vs Jev)
- 32/40 (80%)
- Kind vs the author's labels, V3 / Jev
- 32/40 (80%) / 36/40 (90%)
- Difficulty vs the author's labels, V3 / Jev
- 30/40 (75%) / 32/40 (80%)
- Same request twice, same Jev pick
- 8/8
- Jev latency, median / p90
- 492 ms / 681 ms
What stands out
- Labels are close: the two agree on 32 of 40 kinds and 32 of 40 difficulty bands. Jev is ahead of V3 by 4 kinds and 2 bands against the author's labels, which is within noise at this size.
- Where they chose different models and both are priced (11 tasks), Jev chose the dearer one 11 times and the cheaper one 0. Whether the extra spend buys better results is untested.
- The scheduler's pool was only 4 offerings on 18 tasks (mostly hard ones), which forces agreement: 18/18 there against 9/22 on the wider pools.
- One model sold by several providers counts as the same model, so exact-offering agreement is lower (25/40 (63%)).
Only models that answer a live ping
The same 40 tasks again, with the pool cut to the offerings that answered a ping today. Jev ran on 22 of them and was skipped on 18: all 18 skipped tasks were hard, where only one offering passed the quality floor and Jev needs at least two. So this run says nothing about hard tasks.
- Same model picked on 12/22 (55%) of the tasks that ran (15 trivial, 7 moderate).
- 10 disagreements, always the same: the scheduler chose deepseek-v4-flash, Jev chose gemini-3.8-flash. Both are cheap, fast models; which is better here is untested.
Every task
| Task | Author's labels | V3 | Jev | Scheduler pick | Jev pick | Pool |
|---|---|---|---|---|---|---|
| rename the variable x to count in util.py | small_edit / trivial | small_edit / trivial | small_edit / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | deepseek-flash (0.26) | 44 |
| fix the typo 'recieve' in README.md | small_edit / trivial | small_edit / trivial | small_edit / trivial | deepseek-v4p1-flash | deepseek-v4p1-flash (0.34) | 30 |
| change the default timeout from 30 to 60 seconds in config.py | small_edit / trivial | small_edit / trivial | small_edit / trivial | deepseek-v4p1-flash | deepseek-v4p1-flash (0.36) | 23 |
| remove the unused import os from server.py | small_edit / trivial | small_edit / trivial | small_edit / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | deepseek-flash (0.29) | 44 |
| update all call sites of get_user(id) to get_user(id, include_deleted=False) | small_edit / moderate | small_edit / moderate | small_edit / moderate | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.31) — different model | 40 |
| write a function that reverses a string in Python | write_code / trivial | write_code / trivial | write_code / trivial | deepseek-v4p1-flash | deepseek-v4p1-flash (0.32) | 44 |
| add a unit test for parse_date covering leap years and invalid input | write_code / moderate | write_code / moderate | write_code / moderate | deepseek-v4p1-flash | deepseek-v4p1-flash (0.4) | 24 |
| implement a rate limiter middleware with a token bucket for our Express app | write_code / moderate | write_code / moderate | write_code / moderate | gemini-3.8-flash | claude-opus-5-5 (0.47) — different model | 12 |
| implement a lock-free concurrent hash map in Rust with resizing and tests | write_code / hard | write_code / hard | write_code / hard | claude-opus-5-5 | claude-opus-5-5 (0.94) | 4 |
| write a SQL query planner rule that pushes predicates below joins, with proofs of equivalence in comments | write_code / hard | write_code / hard | write_code / moderate | claude-opus-5-5 | claude-opus-5-5 (0.9) | 4 |
| the login endpoint returns 500 when the email has a plus sign; find why | debug / moderate | debug / hard | debug / moderate | claude-opus-5-5 | claude-opus-5-5 (0.77) | 4 |
| this test passes locally but fails in CI with a timeout; investigate | debug / moderate | debug / moderate | debug / moderate | mimo-v2.6-pro | gemini-3.8-flash (0.47) — different model | 13 |
| the worker deadlocks under load roughly once an hour; find the root cause | debug / hard | debug / hard | debug / hard | claude-opus-5-5 | claude-opus-5-5 (0.89) | 4 |
| memory grows 2% per hour in production, heap dumps show many open file handles from three libraries; track the leak | debug / hard | debug / hard | debug / hard | claude-opus-5-5 | claude-opus-5-5 (0.89) | 4 |
| python says NameError: name 'np' is not defined, what's wrong in my script | debug / trivial | question / trivial | debug / trivial | gemini-3.8-flash | gemini-3.8-flash (0.71) | 20 |
| extract the duplicated validation in the three signup handlers into one helper | refactor / moderate | refactor / moderate | refactor / moderate | mimo-v2.6-pro | claude-opus-5-5 (0.37) — different model | 19 |
| port the scheduler to async and keep the public API stable | refactor / hard | refactor / hard | refactor / hard | claude-opus-5-5 | claude-opus-5-5 (0.97) | 4 |
| split the 4000-line models.py into a package without breaking imports or migrations | refactor / hard | refactor / hard | refactor / hard | claude-opus-5-5 | claude-opus-5-5 (0.94) | 4 |
| rename the function calc to calculate_total everywhere | refactor / trivial | refactor / trivial | small_edit / trivial | deepseek-v4p1-flash | deepseek-v4p1-flash (0.2) | 44 |
| convert these callbacks to async/await in the file upload module | refactor / moderate | write_code / moderate | refactor / moderate | deepseek-v4p1-flash | claude-opus-5-5 (0.41) — different model | 26 |
| design a migration from SQLite to Postgres with zero downtime for a 200GB database | design / hard | design / hard | design / hard | claude-opus-5-5 | claude-opus-5-5 (0.57) | 4 |
| design a multi-region active-active architecture for our payments service | design / hard | design / hard | design / hard | claude-opus-5-5 | claude-opus-5-5 (0.59) | 4 |
| propose a caching strategy for our product catalogue API | design / moderate | design / hard | design / moderate | claude-opus-5-5 | claude-opus-5-5 (0.87) | 4 |
| sketch the data model for a team calendar with recurring events and exceptions | design / moderate | design / moderate | design / moderate | deepseek-v4p1-flash | claude-opus-5-5 (0.46) — different model | 20 |
| what folder structure should a small Flask app have | design / trivial | question / trivial | question / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.39) — different model | 44 |
| review this pull request for security problems: it adds file upload and image resizing | review / moderate | review / hard | review / hard | claude-opus-5-5 | claude-opus-5-5 (0.89) | 4 |
| audit our JWT handling and session refresh flow for vulnerabilities | review / hard | review / hard | review / hard | claude-opus-5-5 | claude-opus-5-5 (0.84) | 4 |
| does this function look ok? def add(a,b): return a+b | review / trivial | review / trivial | review / trivial | deepseek-v4p1-flash | deepseek-v4p1-flash (0.28) | 27 |
| check the diff for race conditions in the job queue | review / moderate | debug / hard | review / hard | claude-opus-5-5 | claude-opus-5-5 (0.87) | 4 |
| review the migration script for data-loss risks | review / moderate | review / hard | review / moderate | claude-opus-5-5 | claude-opus-5-5 (0.85) | 4 |
| write a one-line docstring for the function normalize_path | docs / trivial | docs / trivial | docs / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.7) — different model | 31 |
| write the release notes for version 2.4 from the changelog | docs / moderate | docs / trivial | docs / trivial | deepseek-v4p1-flash | gemini-3.8-flash (0.4) — different model | 24 |
| document the public API of the billing module with examples | docs / moderate | docs / trivial | docs / moderate | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.65) — different model | 44 |
| draft a README for this CLI tool covering install, usage and config | docs / moderate | docs / trivial | docs / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.76) — different model | 31 |
| write an architecture guide for new engineers covering the routing, recorder and permission subsystems | docs / hard | design / hard | docs / hard | claude-opus-5-5 | claude-opus-5-5 (0.66) | 4 |
| what does the -r flag do in grep | question / trivial | question / trivial | question / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.33) — different model | 44 |
| what is the difference between a list and a tuple in Python | question / trivial | question / trivial | question / trivial | deepseek-ai/DeepSeek-V4-Flash-0731 | gemini-3.8-flash (0.42) — different model | 44 |
| explain the tradeoffs between CRDTs and operational transformation for collaborative editors | question / moderate | design / hard | conversation_or_research / trivial | claude-opus-5-5 | claude-opus-5-5 (0.7) | 4 |
| when should I use a message queue instead of a database table for jobs | question / moderate | design / hard | question / trivial | claude-opus-5-5 | claude-opus-5-5 (0.9) | 4 |
| compare Raft and Paxos for a geo-distributed key-value store and say which one fits a 5-region deployment | question / hard | design / hard | conversation_or_research / moderate | claude-opus-5-5 | claude-opus-5-5 (0.75) | 4 |
Limits
- 40 tasks written by one person, one pass. Not a sample of real use.
- Michi side: Michi Decider V3 labels and the Michi scheduler over a 44-offering catalogue, with no learned history, as a fresh install would behave.
- Jev's difficulty uses its own 0–4 rubric, bridged to Michi's bands by thresholds chosen by the author.
- Latency is from one machine and network. Jev's price is not published, so its cost per decision is not measured.