VLM A/B — UI-Venus-1.5-2B vs Qwen3-VL-2B for Pace's vision pillar
VLM A/B — UI-Venus-1.5-2B vs Qwen3-VL-2B for Pace’s vision pillar
Status: PRD (2026-06-09). The 2026-06-09 research sweep flagged Qwen3-VL-2B as a competitor to our UI-Venus-1.5-2B port target. This A/B decides which one we actually port to MLX-Swift / CoreML / ANE for #266.
Why this matters
#266 (UI-Venus M4 port) is 6-9 days of focused MLX-Swift work. We should not port the wrong model. If Qwen3-VL-2B wins the A/B on Pace’s actual use cases:
- One Qwen3 family across planner + embedding + VLM (architectural simplification, one tokenizer, one quant pipeline)
- Already validated for “native computer/mobile control” per research (UI-TARS-2 paper benchmark)
If UI-Venus wins:
- Microsoft tuned it specifically on UI screenshots (Windows/web/mobile)
- Smaller specialization variance
- Stay with our current port plan
Models on disk (downloaded 2026-06-09)
~/.lmstudio/models/mlx-community/UI-Venus-1.5-2B-6bit— already there for the port~/.cache/posttrainllm/models/qwen3-vl-2b-instruct/— 2.8 GB, just downloaded
Both available for A/B without further setup.
Test plan (1-2 days, no porting required)
Run both via mlx_lm in Python (no Swift port needed for A/B). Compare on a Mac-specific fixture set.
Fixture set construction
pace/evals/fm-vlm-fixtures-mac-v1/ — 30 fixtures, captured via posttrainllm ax-capture:
Categories:
- Identity (8): “what app is this” on Mac apps Microsoft probably didn’t train on — Xcode, Final Cut, Logic, OmniFocus, Bear, Things, Tot, Drafts
- Read-on-screen (8): “read the error message” / “what’s the title of this email” / “what’s the current playback time”
- Click target (8): “where do I click to send this” / “find the New Folder button” — model emits (x,y) or label, scored against ground-truth AX bounds
- Activity context (6): “what am I doing right now” — multi-modal answer (app + visible content + task)
Capture process: 30 manual ax-capture sessions across daily Mac use (existing tool, ~30 min).
Eval methodology
For each model:
- Load via mlx_lm chat
- For each fixture: pass the screenshot + AX-tree-as-text + the question
- Score via:
- Identity: string match on app name
- Read: substring containment of expected text in response
- Click: response label matches AX-tree label OR (x,y) lies inside ground-truth bounds
- Activity: LLM-as-judge (Qwen3-14B teacher) scores 0-2 on coverage
Eval script: scripts/archive/eval_pace_vlm_ab.py (write — extends eval_pace_v2.py pattern).
Metrics captured
| Metric | UI-Venus-1.5-2B | Qwen3-VL-2B |
|---|---|---|
| Identity accuracy (/8) | ? | ? |
| Read accuracy (/8) | ? | ? |
| Click accuracy (/8) | ? | ? |
| Activity accuracy (/6) | ? | ? |
| Total (/30) | ? | ? |
| Per-call latency (MPS) | ? ms | ? ms |
| Per-call latency (ANE via CoreML if convertible) | ? ms | ? ms |
| Disk (post-quantization to 4-bit MLX) | ? MB | ? MB |
| Formula score | ? | ? |
Tiebreakers (when scores are close)
- Same family wins (Qwen3) — one tokenizer + one MLX pipeline for planner + embedding + VLM. Worth ~5pp.
- Smaller bundle wins — Pace’s total bundle budget matters for v1 ship.
- Better ANE conversion path wins — whichever model converts to ANE-chunked CoreML with less drama.
Decision tree
| Result | Action |
|---|---|
| Qwen3-VL ≥ UI-Venus by ≥ 5pp on Mac fixtures | Switch port target to Qwen3-VL-2B. Same family alignment compounds the win. |
| UI-Venus ≥ Qwen3-VL by ≥ 5pp | Stay on UI-Venus. Continue M4 port. |
| Within 5pp tie | Pick Qwen3-VL — family alignment + ecosystem momentum (Qwen3-VL native computer/mobile control per research). |
What “Mac-tune later” looks like (out of scope here)
Either winner can be fine-tuned later on Pace’s ax-capture corpus once Pace is deployed and we have real screen+AX pairs. The A/B picks the BASE; tuning is a separate factory pass.
Done when
- 30 Mac fixtures captured + ground-truth labeled
- Both models scored on the same fixtures
- Formula scores computed
- Decision recorded in memory
- Port target locked for #266
Related
- #266 UI-Venus M4 port (consumer of this decision)
project-embedding-swap-2026-06-09(parallel: same-family-wins logic applied to embedding)feedback-posttrainllm-north-star(formula bar applied here)feedback-research-first-doctrine(this A/B IS the research-first response to the Qwen3-VL surfacing)