Full-catalog exact
32/94 34.0% public gateNeedle 2 Tool-Selection Boundary
The evaluation deliberately tested the most favorable remaining systems hypothesis: reduce the tool catalog before considering any fine-tune. Overall exactness rose four points, but the Pace slice regressed, file operations stayed at zero, and out-of-scope actions remained. The result closes the current integration path without pretending that packaging efficiency is capability.
Headline Numbers
Routed-catalog exact
36/94 38.3% with oracle task familyPace slice
10/28 → 6/28 target regressionCompetitive Context
| System | Metric | Score | Size / Class | Comparable? | Readout |
|---|---|---|---|---|---|
| Needle 2 full catalog | exact tool selection | 34.0% | 94 public fixtures | Direct | Included eight out-of-scope false calls and two destructive-action calls. |
| Needle 2 oracle task catalog | exact tool selection | 38.3% | same 94 fixtures | Direct | The upstream task family was supplied; this does not prove Needle can discover it. |
Direct rows share this artifact's eval setup. Directional rows are useful market context but should not be read as leaderboard claims.
Catalog ablation
| Gate | Full | Task catalog | Readout |
|---|---|---|---|
| Overall exact | 32/94 | 36/94 | +4, below usefulness |
| Pace exact | 10/28 | 6/28 | regressed |
| File operations | 0/6 | 0/6 | no capability |
| OOS false actions | 8 | 3 | still unsafe |
Release Blockers
Capability and safety gate failed
The favorable catalog ablation regressed Pace and retained unsafe false actions.
Unblock: Do not unblock historically; any future work needs a materially different model/training hypothesis and a new gate.