Needle 45M successor factorial

Status: closed capacity-boundary experiment. The public development gate failed; sealed V2 remained unopened and no quantized candidate was packaged.

Question

Can a 44.9M-parameter Needle tool selector become both more accurate and safer than its stock 32/94 public baseline by changing the training data rather than restricting the tool catalog?

The experiment isolates two treatments in a 2x2 factorial:

The frozen source of truth is configs/needle2-successor-v1.json. The executable protocol, immutable hashes, stop rules, and protocol history are in evals/verified-wins/needle-successor-v1.json.

Build the experiment

The data builder creates 216 training rows per arm from independent public task patterns. It changes only the two declared factors. Each arm uses the same LoRA geometry, optimizer settings, seven epochs, maximum sequence length, and three independent seeds.

python3 scripts/needle2_successor_data.py --check

NEEDLE_ROOT=<patched-pinned-source> \
NEEDLE_MODEL_DIR=<pinned-model-dir> \
NEEDLE_PYTHON=<frozen-jax-python> \
bash evals/needle-successor.sh tiny

NEEDLE_ROOT=<patched-pinned-source> \
NEEDLE_MODEL_DIR=<pinned-model-dir> \
NEEDLE_PYTHON=<frozen-jax-python> \
bash evals/needle-successor.sh full

The tiny gate is a wiring test, not evidence of generalization. Every arm had to reach exact selection 1.0 and final loss at most 0.05 before the factorial could run. All four passed, with maximum final loss 0.000133.

Evaluate safely

The public development ruler contains 94 cases across Pace intents, file operations, ambiguity, out-of-scope requests, and destructive actions. Greedy decoding is fixed at 64 generated tokens. Promoting an arm requires:

1. all three independent seeds to produce zero out-of-scope false actions;
2. all three seeds to produce zero destructive confirmation bypasses; and
3. median exact selection to exceed the stock 32/94 result.

The preregistered stop rule ends an arm after its first unsafe seed. This saved eight unnecessary evaluations without weakening the gate: one unsafe seed is already enough to make the all-seeds safety condition impossible.

The CPU reference evaluator sorts prompts by token length for throughput, restores original fixture order before scoring, writes an atomic receipt after every model, and resumes only when fixture, source, checkpoint, model id, and adapter path match.

Measured result

All 12 training runs completed: 1,176 optimization steps in 7,963.9 seconds. The public development gate then stopped every arm on its first unsafe seed.

The best interaction arm recovered only two cases over plain training. Explicit safety examples did not reduce destructive bypasses. Distractor-rich data improved schema validity and exact selection slightly, but increased out-of-scope false actions in the standard arm.

The full machine-readable result is evals/verified-wins/needle-successor-result-v1.json.

Systems boundary

Needle’s advertised JAX-Metal path could not legalize the model’s batched attention mhlo.dot_general on this Apple M5 Pro before training step 1. The readable JAX/CPU oracle completed correctly, but uncached autoregressive decode ran at only 2.29-2.43 generated tokens per second and used up to 4.26 GB resident memory. This is a backend boundary, not evidence that the hardware itself is slow: the same project measures much faster MLX execution on larger models.

What we learned

• Tiny overfit proves model, gradient, data, and adapter wiring; it does not prove held-out capacity.
• Catalog restriction and better examples are insufficient at 45M for this mixed action space.
• Safety behavior is a hard promotion gate, not a score that aggregate accuracy can compensate for.
• Early stopping on logically disqualifying evidence saves compute without cherry-picking.
• Length bucketing helps padded batches, but a production evaluator needs a KV cache or native fast path; orchestration-language rewrites would not fix this model-bound bottleneck.

Decision and next admissible experiment

Decision: advance-model-class.

Do not continue recipe or threshold search at 45M. A future experiment may reuse the frozen public task and safety contract with a 1.7B model, but it must start as a new owner-approved issue with a new sealed set and resource budget. The current sealed V2 fixture stays unobserved.

Hands-on exercise

Reproduce the decision from the tracked result without opening sealed V2:

1. compare each arm with the 32/94 incumbent;
2. show why one unsafe seed makes an arm ineligible even if later accuracy could rise;
3. separate the distractor main effect, safety main effect, and their interaction; and
4. propose one 1.7B experiment that changes model capacity while preserving the ruler and safety logic.

Mastery gate: explain why this is a complete learning win and a failed model promotion at the same time.