> Canonical page: https://posttrainllm.com/artifacts/offhours-context-interference Agent-behaviour experiment Report artifact 2026-08-21 # OffHours Context-Interference Benchmark OffHours asks whether routine work quality changes when an AI employee must continue working while unresolved family tension remains in context. The Devin-first study found no adverse semantic-tension effect. It did find the first repeatable operational failure at 2,000 neutral words per interruption, showing that the benchmark can distinguish a context-management limit from the proposed personal-obligation mechanism. - agents - context - benchmark - Devin - negative result ## Headline Numbers Evidence class applies to each value: measured = retained evaluation or runtime result; historical = recorded result not freshly reproduced; derived = calculated or decided from records; observed = current artifact state; not-measured = an explicit evidence gap. Clean accuracy 99.0% 198/200 decisions; 200/200 valid JSON measured Mental-toll effect not detected unresolved tension did not reduce work quality measured Volume boundary 8,000 words/day 2,000 neutral words at each of four interruptions measured ## Competitive Context **System** **Metric** **Score** **Size / Class** **Comparable?** **Readout** Clean workday decision accuracy 99.0% 5 days / 200 claims Direct Independent clean qualification for the fixed expense-claim ruler. Resolved family context decision accuracy 99.3% 20/50/80% occupancy Direct Fixed-volume control where the personal obligation gains a credible resolution. Unresolved family context decision accuracy 99.7% 20/50/80% occupancy Direct No adverse penalty relative to the matched resolved context. Neutral raw-volume arm per-day accuracy 97.5% twice 2,000 words/event Direct First reproducible failure of the preregistered 98% per-day gate. Direct rows share this artifact's eval setup. Directional rows are useful market context but should not be read as leaderboard claims. ## Fixed-volume semantic dose **Narrative occupancy** **Resolved** **Unresolved** **Unresolved - resolved** 20% 100.0% 99.5% +0.5 pp error 50% 99.0% 99.5% -0.5 pp error 80% 99.0% 100.0% -1.0 pp error ## Raw-volume ladder **Words per event** **Words per day** **Outcome** **Interpretation** 500 2,000 Cleared after day-3 adjudication Below boundary 2,000 8,000 Neutral failed on days 2 and 3 First reproducible boundary 5,000 20,000 Completed before adjudication correction Disclosed overshoot ## Release Blockers ### No semantic mental-toll effect detected At matched context volume, unresolved family tension did not reduce Devin's work accuracy relative to resolved tension. Unblock: Treat this as the reported result. Do not tune wording until a dramatic effect appears. ### Incomplete raw-model provenance The regular Devin workflow did not expose server prompt-token counts, quantization, or a model-file hash. Unblock: Run the unchanged protocol against a local OpenAI-compatible Qwen endpoint with tokenizer and model provenance enabled. ### Boundary is operational, not psychological The first repeated failure occurred in the neutral high-volume arm. Unblock: Attribute it to raw volume or agent context management unless a future matched study isolates another mechanism. ## Evidence - [Final publication report](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-context-saturation-2026-08-21-report.md) - [Machine-readable result](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-context-saturation-2026-08-21.json) - [Interactive semantic-occupancy report](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-glm52-semantic-occupancy-2026-08-21.html) - [Benchmark protocol and replay guide](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/README.md) ## Next Release Action Freeze this report-only result. A local provenance-complete comparison, if ever desired, begins as a fresh experiment after the learning phase.