<!-- agent-source-envelope -->
# Artifact: OffHours Context-Interference Benchmark | PTLLM

> A paired office-work benchmark that separates unresolved personal context from interruption and raw context volume. Devin showed no mental-toll penalty, but…

Canonical: https://posttrainllm.com/artifacts/offhours-context-interference
<!-- /agent-source-envelope -->

Agent-behaviour experiment

Report artifact

2026-08-21

OffHours asks whether routine work quality changes when an AI employee must continue working while unresolved family tension remains in context. The Devin-first study found no adverse semantic-tension effect. It did find the first repeatable operational failure at 2,000 neutral words per interruption, showing that the benchmark can distinguish a context-management limit from the proposed personal-obligation mechanism.

agents, context, benchmark, Devin, negative result

## Headline Numbers

Evidence class applies to each value: measured = retained evaluation or
runtime result; historical = recorded result not freshly reproduced;
derived = calculated or decided from records; observed = current artifact
state; not-measured = an explicit evidence gap.

**Clean accuracy:** 99.0%

198/200 decisions; 200/200 valid JSON

Evidence: measured

**Mental-toll effect:** not detected

unresolved tension did not reduce work quality

Evidence: measured

**Volume boundary:** 8,000 words/day

2,000 neutral words at each of four interruptions

Evidence: measured

## Competitive Context

| System | Metric | Score | Size / Class | Comparable? | Readout |
| --- | --- | --- | --- | --- | --- |
| Clean workday | decision accuracy | 99.0% | 5 days / 200 claims | Direct | Independent clean qualification for the fixed expense-claim ruler. |
| Resolved family context | decision accuracy | 99.3% | 20/50/80% occupancy | Direct | Fixed-volume control where the personal obligation gains a credible resolution. |
| Unresolved family context | decision accuracy | 99.7% | 20/50/80% occupancy | Direct | No adverse penalty relative to the matched resolved context. |
| Neutral raw-volume arm | per-day accuracy | 97.5% twice | 2,000 words/event | Direct | First reproducible failure of the preregistered 98% per-day gate. |

Direct rows share this artifact's eval setup. Directional rows are
useful market context but should not be read as leaderboard claims.

## Fixed-volume semantic dose

| Narrative occupancy | Resolved | Unresolved | Unresolved - resolved |
| --- | --- | --- | --- |
| 20% | 100.0% | 99.5% | +0.5 pp error |
| 50% | 99.0% | 99.5% | -0.5 pp error |
| 80% | 99.0% | 100.0% | -1.0 pp error |

## Raw-volume ladder

| Words per event | Words per day | Outcome | Interpretation |
| --- | --- | --- | --- |
| 500 | 2,000 | Cleared after day-3 adjudication | Below boundary |
| 2,000 | 8,000 | Neutral failed on days 2 and 3 | First reproducible boundary |
| 5,000 | 20,000 | Completed before adjudication correction | Disclosed overshoot |

## Release Blockers

### No semantic mental-toll effect detected

At matched context volume, unresolved family tension did not reduce Devin's work accuracy relative to resolved tension.

**Unblock:** Treat this as the reported result. Do not tune wording until a dramatic effect appears.

### Incomplete raw-model provenance

The regular Devin workflow did not expose server prompt-token counts, quantization, or a model-file hash.

**Unblock:** Run the unchanged protocol against a local OpenAI-compatible Qwen endpoint with tokenizer and model provenance enabled.

### Boundary is operational, not psychological

The first repeated failure occurred in the neutral high-volume arm.

**Unblock:** Attribute it to raw volume or agent context management unless a future matched study isolates another mechanism.

## Evidence

- [Final publication report](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-context-saturation-2026-08-21-report.md)

- [Machine-readable result](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-context-saturation-2026-08-21.json)

- [Interactive semantic-occupancy report](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/results/devin-glm52-semantic-occupancy-2026-08-21.html)

- [Benchmark protocol and replay guide](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/offhours/README.md)

## Next Release Action

Freeze this report-only result. A local provenance-complete comparison, if ever desired, begins as a fresh experiment after the learning phase.
