Apple's FoundationModels framework (macOS/iOS 26+) exposes an on-device ~3B model through LanguageModelSession with structured output (@Generable), guided generation (DynamicGenerationSchema), and a Tool protocol. We scored it on our own agentic gates to answer a concrete question: can it ground actions well enough to serve as an agent's brain? The measured verdict: a free, private, battery-cheap floor for lightweight turns — not an agentic action model.
The measured results
| Gate | Result | Why |
|---|---|---|
| BFCL agentic breadth, full catalog (n=8 VehicleControl) | 25% | Schemas present, but the ~3–4.4k-token catalog nearly overflows context |
| BFCL agentic breadth, compact catalog (52 tasks) | ~0% | Fits context, but stripped parameter schemas produce wrong arguments |
| Planner gate (action-grounding) | 13% | Picks the right tool name, cannot ground the arguments |
| Planner gate (out-of-scope refusal) | ~95% | Judgment-light classification is its genuine strength |
Three findings explain the numbers:
- It cannot ground actions. It selects the right tool name under enum constraints, then fills arguments wrong. In one probe the gold call was
lockDoors(unlock=True, door=['driver','passenger','rear_left','rear_right']); the model emittedlockDoors(unlock=false, door="all")— inverted the boolean and guessed a string for an enum list. Some gold arguments are unguessable without the schema. - The context catch-22. The full catalog carries schemas but overflows the 4096-token context; the compact catalog fits but drops the schemas. Either way, the on-device context cannot host a real agentic tool catalog.
- It is not faster. Roughly 3–4s per step — the same ballpark as a 4B MLX model with about 8× the context. Its real advantages are ANE perf-per-watt and zero setup, RAM, or cost.
How we measured it
The on-device model does not speak the OpenAI function-calling API our harness uses, so we built a standalone Swift HTTP bridge (fm_agent_bridge.swift) that exposes it behind an OpenAI-compatible chat-completions endpoint. Tool calling runs through guided generation: each request builds a DynamicGenerationSchema of {tool_calls:[{name, arguments_json}], message} and reads the generated JSON back out. With that bridge in place, the same BFCL multi-turn scorer runs unchanged — the on-device model is just another backend.
Where it fits — and where it does not
Use it as the free floor tier in a routing setup: lightweight, private, offline, battery-sensitive turns such as classification, refusal, and short answers stay on-device; grounded agentic work escalates to a distilled specialist; frontier jobs escalate further. Apple commoditized the plumbing — on-device LLM, tools, and structured output as an OS API — not the model capability your product is differentiated on.
Two paths we evaluated and declined: adapter-tuning Apple's model (.fmadapter) locks the product to Apple's model, format, and OS; Core ML–compiling your own weights to the ANE remains a possible future battery optimization, not a capability bet.
Related
- Evaluate a local LLM honestly
- The eval leaderboard
- Build a routed specialist
- Fine-tune an LLM on a Mac
CTA: Read the full measurement notes in the repository's docs/learn/apple-on-device-foundation-models.md, including the bridge source and per-gate tables.