> Canonical page: https://posttrainllm.com/apple-foundation-models

[← research lab](https://posttrainllm.com/)

Mac-local specialist field guide

# Apple Foundation Models on-device: measured capabilities and limits

We scored Apple's on-device Foundation Models framework on real agentic gates. Measured verdict: a free, private floor for lightweight turns — it cannot ground tool actions and its 4096-token context cannot hold a real tool catalog.

Apple's FoundationModels framework (macOS/iOS 26+) exposes an on-device ~3B model through LanguageModelSession with structured output ( @Generable ), guided generation ( DynamicGenerationSchema ), and a Tool protocol. We scored it on our own agentic gates to answer a concrete question: can it ground actions well enough to serve as an agent's brain? The measured verdict: a free, private, battery-cheap floor for lightweight turns — not an agentic action model.

## The measured results

**Gate**

**Result**

**Why**

 BFCL agentic breadth, full catalog (n=8 VehicleControl) 25% Schemas present, but the ~3–4.4k-token catalog nearly overflows context

 BFCL agentic breadth, compact catalog (52 tasks) ~0% Fits context, but stripped parameter schemas produce wrong arguments

 Planner gate (action-grounding) 13% Picks the right tool name, cannot ground the arguments

 Planner gate (out-of-scope refusal) ~95% Judgment-light classification is its genuine strength

Three findings explain the numbers:

- It cannot ground actions. It selects the right tool name under enum constraints, then fills arguments wrong. In one probe the gold call was lockDoors(unlock=True, door=['driver','passenger','rear_left','rear_right']) ; the model emitted lockDoors(unlock=false, door="all") — inverted the boolean and guessed a string for an enum list. Some gold arguments are unguessable without the schema.

- The context catch-22. The full catalog carries schemas but overflows the 4096-token context; the compact catalog fits but drops the schemas. Either way, the on-device context cannot host a real agentic tool catalog.

- It is not faster. Roughly 3–4s per step — the same ballpark as a 4B MLX model with about 8× the context. Its real advantages are ANE perf-per-watt and zero setup, RAM, or cost.

## How we measured it

The on-device model does not speak the OpenAI function-calling API our harness uses, so we built a standalone Swift HTTP bridge ( fm_agent_bridge.swift ) that exposes it behind an OpenAI-compatible chat-completions endpoint. Tool calling runs through guided generation: each request builds a DynamicGenerationSchema of {tool_calls:[{name, arguments_json}], message} and reads the generated JSON back out. With that bridge in place, the same BFCL multi-turn scorer runs unchanged — the on-device model is just another backend.

## Where it fits — and where it does not

Use it as the free floor tier in a routing setup: lightweight, private, offline, battery-sensitive turns such as classification, refusal, and short answers stay on-device; grounded agentic work escalates to a distilled specialist; frontier jobs escalate further. Apple commoditized the plumbing — on-device LLM, tools, and structured output as an OS API — not the model capability your product is differentiated on.

Two paths we evaluated and declined: adapter-tuning Apple's model ( .fmadapter ) locks the product to Apple's model, format, and OS; Core ML–compiling your own weights to the ANE remains a possible future battery optimization, not a capability bet.

## Related

- [Evaluate a local LLM honestly](https://posttrainllm.com/evaluate-local-llm)

- [The eval leaderboard](https://posttrainllm.com/eval-leaderboard)

- [Build a routed specialist](https://posttrainllm.com/build-small-language-model-specialist)

- [Fine-tune an LLM on a Mac](https://posttrainllm.com/fine-tune-llm-on-mac)

 CTA: Read the full measurement notes in the repository's docs/learn/apple-on-device-foundation-models.md , including the bridge source and per-gate tables.

Continue through the factory

[evaluate local llm](https://posttrainllm.com/evaluate-local-llm)[eval leaderboard](https://posttrainllm.com/eval-leaderboard)[fine tune llm on mac](https://posttrainllm.com/fine-tune-llm-on-mac)[build small language model specialist](https://posttrainllm.com/build-small-language-model-specialist)[experiments](https://posttrainllm.com/experiments)[artifacts](https://posttrainllm.com/artifacts)