LOCAL CODING AGENT GUIDE

Run Inkling 2 Mini Locally for Coding Agents (32GB Mac & Single-GPU Setup)

Thinking Machines Lab’s Apache-2.0 Inkling 2 Mini (26.4B total / 3.4B active looped MoE, ~20GB checkpoint) runs locally on a 32GB Apple Silicon Mac or a single 32GB GPU and exposes an OpenAI-compatible API you can wire directly into OpenCode, Cline, or MCP coding workflows.

Updated 2026-11-22 · Official docs: Hugging Face Model Card · vLLM Recipe.

1. Hardware budget & why 32GB unified memory is the practical floor

Inkling 2 Mini (released November 20, 2026 by Thinking Machines Lab under Apache 2.0) packs 26.4B total parameters into a ~20GB mixed NVFP4/MXFP8 checkpoint, activating 3.4B parameters per pass (5.4B effective active across its 3-pass looped core). Because layers 6–17 loop three times with separate per-pass KV caches, the runtime maintains 48 KV-cache layers even though only 24 physical layers sit in RAM.

That design makes 32GB unified memory (Apple Silicon MacBook Pro, Mac mini, or Mac Studio) or a single 32GB+ VRAM GPU / NVIDIA DGX Spark the practical local baseline. The model supports up to 1M tokens of context on larger clusters, but on a 32GB machine you should keep context bounded to 32K–64K tokens with quantized KV cache so macOS or CUDA does not swap.

  • Apple Silicon (32GB+ unified memory): run via Ollama, Llama App, Exo Labs, or Unsloth GGUF + looped-layer llama-server.
  • NVIDIA DGX Spark (GB10) or single 32GB+ VRAM GPU: run the official vllm/vllm-openai:inkling-2-mini container with native NVFP4/MXFP8 weights and bundled DSpark speculative decoding.
  • Published coding & MCP baselines: 79.2% SWEBench Verified (bash only), 77.9% SWEBench Pro v2 Hard, 82.7% MCP Atlas (Public), and 48.2% DeepSWE v1.1.

2. Option A: Serve Inkling 2 Mini on a 32GB Apple Silicon Mac

On macOS, the fastest path is Ollama (when the inkling-2-mini tag is available in your updated Ollama build) or a quantized GGUF build from Unsloth served via llama-server. Always confirm your llama.cpp binary includes looped-layer support for Inkling 2 Mini—older builds will load weights incorrectly or emit broken tokens.

Exposing an OpenAI-compatible HTTP endpoint on localhost:8080 lets any local coding harness (OpenCode, Cline, Roo Code, or custom MCP clients) talk to the model without cloud API keys.

  • Start with -c 40000 (or 32768) and quantized KV cache (-ctk q8_0 -ctv q8_0) to stay comfortably inside 32GB unified memory.
  • Pass --mmproj only if your coding agent sends screenshots or UI mockups (Inkling 2 Mini accepts 40px–4096px image inputs via its 23-layer ViT).
  • If you prefer zero flag tuning on Apple Silicon, Ollama exposes an identical OpenAI-compatible endpoint at http://localhost:11434/v1 after running ollama run inkling-2-mini.
llama-server (32GB Mac GGUF with OpenAI-compatible API)
./llama-server \
  -m ./models/Inkling-2-Mini-IQ4_NL.gguf \
  -c 40000 -ngl 99 --flash-attn on \
  -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 --port 8080

Requires an updated llama.cpp / Unsloth build with Inkling 2 Mini looped-layer support. Add --mmproj ./models/mmproj-Inkling-2-Mini-Q8_0.gguf for screenshot/vision inputs.

3. Option B: Serve on NVIDIA 32GB+ GPU or DGX Spark with vLLM

For Linux workstations with 32GB+ VRAM or an NVIDIA DGX Spark, use Thinking Machines and vLLM's official Docker image (vllm/vllm-openai:inkling-2-mini, tracking vLLM 0.35.0+ nightly). This path runs the native ~20GB NVFP4/MXFP8 checkpoint directly and activates the bundled DSpark speculator for faster token generation during long multi-file edits.

Include --reasoning-parser inkling so coding agents receive structured reasoning traces separately from final tool calls and diffs.

  • Mount ~/.cache/huggingface so the ~20GB checkpoint persists across container restarts.
  • Bind -p 127.0.0.1:8000:8000 instead of 0.0.0.0 when running on a shared LAN so your local inference server is not exposed externally.
  • Verify startup by querying http://127.0.0.1:8000/v1/models and confirming thinkingmachines/Inkling-2-Mini is listed.
Official vLLM Docker serve command (with DSpark speculator)
docker run --gpus all --privileged --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \
  --tensor-parallel-size 1 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

Source: recipes.vllm.ai/thinkingmachines/Inkling-2-Mini. Requires NVIDIA driver/CUDA stack compatible with vLLM 0.35.0+ nightly.

4. Wire the local endpoint into coding agents & MCP servers

Once your local server responds on http://127.0.0.1:8000/v1 (vLLM), http://127.0.0.1:8080/v1 (llama-server), or http://127.0.0.1:11434/v1 (Ollama), point your coding agent's OpenAI-compatible provider configuration to that base URL with a placeholder API key such as "local".

Because Inkling 2 Mini scored 82.7% on MCP Atlas (Public), it pairs well with scoped local MCP servers (filesystem, git, test runner, or database read-only tools). Keep tool declarations lean—large MCP tool schemas consume context across all 48 looped KV-cache layers.

  • OpenCode / Cline / Roo Code: select "OpenAI Compatible", set Base URL to http://127.0.0.1:8000/v1, and set Model ID to thinkingmachines/Inkling-2-Mini (or your GGUF/Ollama tag).
  • Terminal coding harnesses (Codex CLI / Claude Code via OpenAI-compatible proxy): export OPENAI_BASE_URL=http://127.0.0.1:8000/v1 and OPENAI_API_KEY=local, and verify tool-calling JSON formatting on a read-only repo query first.
  • MCP hygiene: expose only the servers needed for the active repository and review approval prompts before letting a local model run shell or write commands.
Smoke-test local OpenAI-compatible endpoint before connecting an agent
curl -sS http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-2-Mini",
    "messages": [{"role": "user", "content": "Return a JSON object with key status set to ok."}]
  }'

Replace port 8000 with 8080 for llama-server or 11434 for Ollama. Confirm valid JSON output and acceptable tokens/sec before enabling write tools.

5. Verification checklist & common failure modes

Before running autonomous multi-file refactors on a local Inkling 2 Mini setup, run a 2-minute sanity check covering model loading, tool-call parsing, and memory pressure. Most day-one issues trace back to outdated runtimes or overly aggressive context lengths.

  • Corrupted or repetitive output on llama.cpp: your binary predates looped-layer support; pull the latest Unsloth / llama.cpp build.
  • macOS memory pressure or sudden slowdown on 32GB Mac: lower -c from 64K to 32K, ensure -ctk q8_0 -ctv q8_0 is active, and close heavy browser tabs or Docker VMs.
  • Tool calls printed as raw text instead of structured calls: verify --reasoning-parser inkling is enabled on vLLM, or switch your coding agent to explicit tool-call mode and test a single read-only MCP tool first.
  • Factual drift on unfamiliar APIs: Inkling 2 Mini excels at multi-step code reasoning but scores lower than Qwen3.8-27B on AA-Omniscience non-hallucination (63.4% vs 71.4%), so attach a local docs/fetch MCP server rather than relying on memorized API signatures.

Frequently asked questions

Can a 32GB Apple Silicon Mac run Inkling 2 Mini with a coding agent?

Yes. The ~20GB mixed NVFP4/MXFP8 checkpoint (and Unsloth GGUF builds) fits inside 32GB unified memory. Keep context to 32K–64K with quantized KV cache (-ctk q8_0 -ctv q8_0) so the 48 looped KV-cache layers stay out of swap.

Which vLLM command enables reasoning and speculative decoding for Inkling 2 Mini?

Run the official vllm/vllm-openai:inkling-2-mini image with --tensor-parallel-size 1 --reasoning-parser inkling and --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'.

Why does standard llama.cpp output gibberish on Inkling 2 Mini?

Inkling 2 Mini loops its middle 12 layers three times per token with gated cross-pass messages. Older llama.cpp builds do not implement that loop; update to a build with explicit Inkling 2 Mini support.

How well does Inkling 2 Mini handle MCP tools and coding harnesses?

In Thinking Machines' published evaluations (November 20, 2026), Inkling 2 Mini scored 82.7% on MCP Atlas (Public), 79.2% on SWEBench Verified, and 77.9% on SWEBench Pro v2 Hard.

Related guides