1. Hardware budget & why 32GB unified memory is the practical floor
Inkling 2 Mini (released November 20, 2026 by Thinking Machines Lab under Apache 2.0) packs 26.4B total parameters into a ~20GB mixed NVFP4/MXFP8 checkpoint, activating 3.4B parameters per pass (5.4B effective active across its 3-pass looped core). Because layers 6–17 loop three times with separate per-pass KV caches, the runtime maintains 48 KV-cache layers even though only 24 physical layers sit in RAM.
That design makes 32GB unified memory (Apple Silicon MacBook Pro, Mac mini, or Mac Studio) or a single 32GB+ VRAM GPU / NVIDIA DGX Spark the practical local baseline. The model supports up to 1M tokens of context on larger clusters, but on a 32GB machine you should keep context bounded to 32K–64K tokens with quantized KV cache so macOS or CUDA does not swap.
- Apple Silicon (32GB+ unified memory): run via Ollama, Llama App, Exo Labs, or Unsloth GGUF + looped-layer llama-server.
- NVIDIA DGX Spark (GB10) or single 32GB+ VRAM GPU: run the official vllm/vllm-openai:inkling-2-mini container with native NVFP4/MXFP8 weights and bundled DSpark speculative decoding.
- Published coding & MCP baselines: 79.2% SWEBench Verified (bash only), 77.9% SWEBench Pro v2 Hard, 82.7% MCP Atlas (Public), and 48.2% DeepSWE v1.1.
2. Option A: Serve Inkling 2 Mini on a 32GB Apple Silicon Mac
On macOS, the fastest path is Ollama (when the inkling-2-mini tag is available in your updated Ollama build) or a quantized GGUF build from Unsloth served via llama-server. Always confirm your llama.cpp binary includes looped-layer support for Inkling 2 Mini—older builds will load weights incorrectly or emit broken tokens.
Exposing an OpenAI-compatible HTTP endpoint on localhost:8080 lets any local coding harness (OpenCode, Cline, Roo Code, or custom MCP clients) talk to the model without cloud API keys.
- Start with -c 40000 (or 32768) and quantized KV cache (-ctk q8_0 -ctv q8_0) to stay comfortably inside 32GB unified memory.
- Pass --mmproj only if your coding agent sends screenshots or UI mockups (Inkling 2 Mini accepts 40px–4096px image inputs via its 23-layer ViT).
- If you prefer zero flag tuning on Apple Silicon, Ollama exposes an identical OpenAI-compatible endpoint at http://localhost:11434/v1 after running ollama run inkling-2-mini.
./llama-server \
-m ./models/Inkling-2-Mini-IQ4_NL.gguf \
-c 40000 -ngl 99 --flash-attn on \
-ctk q8_0 -ctv q8_0 \
--host 127.0.0.1 --port 8080Requires an updated llama.cpp / Unsloth build with Inkling 2 Mini looped-layer support. Add --mmproj ./models/mmproj-Inkling-2-Mini-Q8_0.gguf for screenshot/vision inputs.
3. Option B: Serve on NVIDIA 32GB+ GPU or DGX Spark with vLLM
For Linux workstations with 32GB+ VRAM or an NVIDIA DGX Spark, use Thinking Machines and vLLM's official Docker image (vllm/vllm-openai:inkling-2-mini, tracking vLLM 0.35.0+ nightly). This path runs the native ~20GB NVFP4/MXFP8 checkpoint directly and activates the bundled DSpark speculator for faster token generation during long multi-file edits.
Include --reasoning-parser inkling so coding agents receive structured reasoning traces separately from final tool calls and diffs.
- Mount ~/.cache/huggingface so the ~20GB checkpoint persists across container restarts.
- Bind -p 127.0.0.1:8000:8000 instead of 0.0.0.0 when running on a shared LAN so your local inference server is not exposed externally.
- Verify startup by querying http://127.0.0.1:8000/v1/models and confirming thinkingmachines/Inkling-2-Mini is listed.
docker run --gpus all --privileged --ipc=host \
-p 127.0.0.1:8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \
--tensor-parallel-size 1 \
--reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'Source: recipes.vllm.ai/thinkingmachines/Inkling-2-Mini. Requires NVIDIA driver/CUDA stack compatible with vLLM 0.35.0+ nightly.
4. Wire the local endpoint into coding agents & MCP servers
Once your local server responds on http://127.0.0.1:8000/v1 (vLLM), http://127.0.0.1:8080/v1 (llama-server), or http://127.0.0.1:11434/v1 (Ollama), point your coding agent's OpenAI-compatible provider configuration to that base URL with a placeholder API key such as "local".
Because Inkling 2 Mini scored 82.7% on MCP Atlas (Public), it pairs well with scoped local MCP servers (filesystem, git, test runner, or database read-only tools). Keep tool declarations lean—large MCP tool schemas consume context across all 48 looped KV-cache layers.
- OpenCode / Cline / Roo Code: select "OpenAI Compatible", set Base URL to http://127.0.0.1:8000/v1, and set Model ID to thinkingmachines/Inkling-2-Mini (or your GGUF/Ollama tag).
- Terminal coding harnesses (Codex CLI / Claude Code via OpenAI-compatible proxy): export OPENAI_BASE_URL=http://127.0.0.1:8000/v1 and OPENAI_API_KEY=local, and verify tool-calling JSON formatting on a read-only repo query first.
- MCP hygiene: expose only the servers needed for the active repository and review approval prompts before letting a local model run shell or write commands.
curl -sS http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "thinkingmachines/Inkling-2-Mini",
"messages": [{"role": "user", "content": "Return a JSON object with key status set to ok."}]
}'Replace port 8000 with 8080 for llama-server or 11434 for Ollama. Confirm valid JSON output and acceptable tokens/sec before enabling write tools.
5. Verification checklist & common failure modes
Before running autonomous multi-file refactors on a local Inkling 2 Mini setup, run a 2-minute sanity check covering model loading, tool-call parsing, and memory pressure. Most day-one issues trace back to outdated runtimes or overly aggressive context lengths.
- Corrupted or repetitive output on llama.cpp: your binary predates looped-layer support; pull the latest Unsloth / llama.cpp build.
- macOS memory pressure or sudden slowdown on 32GB Mac: lower -c from 64K to 32K, ensure -ctk q8_0 -ctv q8_0 is active, and close heavy browser tabs or Docker VMs.
- Tool calls printed as raw text instead of structured calls: verify --reasoning-parser inkling is enabled on vLLM, or switch your coding agent to explicit tool-call mode and test a single read-only MCP tool first.
- Factual drift on unfamiliar APIs: Inkling 2 Mini excels at multi-step code reasoning but scores lower than Qwen3.8-27B on AA-Omniscience non-hallucination (63.4% vs 71.4%), so attach a local docs/fetch MCP server rather than relying on memorized API signatures.
Frequently asked questions
Can a 32GB Apple Silicon Mac run Inkling 2 Mini with a coding agent?
Yes. The ~20GB mixed NVFP4/MXFP8 checkpoint (and Unsloth GGUF builds) fits inside 32GB unified memory. Keep context to 32K–64K with quantized KV cache (-ctk q8_0 -ctv q8_0) so the 48 looped KV-cache layers stay out of swap.
Which vLLM command enables reasoning and speculative decoding for Inkling 2 Mini?
Run the official vllm/vllm-openai:inkling-2-mini image with --tensor-parallel-size 1 --reasoning-parser inkling and --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'.
Why does standard llama.cpp output gibberish on Inkling 2 Mini?
Inkling 2 Mini loops its middle 12 layers three times per token with gated cross-pass messages. Older llama.cpp builds do not implement that loop; update to a build with explicit Inkling 2 Mini support.
How well does Inkling 2 Mini handle MCP tools and coding harnesses?
In Thinking Machines' published evaluations (November 20, 2026), Inkling 2 Mini scored 82.7% on MCP Atlas (Public), 79.2% on SWEBench Verified, and 77.9% on SWEBench Pro v2 Hard.