AI & MACHINE LEARNING · MIT BENCHMARKING SKILL

Inference Optimisation Skill

sybil-solutions/inference-optimisation-skill (MIT, commit fda968e, 964 GitHub stars / 85 forks) is a hardware-agnostic hill-climbing procedure (skills/inference-optimisation/SKILL.md + PROMPT.md) for tuning local LLM servers across vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, and TensorRT-LLM. Rather than throwing random flags at a server, it computes theoretical memory-bandwidth and FLOP ceilings first (scripts/ceiling.py), changes one variable at a time in serialized control / candidate / control runs (scripts/queue.sh, bench.py), tracks real driver residency (scripts/mem.sh), and gates every candidate on needle retrieval and full-KV concurrency.

Explore ceiling loop, harness & setup ↓

1. Ceiling-first hill-climbing loop & benchmarking harness

Pinned at commit fda968e, the skill enforces disciplined single-lever optimization so thermal throttling and chunk-counting noise do not masquerade as wins:

  • Ceilings before tuning (scripts/ceiling.py & references/ceilings.md): Calculates theoretical decode, prefill, TTFT, startup, and KV-cache bounds from measured memory bandwidth, compute FLOPs, and tensor byte sizes.
  • Serialized interleaved A/B (scripts/queue.sh & scripts/bench.py): Queues base -> candidate -> base trials, measures real server usage token counts (never streamed chunk counts), logs results into ledger.jsonl, and flags battery runs, request errors, or degenerate looping.
  • Unified & discrete memory watchdog (scripts/mem.sh): Captures real container and GPU driver memory residency under unified memory where container cgroups underreport.
  • Five technical references (references/): ceilings.md, methodology.md, levers.md (ordered by payoff, tagged general vs. engine-specific), gotchas.md (vLLM, SGLang, llama.cpp, ExLlama; NVIDIA, Intel, Apple, AMD traps), and publishing.md (six acceptance gates and clean pinned builds).

2. Install in Claude Code or run standalone without an agent

1. Install into Claude Code user skills (~/.claude/skills/):

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Other agents can read skills/inference-optimisation/SKILL.md or paste PROMPT.md directly.

2. Quick standalone run against an existing OpenAI-compatible server:

S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json
python3 $S/scripts/bench.py run configs/base.json --long 131072
bash $S/scripts/queue.sh base cand base
python3 $S/scripts/bench.py show

Pair long-running server tuning loops with observability helpers like Claude Platform OpenTelemetry when tracking multi-hour agent runs.

3. Long-context acceptance gates & reporting boundaries

  • Stability over peak short-prompt tok/s: Every candidate configuration must survive exact-match needle retrieval and a concurrent full-KV cache soak under your target memory cap before promotion.
  • Speculative decoding tradeoffs: references/levers.md emphasizes measuring per-step draft cost, draft head vocabulary pruning, draft length sweeps per concurrency/context, and keeping sampling on GPU rather than chasing raw acceptance rate.
  • Hardware specificity: Author-reported upstream speedups (for example on Intel Arc B390 iGPU, Arc Pro B70, RTX 3090, and 4x H200 NVL) depend on specific quantizations (EXL3/GGUF) and custom kernels; treat them as reference ceilings and measure locally with interleaved controls.

Frequently asked questions

Which local LLM engines does inference-optimisation-skill support?

The skill and reference guides cover vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, and TensorRT-LLM, and bench.py benchmarks any local server exposing an OpenAI-compatible HTTP endpoint.

Why does inference-optimisation-skill interleave control/candidate/control runs?

Upstream measurement guidance notes +/-5-10% run-to-run drift from thermal state and background load. Running base -> candidate -> base in queue.sh with medians and usage-token accounting prevents noise from looking like a speedup.

Can I use inference-optimisation-skill without Claude Code?

Yes. You can pass PROMPT.md to any coding agent, or invoke scripts/ceiling.py, scripts/bench.py, scripts/queue.sh, and scripts/mem.sh directly from your shell using Python 3 standard library.

Primary sources