Inference Optimisation Skill
sybil-solutions/inference-optimisation-skill (MIT, commit fda968e, 964 GitHub stars / 85 forks) is a hardware-agnostic hill-climbing procedure (skills/inference-optimisation/SKILL.md + PROMPT.md) for tuning local LLM servers across vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, and TensorRT-LLM. Rather than throwing random flags at a server, it computes theoretical memory-bandwidth and FLOP ceilings first (scripts/ceiling.py), changes one variable at a time in serialized control / candidate / control runs (scripts/queue.sh, bench.py), tracks real driver residency (scripts/mem.sh), and gates every candidate on needle retrieval and full-KV concurrency.
1. Ceiling-first hill-climbing loop & benchmarking harness
Pinned at commit fda968e, the skill enforces disciplined single-lever optimization so thermal throttling and chunk-counting noise do not masquerade as wins:
- Ceilings before tuning (
scripts/ceiling.py&references/ceilings.md): Calculates theoretical decode, prefill, TTFT, startup, and KV-cache bounds from measured memory bandwidth, compute FLOPs, and tensor byte sizes. - Serialized interleaved A/B (
scripts/queue.sh&scripts/bench.py): Queuesbase -> candidate -> basetrials, measures real server usage token counts (never streamed chunk counts), logs results intoledger.jsonl, and flags battery runs, request errors, or degenerate looping. - Unified & discrete memory watchdog (
scripts/mem.sh): Captures real container and GPU driver memory residency under unified memory where container cgroups underreport. - Five technical references (
references/):ceilings.md,methodology.md,levers.md(ordered by payoff, tagged general vs. engine-specific),gotchas.md(vLLM, SGLang, llama.cpp, ExLlama; NVIDIA, Intel, Apple, AMD traps), andpublishing.md(six acceptance gates and clean pinned builds).
2. Install in Claude Code or run standalone without an agent
1. Install into Claude Code user skills (~/.claude/skills/):
git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
Other agents can read skills/inference-optimisation/SKILL.md or paste PROMPT.md directly.
2. Quick standalone run against an existing OpenAI-compatible server:
S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json
python3 $S/scripts/bench.py run configs/base.json --long 131072
bash $S/scripts/queue.sh base cand base
python3 $S/scripts/bench.py show
Pair long-running server tuning loops with observability helpers like Claude Platform OpenTelemetry when tracking multi-hour agent runs.
3. Long-context acceptance gates & reporting boundaries
- Stability over peak short-prompt tok/s: Every candidate configuration must survive exact-match needle retrieval and a concurrent full-KV cache soak under your target memory cap before promotion.
- Speculative decoding tradeoffs:
references/levers.mdemphasizes measuring per-step draft cost, draft head vocabulary pruning, draft length sweeps per concurrency/context, and keeping sampling on GPU rather than chasing raw acceptance rate. - Hardware specificity: Author-reported upstream speedups (for example on Intel Arc B390 iGPU, Arc Pro B70, RTX 3090, and 4x H200 NVL) depend on specific quantizations (EXL3/GGUF) and custom kernels; treat them as reference ceilings and measure locally with interleaved controls.
Frequently asked questions
Which local LLM engines does inference-optimisation-skill support?
The skill and reference guides cover vLLM, SGLang, llama.cpp, ExLlama/TabbyAPI, MLX, and TensorRT-LLM, and bench.py benchmarks any local server exposing an OpenAI-compatible HTTP endpoint.
Why does inference-optimisation-skill interleave control/candidate/control runs?
Upstream measurement guidance notes +/-5-10% run-to-run drift from thermal state and background load. Running base -> candidate -> base in queue.sh with medians and usage-token accounting prevents noise from looking like a speedup.
Can I use inference-optimisation-skill without Claude Code?
Yes. You can pass PROMPT.md to any coding agent, or invoke scripts/ceiling.py, scripts/bench.py, scripts/queue.sh, and scripts/mem.sh directly from your shell using Python 3 standard library.