Interactive inference demos

The capacity planner is also a command-line tool and a Python library, vitalops: pip install vitalops, then vitalops plan meta-llama/Llama-3.3-70B-Instruct --gpu h100 --users 64. It reads any model's shapes from the Hugging Face Hub and applies the same physics, and vitalops bench measures a running server against it.

• Capacity planner

  Estimates how many GPUs you need and the cost per million tokens. It assumes decode is limited by memory bandwidth and prefill by compute, which is where Plan starts before running a full simulation.

• Batching simulator

  Simulates a model server using static batching, continuous batching, chunked prefill and prefix caching on chat, RAG and agent traffic.

• Kernel lab

  Occupancy from the CUDA occupancy-calculator rules for each architecture, the register file and shared memory drawn to scale, spills under a register cap, achievable bandwidth from Little's law and a roofline for the chosen kernel.

• Quantisation lab

  Applies real FP8, INT8, INT4 and NVFP4 rounding to a synthetic layer with outlier channels, similar to real LLM weights, and shows the effect of group size and outlier protection.

• Speculative decoding

  A draft model proposes several tokens and the main model checks them in one pass. The measured speed-up is shown against the closed-form expectation for each draft length.

• Edge fit

  Adds up the weights, KV cache, vision encoder and runtime overhead and compares them with the free memory on a Jetson, a phone or a Raspberry Pi, with an optional on-device fine-tune.