Open source · pure Python · MIT

Open-source tools for running LLMs in production.

vitalops plan works out whether a model fits on a GPU, how fast it will run, how many GPUs it needs and what it will cost. vitalops bench measures a running server and compares it with the plan.

Read the docs Try the planner

$pip install vitalops

$ vitalops plan meta-llama/Llama-3.3-70B-Instruct --gpu h100 --users 64
vitalops plan · Llama-3.3-70B on H100 SXM
─────────────────────────────────────────
  model      70.6B params · 80 layers · KV 160 KB/token (fp8)
  weights    fp8 65.8 GiB · matmuls at 1,979 TFLOPS fp8
  traffic    64 concurrent requests × (2,048 in + 300 out) · TTFT target 500 ms

  layout     TP2 × 1 replica = 2 GPUs · batch 64 per replica (max 218)
  decode     20.4 tok/s per stream · 1,304 tok/s aggregate · 4.35 requests/s
  TTFT       155 ms = prefill 135 ms + one step 20.3 ms · target met
  step       20.3 ms = weights 14.0 ms + KV 4.59 ms + all-reduces 1.65 ms (8%)
  memory/GPU weights 32.9 GiB · KV 11.5 GiB · runtime 7.90 GiB · free 27.8 GiB
  cost       $1.07 per 1M output tokens · $3,648 / month at $2.50 per GPU-hour

$ vitalops plan qwen3-32b --all-gpus -q
vitalops plan · Qwen3-32B · fp8 weights, fp8 KV · 64 × (2,048 in + 300 out)
  GPU             GPUs  TP  tok/s/user  aggregate     TTFT  $/1M out
  ────────────────────────────────────────────────────────────────────
  H100 SXM           1   1        22.5      1,437   134 ms     $0.48
  H200 SXM           1   1        26.1      1,668   127 ms     $0.58
  B200               1   1        52.2      3,341  58.3 ms     $0.46
  A100 SXM 80 GB     2   2        10.3        659   387 ms     $1.35
  L40S               2   1        10.3        657   370 ms     $0.85
  RTX 5090           2   2        10.7        687   348 ms     $0.57
  MI300X             1   1        32.1      2,054  98.5 ms     $0.34

$ vitalops bench http://localhost:8000 --users 1,8,32 --slo 80 --plan h100
vitalops bench · mock/llama-tiny at http://localhost:8000
  users  requests  req/s  tok/s all  tok/s/user  TTFT p50  TTFT p95  ITL p50
  ───────────────────────────────────────────────────────────────────────────
      1  16         1.69        216       227.0     33 ms     34 ms   4.4 ms
         planned    2.34        300       300.1     17 ms             3.2 ms
      8  16         9.48      1,213       162.4     47 ms     48 ms   6.1 ms
         planned   14.57      1,865       233.2     17 ms             3.4 ms
     32  64        15.47      1,980        91.3     95 ms     96 ms  12.1 ms
         planned   33.04      4,229       132.2     18 ms             4.1 ms

  capacity   8 concurrent requests meet the 80 ms p95 TTFT target
  gap        At 32 requests TTFT is 5.3× the estimate: requests wait
             behind other prefills. The planner's TTFT has no queueing.

0dependencies outside the Python standard library

AnyHugging Face model, read from its config.json; 21 built in for offline use

8GPUs with datasheet figures, from Jetson AGX Orin to B200 and MI300X

1 file190 KB, for machines where you would rather not pip install

Commands

One command for each step of running a model

Size the deployment before you rent anything, check the server once it runs, and check that a cheaper format still gives the same answers. Each command works on its own, and they share one catalogue of models and GPUs.

01vitalops plan

Memory, speed, GPU count and cost for any model on any GPU, from public specifications.

How it plans

02vitalops bench

Load any OpenAI-compatible server at several concurrency levels and see where it stops meeting your latency target.

How it measures

03vitalops ui

The planner in a browser, served from the installed package. The same panel runs on this page.

Open the planner

04vitalops quantcheck

Compare two endpoints token by token, such as bf16 and fp8 builds of one model, to see where quantising changes the output.

Follow on GitHub

vitalops plan

Will it fit, how fast will it run, and what will it cost

Give it a model and a GPU. It reads the model's shapes from the Hugging Face Hub, takes the GPU from a catalogue of datasheet figures, and finds the fewest GPUs that meet your time-to-first-token target.

Any repository id works, including mixtures of experts and latent attention; gated models read with HF_TOKEN.
Tries tensor parallelism 1, 2, 4 and 8 over the GPU's real link, and as many replicas as the KV cache needs.
--all-gpus compares the whole catalogue at once, priced per million output tokens.
Prints every assumption, and exits 1 when the model doesn't fit, so a script can check it.

Command reference Python API

Qwen3-32B for 64 concurrent users, by GPUvitalops plan qwen3-32b --all-gpus

GPU	GPUs	tok/s per user	TTFT	$ per 1M output tokens
MI300X	1	32.1	98.5 ms	$0.34
B200	1	52.2	58.3 ms	$0.46
H100 SXM	1	22.5	134 ms	$0.48
RTX 5090	2	10.7	348 ms	$0.57
H200 SXM	1	26.1	127 ms	$0.58
L40S	2	10.3	370 ms	$0.85
A100 SXM 80 GB	2	10.3	387 ms	$1.35

fp8 weights and KV cache, 2,048 tokens in and 300 out, at each GPU's default hourly price. Estimates from public specifications, not measurements.

Where one output token's time goes: Llama-3.3-70B, two H100s, 64 users49.0 ms per token = 20.4 tok/s per stream

weights 14.0 msKV 4.6prefill share 28.7 ms

Weights read

Decode is memory-bound: every step reads the weights, or for a mixture of experts the experts the batch touches.

weight bytes ÷ (bandwidth × 0.75)

KV cache read

Each running request's cache is read every step, so it grows with the batch and the context.

batch × context × KV bytes ÷ (bandwidth × 0.75)

All-reduces

Tensor parallelism adds two all-reduces per layer over the GPU's link. Cheap on NVLink, expensive on PCIe.

2 × layers × (bytes ÷ link + latency)

Prefill share

Prompts are processed between decode steps, so long prompts slow every stream down.

batch × prefill time ÷ output tokens

Not in the estimate: queueing, prefix caching, speculative decoding, chunked prefill and real kernel efficiency at small batches. vitalops bench measures what those do on your server. Every formula.

Live planner

The planner, running the Python package in your browser

Every figure in this panel comes from vitalops's Python source, loaded into Pyodide (about 7 MB on the first visit). Pick a GPU from the photos, change the traffic, or load any public Hugging Face model from the end of the model list. "Command line" shows the command that prints the same report.

First-order estimates from public specifications, not measurements. Every formula is under "How this is calculated" in the panel. vitalops ui serves the same panel from the installed package.

vitalops bench

Measure the server, at the load you will run it at

Point it at any OpenAI-compatible endpoint: vLLM, SGLang, TGI, llama.cpp or a hosted API. It runs each concurrency level as a closed loop of streaming requests and reports what users would see.

Time to first token and time between tokens at p50 and p95, tokens per second per stream and in total.
Prompts are sized with the server's own token counts and differ per request, unless --shared-prefix asks for a common system prompt.
--plan h100 prints the estimate under each measured row and names the likely cause of a large gap.
--require 32 exits 1 when the target is missed at 32 users, to stop a deploy in CI.

How it measures

waiting and prefilla streamed tokenwarm-up, not countedmeasured request

TTFT far above the planQueueing

Requests wait behind other prompts' prefill. The plan's TTFT is one prompt on an idle replica, so this gap grows with concurrency.

one stream far below the planSmall-batch overheads

At one request, kernel launches and scheduling cost more than the memory reads the plan counts.

TTFT below the plan, shared prefixPrefix cache

The shared part of the prompt is served from cache instead of being prefilled again, which the plan does not model.

Install

Python 3.9 or newer, nothing else

No compiled extensions and no dependencies, so it installs anywhere Python runs: a laptop, a CI runner or the GPU box itself.

pip

The vitalops command and the Python package.

pip install vitalops
vitalops plan qwen3-32b --gpu h100

One file

The whole package in a single script, the planner page included.

wget https://github.com/vitalops/vitalops/releases/download/onefile/vitalops.py
python vitalops.py plan qwen3-32b

For coding agents

The API and the commands on one page, for an agent's context.

curl https://vitalops.ai/oss/agent-prompt.txt

Read the docs GitHub llms.txt