Book a review
AI inference engineering

Faster, cheaper inference for open-weight models

We help teams plan, tune and run open-weight models on their own hardware, from GPU clusters to Jetson boards. Every change is tested against your evals before it ships.

Book an inference reviewTry the live demos
Optimize · the runs list · a run ships to production as the clip plays01 / 02
3.3×throughput on the same GPUs in a sample run
−69%lower cost per million tokens
−55%lower p99 time to first token
84.1accuracy, kept above an 83.5 threshold

Hardware and software we support

NVIDIAAMDApple siliconQualcommArmRaspberry PiPyTorchHugging FaceKubernetesLlamaQwenDeepSeekMistralGemma
How it fits together

From buying hardware to running in production

Click a stage to see what it produces and which products are involved.

Choose the hardware

Plan simulates your traffic on each GPU type you're considering and prices the result. You can compare options on latency and cost per million tokens before committing to a purchase or a cloud contract.

Deliverables

  • GPU type and count per SLO
  • Cost per million tokens by option
  • Buy-versus-rent break-even
  • Headroom for next year's traffic

Results

  • −38%hardware spend against the first quote
  • ±8%forecast against measured throughput

Products

PlanCapacity and hardware planningSimKernel work without a GPU

This works the same way for data-centre GPUs and for edge devices: GPU clouds and data centres devices in the field

Products

What we build

Each product can be used on its own. Used together, they share one configuration format, so the setup Plan sizes for is the one Optimize tunes and Serve runs.

PlanEstimates throughput, latency and cost for each GPU option, using your traffic.OptimizeFinds the fastest engine settings, quantisation and kernels that still pass your evals.SimSimulates NVIDIA GPUs on a CPU, so you can benchmark kernels in CI without a GPU.ServeAn OpenAI-compatible model server with canary rollouts and inference metrics.EdgeCompresses models to fit phones, Jetsons and Pis, and fine-tunes them on the device.FleetSchedules agent workloads across your GPUs and reuses shared prompt prefixes.

Where the work happens

Summarise this contract.
Supplier may raise prices yearly, capped at 4%…

Application

Capture the real traffic shape, the eval set and the SLO we optimise to.

VitalOps works here
vLLMSGLangTRT-LLM
fp8 kv cache
spec decode
prefix cache
tok/s1,8403,1205,4606,120
bf16fp8TP 22×H100

01Serving engine

Configure the engine: quantised weights and KV cache, TP/PP, scheduler flags.

02Scheduling

Tune how requests share the GPU: batching, chunked prefill, prefix caching.

03Compile and fuse

Fuse ops, remove copies, capture CUDA graphs, compile the decode step.

04Kernels

Profile, then rewrite the hot kernels for your shapes; verify vs the stock path.

rmsnorm.ptxrmsnorm.sasssteered
1
ld.global.u16 r4,[rd2]LDG.E.U16LDG.E.12831%22%
2
bar.sync 0BAR.SYNCSHFL.BFLY11%7.5%
3
st.global.b32 [rd6],r20STG.E.32STG.E.6414%8.5%

05SASS verification

Inspect the SASS ptxas emits: registers, spills, stalls. Steer the compiler.

Silicon

Measure every candidate on the target GPU and report what it did.

Most tuning happens at the serving-engine level. We work on all five layers and check the results in the SASS that actually runs on the GPU.

Walkthrough

01 · Optimize

Tune a model for a specific GPU and latency target

You give Optimize a model, a target GPU, a latency target and an eval set. It tries different engine settings, quantisation, parallelism, scheduling, speculative decoding and custom kernels, ranks the candidates in a simulator, then benchmarks the best ones on real GPUs. Any candidate that scores below your accuracy threshold is dropped. Kernels are checked numerically and at the PTX and SASS level. The result is saved as a bundle you can rerun.

What changes

  • One run and a report instead of weeks of manual tuning.
  • Each speed-up is reported with its accuracy score.
  • Anyone on the team can rerun the bundle and get the same result.

Works with

vLLM, SGLang, TensorRT-LLM, llama.cpp, Hugging Face Transformers; lm-eval or your own evals. Targets: tokens/s, TTFT, TPOT, p99, memory, cost per million tokens.

02 · Optimize · Kernels

Compare kernels without a GPU

Optimize benchmarks kernel candidates on real GPUs when they're available. When they aren't, it can use Sim, our GPU simulator. A replacement libcuda.so takes the same binary, loads the cubin into a SASS interpreter and executes the launches on CPU threads under a timing model, which gives you an estimated time per GPU, stall counts per instruction and a ranking of candidates, on a laptop or in CI. There are three levels of detail: static analysis (milliseconds), sampled simulation (seconds) and cycle-level simulation (hours). Each run reports how well its ranking matches real hardware, and the top candidates are tested on a real GPU before anything ships.

Why it helps

  • Kernel work doesn't have to wait for a free GPU.
  • Agents can compare thousands of variants on ordinary CPUs.
  • A slower kernel can fail CI before it reaches real hardware.

Works with

CUDA C++, Triton, CUTLASS, PyTorch through a driver shim; KernelBench and TritonBench; GitHub Actions. Targets: H100, H200, B200, L40S, RTX 4090 and 5090, Jetson Orin and Thor.

03 · Serve

Roll out a new configuration gradually

Serve is an OpenAI-compatible server that runs the bundles Optimize produces. It supports multiple models per pool, continuous batching, prefix caching, chunked prefill, speculative decoding and structured output. New releases go out as canaries and can be rolled back immediately. It runs in your cloud account or data centre, including air-gapped setups.

What you get

  • Production runs the same configuration that was benchmarked.
  • A rollback takes seconds.
  • All your models sit behind one endpoint.

Works with

OpenAI SDKs, LangChain and LlamaIndex, Kubernetes, NVIDIA and AMD GPUs. /v1/chat/completions, /v1/embeddings, tool calls, JSON-schema and grammar-constrained decoding.

04 · Serve · Monitoring

Monitor latency, cost and quality in production

General-purpose monitoring tools track CPU and memory. Serve tracks time to first token, time per output token, tokens per second, KV-cache hit rate, queue depth, cost per million tokens and output quality, broken down by model, pool, tenant and prompt template. When a metric gets worse, it can pass a suggested fix to Optimize. It also works with existing vLLM, SGLang and TensorRT-LLM deployments.

What it shows

  • See what a prompt change costs within an hour of shipping it.
  • Sampled evals flag drops in quality.
  • Alerts include a suggested fix.

Works with

Prometheus, OpenTelemetry, Grafana, PagerDuty and Slack; existing vLLM, SGLang and TensorRT-LLM deployments.

05 · Edge

Fit models onto devices and fine-tune them there

Edge picks the largest model that stays above your accuracy threshold within each device's memory and latency limits. A Jetson or a flagship phone takes a 3B vision-language model once the weights, KV cache and encoder are quantised and a couple of layers pruned; a Raspberry Pi gets a 0.5 to 1.5B model and the same accuracy guard. If the data can't leave the device, fine-tuning can run there with QLoRA, and the resulting adapter is pushed to every device using the same base model.

What it allows

  • Models that didn't fit before can run on a phone, a Pi or a Jetson.
  • Accuracy is checked after each compression step.
  • Training data stays on the device.

Works with

NVIDIA Jetson, iPhone and Apple silicon, Pixel and Android NPUs, Qualcomm Snapdragon and Dragonwing, Raspberry Pi 5, x86 CPUs. Exports to GGUF, MLX, ExecuTorch, CoreML, QNN and TensorRT.

06 · Fleet

Run many agents on your own GPUs

Agent traffic is different from chat. Agents share long prompts, start sub-agents in bursts and sit idle while tools run. Fleet schedules agents across your GPUs, sends requests to GPUs that already have their prompt cached, and frees an agent's slot while it waits on a tool. Going from 8 agents to 96 takes one command, and a capacity check estimates how far you can scale before latency suffers.

What you get

  • More agents per GPU, because cached prompts are reused.
  • Scaling is a single setting.
  • A clear estimate of how many agents your cluster can handle.

Works with

Claude Agent SDK, OpenAI Agents, LangGraph, or your own loop; Kubernetes and Slurm; air-gapped by default.

Live demos

Interactive demos

Six estimators built on the first-order models our products start from. The products then measure the real numbers on your hardware.

Capacity plannerGPUs needed, latency and cost per million tokens for a model, a GPU and a traffic level.OpenBatching simulatorA simulated model server under static, continuous and chunked-prefill scheduling.OpenQuantisation labFP8, INT8, INT4 and NVFP4 rounding error on a layer with outlier channels.OpenSpeculative decodingDraft-and-verify decoding against the plain decoder, with the expected speed-up per draft length.OpenEdge fitWhether a model fits the memory of a Jetson, a phone or a Raspberry Pi, and at what speed.OpenKernel labOccupancy, register pressure and achievable bandwidth, per GPU architecture.Open
Open source

The vitalops command line

Pure Python, no dependencies, MIT. vitalops plan reads any model's shapes from the Hugging Face Hub, applies the same first-order physics as the planner demo, and prints the layout, the speed and the cost. vitalops bench measures a running OpenAI-compatible server at the concurrency you care about and prints the plan's estimate next to each measurement.

GitHubDocs and live planner

$ pip install vitalops
$ vitalops plan meta-llama/Llama-3.3-70B-Instruct --gpu h100 --users 64
  layout     TP2 × 1 replica = 2 GPUs · batch 64 per replica (max 218)
  decode     20.4 tok/s per stream · 1,304 tok/s aggregate · 4.35 requests/s
  TTFT       155 ms = prefill 135 ms + one step 20.3 ms · target met
  step       20.3 ms = weights 14.0 ms + KV 4.59 ms + all-reduces 1.65 ms (8%)
  cost       $1.07 per 1M output tokens · $3,648 / month at $2.50 per GPU-hour
Solutions

Where we work

LLM products and APIsChat, copilots and model APIsAI agentsLong prefixes, sub-agents, tool callsVision and multimodalVLMs, documents, speechRobotics and devicesModels that run in the fieldGPU cloudsMore tokens per GPU you rent outRegulated and on-premAir-gapped, in your data centre
Services

Engineering services

We built the products from client work, and we still take on projects at a fixed scope and price for teams who'd rather have us do it.

Inference optimisation sprint

You give us the model, the hardware and the latency target. Two weeks later you get a tested configuration and a report. Fixed fee.

vLLM · SGLang · TensorRT-LLM · FP8 · INT4 · NVFP4 · speculative decoding · prefix caching

Custom kernels

Fused and specialised kernels for the cases where the standard ones are slow, tuned and checked at the SASS level.

Triton · CUDA · CUTLASS · wgmma · TMA · cp.async · Hopper · Blackwell · Ada

Speculative decoding

Draft models trained on your own traffic, so the speed-up holds up on your real prompts.

EAGLE-3 · Medusa · n-gram and prompt-lookup drafts · acceptance tuning

Quantisation and distillation

Smaller models that keep the accuracy you approved, with eval results to show it.

AWQ · GPTQ · SmoothQuant · KV-cache quantisation · distillation · pruning

Edge deployment and on-device training

Getting a model to fit on your device, run well, take updates and fine-tune in the field.

Jetson · Apple silicon · Qualcomm · Raspberry Pi · ExecuTorch · GGUF · MLX · QLoRA on device

Serving for agent workloads

Serving setups designed for long shared prompts, bursts of sub-agents and pauses during tool calls.

KV-aware routing · prefix sharing · capacity planning

Benchmarking and capacity planning

Tokens per second per dollar on different GPUs and clouds, measured on your workload.

H100 · H200 · B200 · MI300X · L40S · consumer cards · buy-versus-rent models

Training throughput

Speeding up training runs: data loading, parallelism, kernels and memory use.

FSDP · DeepSpeed · Megatron · torch.compile · fused optimisers · MFU audits

All services

Clients

“VitalOps helped us optimise our data pipelines for the finance databases, which directly translated into a smoother and more intuitive experience for our users.”

ChatfinCEO, Chatfin · AI for finance teams

“The team is highly responsive, pragmatic, and product-minded, and the best part is that they understand what it takes to operationalize data, not just analyze it.”

StraightopsCEO, Straightops · Source-to-Pay automation
Questions

What engineering teams ask us

Do we have to send you our weights or data?

No. Everything runs in your cloud account or data centre, and evals run on your data where it is. During sprints our engineers work inside your environment.

Which models can you speed up?

Open-weight models and your own fine-tunes, including Llama, Qwen, DeepSeek, Mistral, Gemma, GLM, Whisper, vision-language models, and embedding and reranker models. We can't speed up closed API models, but we can help with routing, caching and batching around them.

How do you make sure the faster model is still correct?

You set a minimum score on your eval set. Every candidate configuration is tested against it over several seeds and rejected if it falls short, however fast it is. Custom kernels are compared numerically with the reference and inspected at the SASS level.

Can we start with one product?

Yes. Most teams start with an Optimize sprint on one model, or use Plan before buying hardware. Bundles can also be exported to standard vLLM, SGLang and TensorRT-LLM.

Do you support AMD or non-NVIDIA hardware?

We support AMD MI300-class GPUs through ROCm, writing kernels in Triton where we can. Edge supports Apple, Qualcomm, Arm and Hailo devices. Sim only models NVIDIA GPUs for now.

Get started

Tell us what you need the model to do

Send us the model, the hardware and your target, whether that's tokens per second, latency, memory or cost. In a 30-minute call we'll tell you whether we think it's doable and roughly what it would take.

Book an inference reviewhello@vitalops.ai
  1. ReviewA free 30-minute call. We look at your model, hardware and target and give you an honest estimate.
  2. SprintTwo weeks at a fixed fee, on your hardware. If we can't beat your current setup, you don't pay.
  3. RunWe run Serve and monitoring for you and keep re-tuning as traffic changes, under an agreed SLO.