You give Optimize a model, a target GPU, a latency target and an eval set. It tries changes to the serving engine, quantisation, parallelism, scheduling, speculative decoding and kernels, and only keeps the ones that stay above your accuracy threshold.
The runs list shows each model and GPU with its best verified result, the speed-up over the baseline and whether it stayed above the accuracy threshold. You can send a result to Serve from here.
Each candidate is a specific change, such as fp8 weights and KV cache, chunked prefill, an EAGLE-3 draft model trained on your traffic, or a fused kernel. Each one is benchmarked on the target GPU and evaluated over several seeds. Anything below the accuracy threshold is rejected.
This applies real FP8, INT8, INT4 and NVFP4 rounding to a synthetic layer with outlier channels. Optimize does the same with your weights and then checks the result on your eval set.
Engine choice and settings, weight and KV-cache quantisation, tensor and pipeline parallelism, batching and prefill settings, speculative decoding, CUDA graphs, torch.compile and custom kernels.
Use lm-eval tasks or your own test set and set a minimum score. Accuracy is reported next to every throughput number, and nothing below the minimum ships.
The winning settings, kernels and draft model are packaged with the exact engine version, so Serve runs what was benchmarked.
vLLM, SGLang, TensorRT-LLM, llama.cpp, Hugging Face Transformers · lm-eval or your own evals · targets: tokens/s, TTFT, TPOT, p99, memory, cost per million tokens.
Open-weight models and your own fine-tunes, including Llama, Qwen, DeepSeek, Mistral, Gemma, GLM, Whisper, vision-language models, and embedding and reranker models.
You set a minimum score on your eval set. Every candidate is tested against it over several seeds and rejected if it falls short. Custom kernels are compared numerically with the reference and inspected at the SASS level.
No. Optimize runs in your cloud account or data centre, and evals run where your data already is.
Send us the model, the hardware and your target, whether that's tokens per second, latency, memory or cost. In a 30-minute call we'll tell you whether we think it's doable and roughly what it would take.
