Serve is an OpenAI-compatible model server that runs Optimize bundles. It supports several models per pool, continuous batching, prefix caching, chunked prefill, speculative decoding and structured output, along with canary rollouts and monitoring. It runs in your cloud account or data centre.
Deploy a bundle to a pool, send it 10% of traffic, and compare latency, cost and errors with the current release. Then promote it or roll it back.
Serve tracks time to first token, time per output token, tokens per second, KV-cache hit rate, queue depth, cost per million tokens and output quality. You can break these down by model, pool, tenant and prompt template. When a metric gets worse, Serve suggests a fix and can pass it to Optimize.
A simulated model server under static batching, continuous batching, chunked prefill and prefix caching, with time to first token and inter-token latency measured on the same requests.
Supports /v1/chat/completions, /v1/embeddings, tool calls, and JSON-schema and grammar-constrained output. Existing SDKs only need a new base URL.
Canaries, traffic splits, shadow traffic and rollback, with a side-by-side comparison against the previous release.
Exports to Prometheus and OpenTelemetry, comes with Grafana dashboards, and sends alerts to PagerDuty and Slack.
OpenAI SDKs, LangChain and LlamaIndex · Kubernetes and Slurm · NVIDIA and AMD GPUs · Prometheus, OpenTelemetry, Grafana, PagerDuty and Slack.
No. Bundles can be exported to standard vLLM, SGLang and TensorRT-LLM. Serve adds rollouts, monitoring and the link back to Optimize.
Yes. It connects to existing vLLM, SGLang and TensorRT-LLM servers through their metrics endpoints and a small sidecar.
In your cloud account or data centre, including air-gapped environments. We don't see your prompts or outputs.
Send us the model, the hardware and your target, whether that's tokens per second, latency, memory or cost. In a 30-minute call we'll tell you whether we think it's doable and roughly what it would take.
