How to Deploy a Fine-Tuned Small Language Model
Choose from your hardest constraint, not your preference. A managed endpoint if the model only has to be reachable, llama.cpp if it has to run on a device, vLLM if it has to serve concurrent traffic, and a plain weights download if the model can never leave your network.
Which deployment path fits your constraint?
Work down this table until a row describes something you can’t negotiate, and take that row.
| Binding constraint | Path | How you start it |
|---|---|---|
| You just want to try the model against real inputs | distil labs managed endpoint | distil deployment create-from-slm <slm-id> |
| It must run on a laptop, a handheld, or an embedded board | llama.cpp | distil slm download <slm-id>, then llama-server |
| It must serve concurrent requests behind a service | vLLM on your own GPU | vllm serve model --api-key EMPTY |
| Weights must stay inside your network | Download and self-host | distil slm download <slm-id> |
| You want it in a local development loop with model management | Ollama | ollama create from the GGUF |
The commands come from the inference and local deployment docs. Each path has its own recipe: llama.cpp, vLLM, Ollama, and CPU-only.
What does the managed endpoint actually give you?
A hosted, OpenAI-compatible URL and an API key, provisioned by one command, with no infrastructure on your side.
distil deployment create-from-slm <slm-id> starts one and hands back a deployment id. distil deployment endpoint <deployment-id> then returns the URL and the API key, and distil slm download-metadata --destination ./model <slm-id> fetches the matching client script without the weights. When you’re done, distil deployment delete <deployment-id> releases it, because a deployment bills until its idle timeout.
These deployments are for testing, not production, and the docs say so plainly. Production hosting is a separate conversation with distil labs. Treat the playground as the fastest way to find out whether the model is good, not as the thing you point traffic at.
What does self-hosting require?
A GPU or a CPU with enough memory, a runtime, and someone who owns the process when it falls over.
distil slm download --destination ./model <slm-id> writes model.tar and config.yaml. The tarball extracts to model/, a model-adapter/ when LoRA was used, model_client.py, and a README. The model/ directory is what you hand to vLLM. Models can also be pushed to a private Hugging Face repository through the API, which produces two repos, one in GGUF and one in safetensors, so you can pull whichever format your runtime wants.
Pick the runtime from the workload shape. vLLM exists for throughput and concurrency; llama.cpp exists to run quantized models with minimal setup on whatever hardware is in front of you. Picking between them is covered in self-hosted vs managed inference and, at the runtime level, in Q4 vs Q8 vs FP16.
What goes wrong most often?
Changing the prompt format. This is the one that costs teams the most time. Both deployment docs warn about it explicitly: a fine-tuned SLM expects exactly the message formatting and system prompt it saw during training. Swap in your own system prompt and accuracy drops sharply, with no error to tell you why. Use the client script the platform hands you, and if you must reimplement it, reimplement it byte for byte. Open-book question-answering models are the sharpest case. The context has to be wrapped in a <context> tag followed by a newline inside the first user message.
Sizing for weights and forgetting the KV cache. A model that loads happily at a 2K context won’t necessarily survive at 32K. The working is in how much VRAM does a 1B, 3B or 8B model need.
Benchmarking latency in the wrong regime. A single-stream local number and a p50 under saturation are different measurements that people compare as if they were the same. What latency can you expect from an SLM separates them.
Quantizing before you have a baseline. Quantize after you know the FP16 score on your own test set, never before, or you won’t know which of the two changes moved the number.
Deploying a size you never validated. Deployment doesn’t fix a model that was too small for the task. What size model do you need is the earlier decision.
How do you check the deployment is correct?
Send the same test set through the endpoint that you evaluated on the platform, and compare the score.
This sounds obvious and is skipped constantly. The failure modes above (wrong chat template, wrong system prompt, wrong quantization) all produce a server that responds fine and answers worse. A response with a 200 status isn’t evidence of a working deployment; a matching accuracy number is.
Three checks, in order. Confirm the server is up: llama.cpp exposes GET /health, and both llama.cpp and vLLM expose GET /v1/models. Confirm the format round-trips by running one known example through the provided client. Then run the full test set and compare against the evaluation metrics the platform reported. If the numbers diverge, the deployment is wrong, not the model.
For environments with no network at all, the constraints change again. See air-gapped and on-premise LLM deployment.