Run a Fine-Tuned SLM with llama.cpp
llama.cpp is the right runtime when the model has to run somewhere you don’t control: a laptop, a handheld, an embedded board, a machine with no Python. It’s a C/C++ binary with no dependencies that loads a single quantized file and serves an OpenAI-compatible API.
When is llama.cpp the right runtime?
When the deployment target is a device rather than a server.
The project’s stated goal is “LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware”, locally and in the cloud, and that’s exactly the property you’re buying. One binary, one .gguf file, no virtualenv, no CUDA install, and backends for CPU, CUDA, Metal, Vulkan, HIP and SYCL from the same source tree. Apple silicon is a first-class target.
What you give up is serving throughput. llama.cpp will handle a request at a time comfortably and concurrency poorly compared to a batching server. If you’re fronting real traffic, vLLM is the other half of this decision. If you want model management and a pull-by-name workflow on top of the same engine, that’s Ollama.
What you need before you start
Three things: your trained model as a GGUF file, the llama-server binary on your PATH, and uv if you want to use the generated client script.
brew install llama.cpp # macOSBinaries are also published on the releases page, and the project builds from source on Linux and Windows. Confirm the install with llama-server --help before going further. Nothing checks it for you, because you start the server yourself.
Step 1: Get the weights and the client
distil slm download --destination ./model <slm-id>That writes model.tar and config.yaml. Extract the tarball and you get the merged weights under model/ as safetensors, along with model_client.py and a README. See the local deployment docs for the full contents.
llama.cpp doesn’t load safetensors, so you need a GGUF. Two routes: convert the extracted weights with llama.cpp’s own convert_hf_to_gguf.py and then llama-quantize, or push the model to a private Hugging Face repository, which produces a GGUF repo alongside the safetensors one and skips the conversion entirely.
Step 2: Start the server with the right flags
llama-server -m ./model.gguf --jinja --port 8000--jinja tells llama-server to apply the chat template embedded in the GGUF’s metadata rather than a built-in default. A fine-tuned SLM was trained against one specific template; render its messages with a different one and the model will answer, badly, with nothing in the logs to indicate why. It’s the one flag you can’t leave off.
The other flags you’re most likely to add, from the server documentation:
| Flag | What it does | When you need it |
|---|---|---|
-ngl, --gpu-layers N |
Offload N layers to the GPU | Metal, CUDA or Vulkan is available |
-t, --threads N |
CPU threads | CPU-only boxes, match physical cores |
-c, --ctx-size N |
Context window in tokens | Long prompts, RAG contexts |
--host HOST |
Bind address | Serving to another machine on the LAN |
Note that llama-server’s own default port is 8080. The 8000 above is deliberate, because it’s the port the generated client expects.
Step 3: Query it
Use the model_client.py that came out of the tarball. It defaults to http://127.0.0.1:8000/v1, so a local server on port 8000 needs no arguments beyond the conversation:
uv run ./model_client.py --conversation '[{"role": "user", "content": "Your question here"}]'For an open-book question-answering model, the retrieved context goes inside a <context> tag followed by a newline, inside the first user message:
uv run ./model_client.py --conversation '[{"role": "user", "content": "<context>Your context here</context>\nYour question here"}]'Use the generated script rather than writing your own request. It encodes the system prompt and message formatting the model was trained on, and the docs are blunt about the consequence of deviating: a different system prompt or formatting gives you poor performance, silently.
Step 4: Verify it is serving your model
Three checks, cheapest first.
curl http://localhost:8000/health
curl http://localhost:8000/v1/models/health returns {"status": "ok"} once the model is loaded, so poll it before you send anything real. /v1/models confirms which model is actually loaded. Then run your held-out test set through the endpoint and compare with the score from teacher evaluation and training metrics. If accuracy is materially below what the platform reported, suspect the template before you suspect the model.
What breaks, and how it looks
The model loads but answers generically. Almost always the chat template. Check --jinja is set.
llama-server exits immediately. Read the output in the terminal it was running in. Out-of-memory on a large context is the usual cause; drop -c.
It’s slower than you expected on a GPU machine. No layers were offloaded. Set -ngl explicitly.
Throughput collapses under concurrent callers. Working as designed. This is the case that wants vLLM, or a managed endpoint. See self-hosted vs managed inference.
If you want to understand the file you’re loading, what is GGUF covers the format and its quantization types.