Serving behind the endpoint

Your student was trained on traffic that an inference endpoint collected. This page puts it in front of that traffic: a hosted deployment serves the model, a new endpoint calls it first and falls back to the production model when it can’t answer, and your application moves over by changing one string.

A hosted deployment is a session for trying the student on live traffic: it stops after six hours, or after an hour with no traffic. To serve the student permanently, email contact@distillabs.ai.

1. Deploy the model

Deploy the trained model. The command prints the new deployment’s id:

distil deployment create-from-slm <slm-id>

Check its status with that id until it reads Success:

distil deployment status <deployment-id>

2. Create an endpoint with the deployment as primary

Create the endpoint once the deployment status reads Success:

distil inference-endpoint create-from-deployment --name support-slm \
  --fallback-model "openai/gpt-4.1-mini" <deployment-id>

The command prints the unique endpoint name on its Endpoint Name line, here support-slm-Qk3bZ1. Link it to support-prod, the key you created in Inference endpoint:

distil inference-endpoint link-api-key support-slm-Qk3bZ1 support-prod

--fallback-model is the model production runs today, so a request the student can’t serve is answered the way it was answered before. The endpoint calls the deployment first and falls back whenever it fails, a timeout included.

An endpoint can’t be edited, so this is a new endpoint, never the one that collected the traces. The old one keeps recording until your application moves, and the key linked to it can be linked to this one too.

Try a few test-set rows before the application moves, with the model’s own client, as on distil labs inference. Download the client into ./model and send a row through it, where <api-key> is the secret of support-prod:

distil slm download-metadata --destination ./model <slm-id>
uv run ./model/model_client.py --base-url https://inference.distillabs.ai/v1 \
  --api-key <api-key> --model support-slm-Qk3bZ1 --conversation '[{"role": "system", "content": "<system prompt>"}, {"role": "user", "content": "Your question here"}]'

Then check which model answered. Every record the endpoint writes names it: metadata.source is primary for the student and fallback for the production model. A fallback answer to these calls usually means the deployment failed or has stopped. Download the records and read metadata.source on the calls you just made:

distil inference-endpoint download-traces support-slm-Qk3bZ1

3. Move the application

Change the model string to the new unique endpoint name. The model must run with a system prompt in every request: the task_description it was trained with, or, with variable_system_prompt on, the system prompt your application chooses for that request. With the flag off, if your production prompt differs from task_description, change the prompt to it (Why the client).

client.chat.completions.create(
    model="support-slm-Qk3bZ1",
    messages=[
        {"role": "system", "content": "<the task_description from the job description>"},
        {"role": "user", "content": "…"},
    ],
)

Any new caller or integration uses the model’s model_client.py instead, which sends exactly the request the model was trained and evaluated with.

The deployment is a session

Once the deployment stops it can’t be restarted. From then on every call goes to the fallback, the endpoint keeps answering, and source reads fallback on every record. Delete the deployment when the student should stop answering: distil deployment delete <deployment-id>.

4. Start the next iteration

The new endpoint records its traffic like the first one did. With a permanent deployment, those records are the student’s traffic; after a session ends they are the fallback’s answers. Either way they are the traces for the next iteration: turn them into a traces object as in Endpoint records to traces, filtering on source when you want only one model’s answers, and continue with the next iteration (Model iterations). Model iterations covers what to change from one iteration to the next.