Vision models
A vision student reads images: it takes a prompt with one or more images and answers in text.
One config key controls it: base.visual_task (default false). With it on, user messages carry
images, the teacher and the judge see them, and the student is a vision model trained, evaluated
and served with them. It works for question-answering only.
When to use it
Section titled “When to use it”Use it when what the model must read is an image: scanned or photographed pages (OCR, handwriting), forms, receipts and invoices, screenshots, photos to describe or classify. When your application already extracts the text reliably, a text task on that text is simpler, and its train set can grow with synthetic data.
Two things set a vision project apart from a text one:
- No synthetic data. Synthetic data generation is rejected for visual tasks, because the generators can’t produce images. Every row, train and test alike, is an image you supply with an answer the teacher writes by relabeling, or a row you label yourself.
- No endpoint traces. A traces object made directly from an inference endpoint rejects image content, so the images come from files, in every iteration.
The images you collect therefore set the size, the coverage and the quality ceiling of the model.
Supported models
Section titled “Supported models”Every teacher and judge role needs a vision model: base.teacher_model_name,
trace_processing.teacher_model_name, evaluation.llm_as_a_judge_model_name and every model in
trace_processing.relabelling_committee_models. The student must be a vision student. The lists
are in Supported models. One non-vision model in any
role fails the config, and the recommended large teacher, zai.glm-5.3-low-thinking, is
text-only, so set all three fields.
moonshotai.kimi-k3-low-thinking as the teacher and the judge is a good start, or
zai.glm-5.3-flash-low-thinking as the teacher when cost matters more. Answers with counts or
computed values (word counts, coordinates, totals) can need more reasoning to be exact, so put
moonshotai.kimi-k3-max-thinking on the teacher evaluation shortlist for them. Proprietary
teachers can read images for teacher evaluation, which is how a production model such as Gemini
is scored as the baseline.
Where to set it
Section titled “Where to set it”base:
task: question-answering
visual_task: true
student_model_name: Qwen3.5-4B
teacher_model_name: moonshotai.kimi-k3-low-thinking
evaluation:
llm_as_a_judge_model_name: moonshotai.kimi-k3-low-thinking
trace_processing:
teacher_model_name: moonshotai.kimi-k3-low-thinking
observation_format: openai_messages_with_images
tuning:
max_completion_length: 4096
task_descriptionin the job description is the whole system prompt: the output format and the conventions a correct answer follows. Put the same conventions intollm_as_a_judge_instructions(Job description). For a JSON answer, also setsynthgen.output_is_json: true.tuning.max_completion_length(default 2048 tokens) must be above the longest answer. A full-page transcription or a JSON record is long, and a cut-off answer scores near zero.tuning.rlvr_dataset_sizestays at 0. RLVR fails the config withRLVR is not supported for visual tasks.
Building a vision model
Section titled “Building a vision model”The usual flow from files, without synthetic data:
-
Collect the images. Sample at random from the real source, so every image type, layout, quality and empty case is there in roughly its production share. Split them into test and train by source document (a booklet, a user, a session), not by image: pages of one document look alike, and a test page whose document is also in train inflates the score. A few hundred test images is enough (How many rows); the train set is as many as your budget allows. To check the final model on images no iteration has seen, keep a further set aside.
-
Write two trace files,
test-traces.jsonlandtrain-traces.jsonl: one user message per line, with the prompt and the image, and no answer (Images in your data). -
Build the test set. Create a dataset from the test traces only, since relabeling picks traces at random from the whole pool. Relabel them into the test split, with
trace_processing.num_test_relabelledset to the number of test traces, then review the rows next to their images and lock the set (Building the test set):distil dataset create --config input/config.yaml --job-description input/job_description.json \ --traces test-traces.jsonl distil dataset relabel-traces-test <dataset-id> distil dataset download -d test-set <test-set-dataset-id> -
Evaluate the teachers on the test-set dataset: one run per vision candidate, and one with your production model as the baseline (Teacher evaluation).
-
Build the train set. Create a new dataset with the chosen teacher in the config, the locked
test.jsonland the train traces. Relabel a smoke of 128 traces, read its rows next to their images, then relabel the rest, withtrace_processing.num_train_relabelledset to the number of train traces (Relabeling):distil dataset create --config input/config.yaml --job-description input/job_description.json \ --test test-set/test.jsonl --traces train-traces.jsonl distil dataset relabel-traces-train --config smoke/config.yaml <dataset-id> distil dataset relabel-traces-train <dataset-id>Relabeling drops some traces, so expect fewer rows than images.
-
Train on the dataset the full run wrote (Model training), and serve the model. When it falls short, the main lever is more images of the cases it gets wrong, relabeled into the train split of a new dataset that holds the current
train.jsonl,test.jsonland the new traces (Model iterations).
Images in your data
Section titled “Images in your data”Traces use the openai_messages_with_images observation format: the user message’s content is a
list of text parts and image_url parts.
{"messages": [{"role": "user", "content": [{"type": "text", "text": "Transcribe the handwriting on this page."}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]}]}
Train and test rows carry the images as an images list on the user message. Relabeling writes
its rows this way, and rows you upload use the same form:
{"messages": [{"role": "user", "content": "Transcribe the handwriting on this page.", "images": ["data:image/jpeg;base64,..."]}, {"role": "assistant", "content": "..."}]}
- Data URIs only. Every image is a
data:image/...;base64,URI. A URL failsdistil dataset createwithbase.visual_task=true requires every image to be inlined as a data:image/...;base64 URI. - Images on user messages only. A trace with an image on any other message fails to parse.
- Order inside a message isn’t kept. The text parts become one text and the images a separate list, so a prompt can’t rely on text sitting between two images. Say in the text what each image is, in the order the images are attached.
- Facts the image can’t show (an ID, the image’s pixel size, a source name) go in the user
text, with an instruction in
task_descriptionto copy them. Nothing is added to the answers after the teacher writes them, and the student learns to produce the whole answer. - Leave answers out of the traces. A trace that ends on the user message gets an answer the
teacher writes from the image. An answer already in the trace is corrected while staying as
close to it as possible, so another model’s answer passes on its habits and mistakes. Answers
you trust, such as human transcriptions, belong in
train.jsonlandtest.jsonlas rows. synthgen.validation_max_total_lengthcounts the text only, images excluded.
Identical rows
Section titled “Identical rows”Traces with the same text and different images are distinct: deduplication compares the image
bytes. The check that train and test share no rows doesn’t. It replaces each image with a
placeholder for its position, so two rows with the same user text and the same answer count as
one row whatever their images, for example two blank pages answered the same way. Relabeling drops
a new row that matches a row of the other split, and an upload with such a pair in train and test
fails with train and test datasets have N overlapping examples. When those rows matter, put
something image-specific in the user text, such as an ID your application also sends.
Training
Section titled “Training”Training is SFT only. Every image is stored at most 1024 pixels on its long side, while the teacher saw the full image, so small text, dense pages or fine detail can be legible to the teacher and not to the student. Compare a few failed predictions with their images before blaming the student size; when detail is lost, crop the images to the region that matters, in the traces and in production alike.
Evaluation
Section titled “Evaluation”The judge metrics receive the images with the prompt. When the teacher wrote the test set, the
primary metric is llm-as-a-judge-reference-free
(Primary metric per task), so the judge checks
each prediction against the image itself. A pass/fail verdict can’t tell a page with one wrong
word from a perfect one; ask for a graded score
when small errors matter.
Deployment
Section titled “Deployment”- Local vLLM. A vision student needs one more flag on top of the
family flags:
--enable-tower-connector-lora. - Requests.
model_client.pysends the conversation as it is, so build each user message as content parts: a text part, then oneimage_urlpart per image, as data URIs. A test row’simageskey isn’t understood by the server, so convert it before using a test row to try the model. - Hosted inference. Serving image requests through distil labs inference isn’t documented yet; contact contact@distillabs.ai before planning on it.
What doesn’t apply
Section titled “What doesn’t apply”Everything that tunes generation: synthgen.train_generation_target and test_generation_target,
mutators, synthetic_data_generation_instructions, and a top-up by generation. The train set grows
with more images, relabeled into the train split. Tools aren’t available either: visual tasks are
question answering.
Every other parameter: Config file. The stage that writes every row: Relabeling and synthetic data.