Agent Frameworks vs a Fine-Tuned Tool-Calling Model
They operate at different layers, so this is rarely a genuine either-or. A framework connects tools to a model, manages conversation state, and executes calls. A fine-tuned model decides which tool to call and with what arguments. If your agent picks the wrong tool, no framework fixes that.
What does each layer actually own?
Splitting the responsibilities makes the diagnosis obvious.
| Concern | Agent framework | Fine-tuned model |
|---|---|---|
| Tool discovery and schemas | Yes | Consumes them |
| Executing the call | Yes | No |
| Conversation state | Yes | Sees it as context |
| Retries, timeouts, tracing | Yes | No |
| Choosing which tool | No | Yes |
| Filling arguments correctly | No | Yes |
| Knowing when to call nothing | No | Yes |
| Staying in schema under pressure | No | Yes |
The rows split cleanly at “choosing”. Everything above it is plumbing, and plumbing is what frameworks are good at. Everything below is model behaviour.
What are agent frameworks good at?
They’re good at the integration problem, which is real and which nobody should be solving by hand. Standardising how a model reaches external systems is the whole point of the Model Context Protocol, which describes itself as “an open-source standard for connecting AI applications to external systems” and compares itself to a USB-C port for AI applications.
Frameworks also give you things a model can’t:
- Retry, timeout and error-handling policy around each call
- Tracing, so you can see what happened on turn nine
- State across turns without you rebuilding it
- Swappable model endpoints behind one interface
- A tool registry that multiple agents share
None of that is optional in production. Building an agent without a framework means writing all of it yourself, usually less well.
What is a fine-tuned tool-calling model good at?
It’s good at the decision, and the decision is where accuracy lives. Gorilla made the case early by combining retrieval with fine-tuning specifically to reduce API hallucination, noting that LLMs’ “potential to effectively use tools via API calls remains unfulfilled” without it.
The size of the effect on a fixed schema is larger than most teams expect. From the distil labs platform benchmarks, the same student model prompted versus fine-tuned:
| Tool-calling benchmark | Base student (prompted) | Trained student | Teacher |
|---|---|---|---|
| Git tool calling | 0.03 | 0.95 | 0.81 |
| Pizza tool calling | 0.00 | 0.70 ± 0.03 | 0.50 ± 0.04 |
A prompted 3B model producing schema-valid git commands scores near zero. That failure is invisible to your framework. The call is well-formed enough to execute, it’s just the wrong call. And in both rows the fine-tuned student beats the teacher that generated its training data, because a general model has never seen your schemas. What tool calling is and how you train for it covers that training side.
Multi-turn work makes this compound. As the traces benchmark puts it, 95% per-turn accuracy leaves roughly 35% of 20-turn conversations fully correct, the compounding that multi-turn tool calling explained sets out. Per-turn accuracy is a model property.
Are they actually alternatives?
Only in one narrow case: if your agent is a single tool call per request with no state, a framework adds more than it earns. Call the model, parse the call, execute it. That isn’t an agent framework, it’s fifteen lines.
Everywhere else they’re layers, so work out which layer your failures live in. Instrument first:
- Wrong tool selected, or arguments invented → model layer
- Right tool, execution failed or timed out → framework layer
- Right tool, right arguments, wrong turn to call it → model layer
- Context lost between turns → framework layer
Teams commonly respond to the first and third by adding prompt scaffolding inside the framework (more examples in the system message, stricter instructions, a validation retry loop). That treats a weights problem with tokens, which is the trade-off set out in prompt engineering vs fine-tuning. It sometimes works, and it costs latency on every request forever.
How do you combine them well?
Keep the framework thin and put the intelligence in the model. The distil labs deferral demo is a clean example of the shape: every assistant action is a single tool call, including talking to the user via respond_to_user, so the orchestrator only calls the model, executes one call, and feeds the result back. Both the tool set and the policy load from job_description.json, the same artefact the model was trained on, so the runtime and the model can’t drift apart.
That last detail is the one worth copying. If your tool schemas live in framework code and your training data lives somewhere else, they’ll disagree eventually.
To train the model half, the platform tasks are tool-calling-closed-book and multi-turn-tool-calling-closed-book, described in task selection, with data formats in tool calling data preparation. Only a subset of students and teachers support them, and the supported models catalog marks which.
The broader argument for putting small specialised models at agent nodes is made by Belcak and colleagues and echoed in the distil labs agentic AI post: most agent nodes are narrow, repeatable tasks, and running a frontier model at every one of them costs latency and money you don’t have to spend. For picking the student, see qwen vs llama vs gemma for tool calling and functiongemma 270m for multi-turn tool calling.