Tool calling
How open models call tools
Tool calling is what turns a model that writes text into one that can act: search the web, run code, read a file, query a database. It is a cross-cutting concern, so the pieces live on different layers of the stack. This page walks the whole arc in order and links each piece.
The capability, the mechanism that makes a call valid, the open tooling that produces it, the wire that standardizes it, and the benchmarks that measure it.
1 · The loop
A model never runs anything itself. It emits a structured function callingagentsA pattern where a model emits a structured call (function name plus arguments), the runtime executes it, and the result returns as input on the model's next turn. Open full entry ; the host program executes the actual function and feeds the result back; the model reads it and either answers or calls again. That cycle is the tool-call loopagentsThe host-orchestrated cycle behind tool use: the model emits a tool call, the host (not the model) executes it, and the result is fed back for the next turn. Open full entry , and the hard parts (how many iterations, what to do on error, how to truncate a large result) live in the loop, not the call. It is the engine under ReActagentsAn agent loop where the model alternates between reasoning steps (thought) and acting steps (tool call), explicitly interleaving free-form deliberation with structured tool use. Open full entry -style and multi-agentagentsArchitectures where multiple LLM-driven agents collaborate or compete on a task, each with its own role, prompt, or specialization, coordinated by an orchestrator or message-passing protocol. Open full entry systems, and it sits at the agents layer.
2 · What makes a call valid
A tool call is only useful if it parses. Getting that guarantee is structured outputruntimeMaking a model emit text that conforms to a schema or type (often JSON matching a JSON Schema) rather than free-form prose, so downstream code can parse it reliably. Open full entry : making the model emit text that conforms to a schema rather than free-form prose. The strong form is constrained decodingruntimeMasking the model's next-token probabilities at each step so only tokens allowed by a grammar or schema can be sampled, forcing the output to be structurally valid. Open full entry , where the runtime masks the next-token choices at each step so only schema-valid tokens can be sampled, usually by walking a grammarruntimeA formal description of allowed output structure (for example llama.cpp's GBNF) that a decoder is constrained to; a JSON Schema can be compiled into one. Open full entry compiled from a JSON Schema. The weaker predecessor is JSON moderuntimeA weaker predecessor to structured output: the model is constrained to emit syntactically valid JSON, but not necessarily JSON that conforms to a specific schema. Open full entry , which promises valid JSON but not a specific shape. This is a runtime-layer concern, and it is also covered in the self-host course module on decoding.
3 · The open tooling
The token masking is done by a small set of open libraries. Outlines, XGrammar, and llguidance are the enforcement engines that serving runtimes call underneath; Guidance, Instructor, and LM Format Enforcer sit closer to the developer. The open inference engines themselves (vLLMruntimeAn open-source inference engine introduced by UC Berkeley in 2023, built around PagedAttention to manage KV cache memory and serve tokens efficiently under load. Open full entry , SGLangruntimeAn open inference engine from the LMSYS team featuring RadixAttention for prefix sharing and a structured-generation frontend, particularly strong on agent and tool-calling workloads. Open full entry , llama.cppruntimeGeorgi Gerganov's C++ inference engine optimized for CPUs and consumer GPUs, the on-device standard and the engine behind Ollama, LM Studio, and most local-first AI products. Open full entry ) expose a schema or grammar parameter on the request and hand the work to one of these engines.
4 · The formats differ
There is no single wire format for a tool call. Each model emits only the serialization it was trained on, and runtimes ship per-model parsers to read them back. Pick a family to see how it serializes the same call.
5 · The wire that standardizes it
Per-model formats handle how a model emits a call. A separate problem is how the host discovers and connects to tools in the first place. The MCPprotocolsAn open protocol from Anthropic that standardizes how language models discover and call external tools, data sources, and prompts via a small JSON-RPC interface. Open full entry standardizes that tool layer so the same server works across models, with A2AprotocolsA Google-launched open protocol for agent-to-agent communication, letting agents from different vendors discover each other's capabilities and exchange structured messages. Open full entry covering agent-to-agent communication. Both sit at the protocols layer.
6 · How it is measured
Two evaluations anchor tool use. The Berkeley Function Calling LeaderboardevaluationA widely-cited benchmark for function-calling ability that checks emitted calls against expected ones by abstract-syntax-tree match and by executing them, across simple, parallel, and multi-turn cases. Open full entry checks the correctness of individual calls; tau-benchevaluationAn agentic benchmark where a model completes realistic retail and airline customer-service tasks through tool calls and simulated user turns, scored by reliability across repeated runs. Open full entry measures end-to-end agentic reliability on realistic multi-step tasks. Together they cover the two halves of the loop: emitting a valid call, and driving a task to a correct finish. Where a lab has published a score, it appears on the model's page and you can filter the models table to the ones that carry a tool-use benchmark.
Related: the course module on tool use and agents, the evaluation and agents layers, and the glossary.