Glossary
tau-bench
An agentic benchmark where a model completes realistic retail and airline customer-service tasks through tool calls and simulated user turns, scored by reliability across repeated runs.
A benchmarkevaluationA standardized dataset and scoring rubric used to compare model capability on a defined task, the unit of model evaluation since GLUE made the format the default. Open full entry that measures tool useagentsThe general pattern of an LLM invoking external functions, APIs, or systems to fetch data or take action, distinct from generating an answer purely from its weights. Open full entry the way it actually fails in production: across a whole conversation, not a single call. A model plays a customer-service agent in a realistic domain (retail and airline are the original two), with a real tool set, a policy document it must follow, and a simulated user on the other side. To succeed it has to ask the right questions, call the right tools in the right order, respect the policy, and leave the backend in the correct final state.
What makes it a sharper signal than single-call benchmarks is the scoring. Tasks are checked by the end state of the database plus required outputs, and runs are repeated to measure reliability rather than a lucky single pass. A model that completes a task once but fails it on re-runs scores poorly, which surfaces the brittleness that matters when an agent runs thousands of times a day.
tau-bench, alongside the Berkeley Function Calling LeaderboardevaluationA widely-cited benchmark for function-calling ability that checks emitted calls against expected ones by abstract-syntax-tree match and by executing them, across simple, parallel, and multi-turn cases. Open full entry , is one of the two tool-use evaluations labs now report for open models. The former tests end-to-end agenticagentsAn informal descriptor for AI systems that pursue multi-step goals via tool use, planning, and self-correction, rather than single-turn question-answering. Open full entry reliability; the latter tests the correctness of individual calls. Together they cover the two halves of the tool-call loopagentsThe host-orchestrated cycle behind tool use: the model emits a tool call, the host (not the model) executes it, and the result is fed back for the next turn. Open full entry : emitting a valid call, and driving a multi-step task to a correct finish.