ToolGrad Makes Tool-Use Data the Focus of AI Agent Improvement

Google Research’s ToolGrad argues that the next step in improving AI agents may depend less on scaling frontier models and more on generating better tool-use training data at scale. Published on September 10, the method starts with a verified API workflow, then asks a model to write the user request and final response that fit it. That reverses the common query-first approach used in earlier tool-use dataset generation systems.¹

The headline claim is substantial: ToolGrad reportedly reached a 99.8% pass rate when generating tool-use data against ToolBench’s catalog of more than 16,000 APIs. It also produced more complex examples at lower cost than query-first search baselines.²³ Google Research then used ToolGrad-500, a compact 500-example dataset, to fine-tune Gemma 3 models. The 12B-parameter version reportedly scored 83.1 on the Berkeley Function Calling Leaderboard, close to Gemini 2.5 Pro at 83.2 and above several larger or proprietary systems listed in the same reports.¹⁴

For AI researchers and ML engineers, the core implication is not that small models now generally outperform frontier models. It is narrower and more operationally useful: verified tool-call trajectories may be a high-leverage training signal. If the result holds across broader tool ecosystems, agent performance could improve materially through better data-generation pipelines, sandboxed execution, and targeted fine-tuning, not only through larger base models.⁶⁷

What ToolGrad changes

Traditional synthetic tool-use data generation usually starts with a user query. A system samples possible APIs, asks a model to invent an instruction, and then sends an agent to search for a valid sequence of tool calls that satisfies it. That creates a structural problem: the generated request may not have a feasible solution in the available API set, causing the search procedure to spend compute on dead ends.²

ToolGrad inverts that order. It first constructs an executable chain of API calls, validates those calls through execution feedback, and then generates the natural-language request and final response that match the completed chain.¹³ In practical terms, it turns a difficult open-ended search problem into a more constrained annotation problem. Once the system has a working answer path, writing a prompt that asks for it is comparatively easier.

The method uses a loop with four components: an API proposer, API executors, an API selector, and an LLM updater. Candidate calls are proposed, executed, evaluated, and appended to the workflow. The synthetic user query and response are then revised to stay consistent with the expanding chain.³⁸ Reports describe this as an application of “textual gradients,” where natural-language feedback from execution and selection steps guides the next stage of workflow construction.⁴

Why the pass rate matters

A 99.8% generation pass rate is primarily a data-yield metric. It means that almost every generated example survived the pipeline’s checks and could be used for downstream training, according to the reported ToolBench experiments.²³ It does not mean a trained model will solve 99.8% of real user tasks.

But for dataset construction, yield matters. Failed trajectories are expensive: they consume model calls, tool executions, search steps, and validation work without producing usable training examples.

The comparison is especially relevant for organizations building agents over internal API catalogs. Many teams lack large volumes of clean production traces showing successful multi-step tool use. ToolGrad-style generation could let them execute workflows in a sandbox, filter for valid trajectories, and fine-tune smaller models on the resulting examples.²⁵

That changes where engineering effort is spent. Instead of manually authoring each trajectory or relying on an agent to discover a valid path after the fact, teams would need high-quality machine-readable API specs, safe test credentials, realistic execution environments, validation checks, and held-out evaluations.² In other words, the bottleneck shifts from model search to data infrastructure.

What the Gemma result does—and does not—show

The most attention-grabbing downstream result is ToolGrad-12B’s reported BFCL score of 83.1 after fine-tuning on ToolGrad-500, a dataset of 500 synthetic examples.¹⁴ Reports also say the model performed near Gemini 2.5 Pro and surpassed the Gemini 2.5 Flash-Lite model that generated the training data.³⁶

That supports the research claim that smaller models can become substantially better at tool use when trained on verified API workflows. The plausible mechanism is straightforward: the smaller model is not learning from arbitrary synthetic text, but from compact examples that contain successful tool-call structure, argument patterns, and multi-step dependencies. Execution filtering removes many low-value or invalid samples before fine-tuning.

Still, the claim should be interpreted carefully. BFCL is a benchmark, not a full deployment environment. Tool-use agents in production must handle ambiguous user intent, authentication constraints, partial failures, rate limits, changing schemas, unsafe side effects, and recovery from wrong calls. Successful execution in a sandbox also does not guarantee that the generated workflow reflects a common or valuable real-world user request.²

The stronger conclusion is that ToolGrad demonstrates data efficiency under the reported experimental setup. It does not prove that 500 examples are sufficient for every domain, or that answer-first synthesis will automatically cover the long tail of production workflows.

Why answer-first generation could scale agent training

The method is attractive because it aligns synthetic data generation with what tool-use models actually need: valid examples of how requests map to executable tool calls and final answers. Query-first approaches ask an agent to discover whether a solution exists. Answer-first approaches verify that a solution exists, then generate the request around it.⁷⁸

For ML teams, that opens several practical directions:

Generate domain-specific trajectories over private APIs without exposing real user data.
Control workflow complexity by targeting chain length, dependency patterns, or API families.
Use cheaper generator models to create supervised data for smaller deployable models.
Build evaluation sets from held-out APIs to test whether models generalize rather than memorize.

The availability of links to the paper, repository, dataset, models, and package in coverage of the release also makes the work more actionable for researchers who want to reproduce or adapt the pipeline.⁵

The open questions

ToolGrad’s most important open question is whether execution-verified synthetic workflows remain useful as API ecosystems become larger, messier, and more dynamic. Real tools change. Schemas drift. Authentication fails. APIs return unexpected values. User requests may be underspecified or conflict with policy constraints. A generation pipeline that works on a benchmark catalog still needs domain-specific validation before it can support production agents.

There is also a data-distribution issue. Because ToolGrad writes the user request after the workflow exists, its prompts may differ from real user language. Human review, production-log comparison, diversity filters, and adversarial evaluations may be needed to keep models from overfitting to synthetic phrasing or unrealistic tasks.

Even with those caveats, the work points to a practical shift in agent development. If verified workflows can be generated cheaply and used to fine-tune compact models, organizations may be able to build more capable specialized agents without waiting for the next frontier model release.

ToolGrad’s contribution is best understood as a data-generation result with model-performance consequences: better tool-use data can make smaller models much more useful.¹⁶