Root-cause themes for Berkeley Function Calling Leaderboard (v4) failures of three Qwen3 thinking models.
Data: HuanzhiMao/BFCL-Result, snapshot 2025-12-16 — one pass per
problem at temperature 0.001. Themes were labeled on a stratified sample of 70 failures per model (seed 0).
BFCL v4 spans four kinds of task. They differ in what the model is asked to produce and — crucially — in what the grader checks: the emitted function call (AST match against ground truth), or the resulting state / answer value. Nothing here executes the target functions themselves except multi-turn, whose backend does.
Single-turn (non-live & live). The model gets one query plus a set of function schemas and must emit the correct call(s) — right function, right arguments. Only the call is checked, by AST match against ground truth; the function is never executed. Non-live uses expert-curated cases (simple / multiple / parallel / parallel-multiple, plus Java & JavaScript); live uses real crowd-sourced user queries and APIs. Both include irrelevance / relevance detection, where the correct behaviour is to make no call (or decline).
Format sensitivity. A robustness ablation, not a new skill: 200 single-turn items (drawn from simple / multiple / parallel) are re-posed under 26 prompt variations across five axes — return format (Python / JSON / verbose or concise XML), function-doc format, tool-call tags, plaintext vs Markdown, and original vs rephrased wording. As in single-turn, the call is AST-checked, never executed; the metric is whether the same correct call survives every format.
Multi-turn. A stateful conversation over several user turns against a provided function schema. Unlike the above, calls are actually executed against a mock API backend, and grading is state-transition based — it verifies the resulting backend state, not just the call string. Subcategories stress-test recovery: base, missing-function, missing-parameter, long-context.
Agentic — memory. A user–agent interaction in two phases. First a write phase: the model always stores facts from the conversation via a memory tool call (memory_add / memory_append…). Then a recall phase: answering a follow-up. A retrieval tool call is optional — the harness dumps stored memory back into context, so the model often answers without one (in our rollouts only 5 / 186 passing episodes issued a retrieval call). Scoring is on the answer's value against ground truth, not on tool usage. Three backends: kv (BM25 keyword search), vector (MiniLM embeddings + FAISS), rec_sum (a 10k-char summarization buffer whose memory_retrieve() dumps everything rather than searching).
Agentic — web search. The other agentic sub-domain: multi-step tasks requiring live web-search tool use. (Deferred in our rollouts — needs an interactive harness and live internet.)
Dataset
What BFCL v4 contains
10,306 scored test cases per model. "~prompt tokens" = the median full prompt we feed the model (system prompt + function schemas + question, chars/4) for single-turn tasks; multi-turn/agentic lengths are the median user text per episode.
Sampling distribution — t=1.0, n=100
Pass@k, problem mix, and t=1.0 failure themes
728,200 completions: Qwen3-8B and 14B thinking, the 13 single-turn categories (3,641 problems × 100 samples, temperature 1.0, Qwen3-card sampling), generated on Euler, graded offline with bfcl_eval's AST checker. 32B deferred; multi-turn/agentic need an interactive harness and are not sampled — so no official-weighted score exists in this mode (it would require the unsampled categories).
pass@1 → pass@100 per category group (per model):
Failure themes at t=1.0 — 80 labeled failures per model, split by whether the problem is never solved in 100 samples (systematic) or sometimes solved (stochastic; counts pooled over both models):
Failure themes
What actually went wrong
Counts over the 70 sampled failures per model, one theme per failure. Click a row to browse its examples.
Where it fails
Accuracy by category
Full benchmark, 10,306 tests per model. The official leaderboard overall weights these very differently from their test counts: non_live 10%, live 10%, irrelevance 10%, multi_turn 30%, agentic 40% — format_sensitivity excluded. That is why ~71-75% micro accuracy becomes 40.4 / 37.8 / 46.8 official (frontier reference, same formula and snapshot: Claude Opus-4.5-FC 77.5, Gemini-3-Pro-FC 68.1 — the gap is entirely multi_turn + agentic). Click a row to browse that category.
Accuracy by task
All 23 scored tasks
Full data, sorted hardest first. Click a row to search its examples in the browser.
Grader verdicts
Error types over all failures
Counts over every failed test in the full score files (8B 2,984 / 14B 2,836 / 32B 2,589 failures) — not the sampled subset. Click a row to search matching examples.
Example browser
210 labeled failures + 186 memory-recall passes
Each failure example: the benchmark input (user request + available functions), the expected ground truth, the model's output, and the failure mode (theme + grader verdict). AST ground truth shows allowed values per argument (alternatives separated by |); agentic tasks show the accepted answer strings; multi-turn shows the reference call sequence per turn. The memory-recall passes (all 186 archive successes across the three backends, filter outcome = successes) show the full recall loop: the memory state injected in context, the CoT, any executed retrieval calls with their results, and the final answer. Striking pattern: only 5 of 186 passes execute a retrieval call at recall (4 of them Qwen3-32B on kv) — passing models answer by reading the context-injected memory, while retrieval calls appear more often in failures (42 of 1,209, mostly 32B kv). Recall-phase function calling is thus barely exercised by passes; the kv/vector archival APIs are the only place it is genuinely required.