TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

TauIndianBankBench is a 1,000-task Indian retail-banking benchmark that scores both required procedure steps and final records for tool-using conversational agents.

Indian banks need assistants that both give correct information and follow written servicing rules, which prescribe required steps such as obtaining a closure quote before closing a deposit. Existing tool-use benchmarks often judge only final records, but many required steps leave records unchanged, so an assistant that skips them can still be judged successful. The paper presents a benchmark that scores procedure and outcome for Indian retail banking.

TauIndianBankBench is a synthetic Indian retail-banking environment with a tool catalogue, a written policy, a regulation knowledge base, and 1,000 tasks, run in the tau2-bench harness with a simulated customer and shared database. Tasks fall into sequencing, edge-case, rare-tool, and control categories, with train and test splits drawn from the base set. Each session gets two pass/fail verdicts: a final-state check comparing records with an oracle replay of the gold chain, and an action check requiring every gold-chain call including reads plus write purity. A session passes only if both checks pass. The paper evaluates five Gemma 4 models and Qwen3.8-27B with one attempt per task under deterministic sampling and the same simulated customer.

• Overall pass rates on the 1,000-task base split range from 0.849 for Qwen3.8-27B to 0.479 for Gemma 4 E2B. §5.1
• Scoring only the final records raises each model’s pass rate by 6.5–12.0 points, the share of its sessions that are procedural violations. §5.2
• For the three strongest models, sessions that pass only the final-state check exceed those that pass only the action check (p<10^{-8}); for E2B the reverse holds (p<10^{-3}). §5.3
• The control category has 112 tasks, 1.38 calls on average, and 93 with no write; across the four dense Gemma models pass rates remain between 0.804 and 0.929 without a size trend. §5.6
• Scripted agents pass no task: the first only replies, the second looks up records without required calls, the third claims completion, and the fourth makes every available change and fails write purity. §5.7
• The gap between the strongest and weakest model is widest on five-call tasks (0.614, against 0.024 on one-call tasks). §5.5

• The paper states that a score depends on both the assistant and the simulated customer, and the simulator’s own error rate was not measured.
• All customer data are synthetic, so the frequency of each edge-case type reflects generated records rather than actual bank traffic.
• Version 1.0.0 covers Indian retail banking in English only.
• Evaluation is computationally costly: a base-split run requires 1,000 conversations of up to 200 steps each, and estimating pass^k multiplies this cost by k.

• The benchmark has 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. §Abstract
• In the corpus, 962 of the 1,000 released tasks require at least one read-only call, and 114 require no write call. §1
• Task counts are sequencing 470, edge case 266, rare tool 152, and control 112. §D
• The tool catalogue has 33 servicing tools and 13 writes; a 34th tool transfers the customer to a human staff member. §B
• The action check examines the trajectory, the final-state check examines the final database, and write purity requires every write to match a gold-chain write. §3.2
• Stopping short accounts for 49.6–64.0% of failed sessions. §G.3
• A second run of the five Gemma models changes each pass rate by at most 2.7 points, although 14–26% of tasks change outcome. §5.8
• The 200-task lite split is provided for development and ablations, and the base split for final reporting of untuned models. §Limitations