TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens

TAC-CoT interleaves compact continuous reasoning segments with explicit tool calls, observations, and answers, improving agentic math and QA accuracy while reducing forward passes.

Explicit CoT verbalizes all intermediate reasoning, which increases inference cost and makes the model more susceptible to error accumulation. Implicit CoT replaces textual rationales with compact continuous tokens, but prior methods consume a fixed implicit-token budget in a single uninterrupted block. This makes tool use structurally infeasible because there is no intermediate point to emit a tool call or ingest an observation, and it under-allocates reasoning capacity to more difficult queries.

TAC-CoT replaces each explicit reasoning segment with an implicit segment of b continuous thought tokens, while keeping tool calls, observations, and the final answer in explicit textual form. Each continuous thought token is obtained by projecting the last-layer hidden state and feeding it back via input embeddings, bypassing the vocabulary projection. Training data is constructed by converting executable annotations when sources contain tool interactions, and by using DeepSeek-V3 as a tool-augmented solver when sources do not, retaining only trajectories whose predicted answers match the gold answers. The model is trained by self-distillation with the same backbone as student and teacher, using the explicit trajectory as the teacher signal. At inference, the model generates an implicit segment, compares action and answer routing logits, and either decodes and executes a tool call or produces the final answer. After each tool observation, it resumes with another implicit segment, so the number of segments is determined on the fly.

• TAC-CoT improves accuracy by up to 8.8 points while reducing model forward passes by up to 6.8 times. §Abstract
• TAC-CoT improves answer coverage at retrieval steps over Explicit-CoT on QA benchmarks. §3.3
• Using continuous thought tokens improves average math accuracy over the 0 CT setting. §C.1
• Under full fine-tuning of Llama-3.2-1B-Instruct, TAC-CoT improves math accuracy over the Explicit-CoT agent on GSM8K, GSM-Hard, SVAMP, and average. §C.3
• Under a GPT-2 backbone with full fine-tuning, TAC-CoT improves math accuracy over the Explicit-CoT agent on GSM8K, GSM-Hard, SVAMP, and average. §C.3
• TAC-CoT reduces forward passes and wall-clock latency, with average forward-pass reductions and practical speedups reported on math and QA benchmarks. §C.6

• The adaptive routing mechanism operates at the segment level rather than the token level.
• Longer interaction horizons remain an important direction for evaluating practical deployment of TAC-CoT.
• Wall-clock speedups are smaller than forward-pass reductions because total latency includes non-optimized execution paths, parallelization overheads, and API-call latency.

• TAC-CoT interleaves implicit reasoning segments with explicit tokens for tool calls, observations, and final answers, preserving an executable interface for agentic tool use. §2.1
• Each explicit reasoning segment of n discrete tokens is replaced by an implicit segment of b continuous thought tokens, where b is less than n. §2.1
• Continuous thought tokens are produced by projecting the last-layer hidden state and fed back as input embeddings, bypassing vocabulary projection following CODI. §2.1
• Training trajectories are built from sources with tool interactions via conversion, and from sources without tool interactions via a stronger LLM solver with gold-answer filtering. §2.2
• Training uses self-distillation where the same backbone serves as student and teacher, with the explicit trajectory providing the teacher signal. §2.3
• At each segment boundary, routing logits decide whether to decode a tool call or answer; tool observations are appended before resuming implicit reasoning. §2.1
• GSM8K accuracy improves monotonically as the number of allowed calculator calls increases from 0 to 4 and to unlimited calls. §C.4
• Explicit natural-language reasoning can help under oracle reasoning, but its advantage diminishes as the number of generated reasoning steps increases due to error accumulation. §C.5