Instruction Capacity: Behavior and Degradation of Language Models under Controlled Load

The paper instruments atomic single-token word-replacement rules and a binned logit model to measure instruction capacity and degradation laws, reporting a GPT-4o mini passport with capacities in rules and uncertainties.

Instruction-following quality falls as load grows, but existing studies do not give a quantity in units of load with an uncertainty. In prior work the number of instructions is confounded with instruction composition and task difficulty because instructions differ in nature, share one coherent text, or are checked by a judge model. The paper argues that a load measure needs an instruction that is atomic, independent, homogeneous, program-checked, and observable at each place in the answer.

The instrument defines an instruction as a rule replacing one single-token word with another in a meaningless word stream generated from three seeds. Each occurrence of a word to be replaced is a Bernoulli trial with known rule number and answer position; the reference answer is computed by a program and answers are matched line by line with three failure kinds. The statistical model uses binned logit contributions plus a random request shift, fits smooth laws to bin profiles, and applies a shift correction to restore a reference level independent of reference-bin coding. Instruction capacity is defined as the number of rules at which cumulative rule price exhausts the reserve to a quality threshold q, with delta-method uncertainty. GPT-4o mini is measured to produce a passport of laws and capacities.

• The GPT-4o mini measurement covered 1890 experimental points and 2958 requests with 642 430 trials. §5.1
• The text length and word position grid is described by a smoothed minimum of a plateau and a slanted asymptote, with common slope and transition width across slices. §5.3.1
• Replacement density follows an exponential saturation law, and the loss at full density reaches about nine logits relative to rare replacements. §5.3.2
• Each doubling of the rule list lowers the logit by a constant amount, and at 65–128 rules the decrease reaches about 2.8 logits. §5.3.3
• In the GPT-4o mini measurement 995 requests out of 2958 lost the text, that is 33.6%, and the collapse rate law has a length slope and a full-density contribution. §5.4.1
• The passport capacities include instruction capacity at q=0.95, guaranteed instruction capacity at q=0.95, and fatigue capacity at q=0.95. §5.5.4

• The paper states that the mechanistic interpretations of the measured laws require separate experimental tests; the instrument measures the form of dependences but not their causes. §5.3.1
• The passport describes only GPT-4o mini; the paper says there are no grounds to assume its regularities are universal and calls for passports of many models. §6.1
• The fatigue capacity at q=0.5 is beyond the measured region and is obtained by extrapolating the law. §5.5.2
• Prompt weighing assumes that a prompt's extra load acts on the logit like added rules; the paper says this representation is yet to be established experimentally. §6.2

• A rule such as lake -> stone is atomic: it applies to one word, has exactly one right outcome per application, and rules do not depend on each other. §3.1 §3.1
• Vocabulary v1_intersection contains 2220 monosyllabic singular nouns, each one token for o200k_base, cl100k_base and the Qwen2.5 tokenizer. §3.2 §3.2
• Answer validation matches line by line, distinguishes failures of first, second and third kind, and counts unmatchable positions as unsuccessful. §3.4 §3.4
• The binned logit model includes a random request shift b_j ~ N(0, σ²) and no free term tied to reference bins. §4.3 §4.3
• The approximation shift correction sets each axis law to zero at its minimum load and restores the coding-independent reference level α_phys. §4.4 §4.4
• Five axes V, S, P, M, R were selected; V and S were combined into a two-dimensional grid because their contributions are non-additive. §5.2 §5.2
• Late rules degrade with A_R=0.041±0.040 and γ=0.74±0.20; rules 65–128 are worse than the first by 1.05±0.12 logits. §5.3.4 §5.3.4
• At fixed text length and replacement density the number of rules does not affect collapse probability; all nine requests with 256 rules lost the text. §5.4.1 §5.4.1