Hard facts, soft prose: Blended Generation with Recursive Language Models

The rush to integrate Large Language Models within core business processes by way of Agentic Workflows has surfaced a core contradiction between the precise analysis commanded by these processes, and an antithetical neural architecture and training regimen which makes Large Language Models ill suited for numerical analysis.

We believe that a blended strategy, where LLMs merely orchestrate the analytical process by writing symbolic programs evaluated by an interpreter is a way to bridge this gap.

Why language models get the numbers wrong

Autoregressive sampling has no “exact” mode, and the reason is visible in the training objective itself. A language model is trained to minimise cross-entropy over the next-token distribution:

\mathcal{L}(\theta) = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})

Nothing in this loss distinguishes a factual token from a stylistic one — a numeral is a token like any other. If the training contexts that resemble yours contain market shares of 4.2%, 4.3%, and 4.7%, the loss-optimal model spreads probability mass across all of them, because that is what the data distribution looks like. For prose this is exactly what you want: synonyms are interchangeable, and a model that hedges between them writes fluently. For numbers it is fatal, because distance in probability space is not distance in fact space. To the objective, “4.3” is a near-miss; to the reader, it is simply a wrong answer.

Tokenisation makes this worse. BPE segments text by corpus frequency, not by place value, so an exact figure is shattered into arbitrary chunks. Modern tokenisers address numeral tokenisation by either splitting to single digits, like in the case of Deepseek and Llama tokenisers, or splitting in groups of 3 tokens, from right to left, e.g. $1{,}234{,}567 \rightarrow \texttt{1}\,|\,\texttt{234}\,|\,\texttt{567}$ . Emitting one figure is therefore a sequence of $m$ sampling events, and the figure is only correct if every one of them lands:

P(\text{figure correct}) \;=\; \prod_{i=1}^{m} P_\theta(v_i \mid v_{<i},\, c)

Every factor is at most 1, so confidence decays multiplicatively with the length of the number. Even a model that places 98% of its mass on the correct chunk at every step gets a six-token figure right only $0.98^{6} \approx 0.89$ of the time. This is a double-digit hallucination rate that looks, token by token, like near-certainty. And unlike prose, numerals offer no partial credit: one wrong chunk anywhere yields a wrong number, and it fails silently, formatted just as confidently as a right one.

Correctness of a sampled figure decays as p^m with the figure’s length in tokens; a fact slot is a single event, flat in m.

Long contexts compound both effects: as the prompt grows, retrieval of the exact source value degrades well before the context window is nominally exhausted, a phenomenon Hong et al. (2025) named context rot. The conditional $P_\theta(v_i \mid c)$ flattens exactly in the regime where business documents live.

Recursive Language Models in thirty seconds

Recursive Language Models, introduced by Zhang, Kraska & Khattab in 2025, attack context rot at its root: they stop asking the model to read the long prompt at all.

In the RLM setup, the prompt never enters the model’s context window. It is stored as a variable inside a Python REPL environment, and the root model is told only that the variable exists, along with some metadata. The LLM is encouraged to write symbolic programs to peek at slices, grep for patterns, and invoke itself on as many slices of the input as necessary. When the LLM has assembled what it needs, it finishes with either FINAL(...), verbalising the answer, or FINAL_VAR(...), returning a REPL variable.

The inversion is the whole trick. A vanilla LLM conditions on the entire haystack, and the conditional $P_\theta(v_i \mid c)$ flattens as $c$ grows. An RLM not only conditions each call on a chosen sliver of the haystack, so no single forward pass ever operates in that regime, but it also transforms an out-of-distribution task (proprietary business questions) into an in-distribution one (writing Python code).

What matters for this post is the primitive RLMs establish: an agent whose relationship to both its input and its output runs through an interpreter rather than through attention. Zhang et al. use that primitive to remove two ceilings: on how much can be read, and on how much can be written. Blended Generation asks what the same primitive is worth if you point it at a different problem: not the size of the answer, but whether it is true.

Blended Generation: hard facts, soft prose

Zhang et al. treats verbalising an answer as a flaw, because a verbalised answer inherits the model’s output length ceiling. Blended Generation treats verbalisation as a flaw due to a lack of determinism and falsifiability.

What’s missing from Zhang et al. is a policy for what may be returned via the FINAL_VAR(...) mechanism. Returning a variable is not the same as returning something the interpreter derived, and the paper’s own trajectory analysis says what tends to fill it: on long-output tasks the root builds the final variable as a “mixture of programmatic and sub-(R)LM output calls”, stitching together the returns of llm_query(). A figure that passes through a leaf call has been through $P_\theta$ again, and arrives in the variable with exactly the failure mode from the first section, now laundered through something that looks symbolic. FINAL_VAR moves the answer out of the root’s token stream; it does not make the answer computed.

The paper also reports how unstable the choice between the two exits is in practice. Their negative-results appendix puts it plainly: distinguishing a final answer from a thought “is brittle for RLMs”. The exit is a formatting convention, and a model reaches for whichever one its sampling favours.

Blended Generation is that missing policy. The root agent’s final act is not to choose an exit but to build a deliverable: it writes an output program, and the contract is that everything in the returned variable is either computed by the interpreter or prose that no fact depends on. Not “return a variable”; this is what a variable is allowed to contain, and here is the machinery that refuses it otherwise.

The root agent’s final act is not to choose an exit but to build a deliverable.

An output program is made of two kinds of material. Fact slots are holes filled by computed values: dataframe lookups, aggregations, arithmetic, interpolated into the output by the interpreter. The narrative shell is the prose around them, generated autoregressively exactly as before. The smallest complete example:

df = pd.read_csv("market_share.csv")
top = df.sort_values("market_share", ascending=False).iloc[0]

output = (
    f"The biggest company by market share is {top['company']} "
    f"with a market share of {top['market_share']:.1%}"
)

Every character of the shell: “The biggest company by market share is…”, was sampled from $P_\theta$ , and that is fine: it is prose, and prose tolerates variance. The two slots were never sampled at all. The model decided which computation produces the figure; the interpreter copied the figure into place. Even the formatting is symbolic, :.1% rounds and renders by rule, not by the model’s recollection of how percentages usually look.

Return to the arithmetic of the first section. Sampled, a figure is correct only if all $m$ of its chunks land:

P(\text{figure correct}) \;=\; \prod_{i=1}^{m} P_\theta(v_i \mid v_{<i},\, c)

a product that decays with the length of the number and flattens with the size of the context. In a fact slot, the same probability collapses to a single event:

P(\text{figure correct}) \;=\; P(\text{program correct})

with no dependence on $m$ and no dependence on $|c|$ . A twelve-digit revenue figure is exactly as safe as a single digit, over a million-token context or a one-line one, and the likelihood that post-training included trajectories where the LLM learned to sort Pandas dataframes is higher than the likelihood that it contained market share analysis.

An added benefit, which we will discuss later, is that errors in execution have a built-in, deterministic critic signal. Stack traces, errors and assertions, all provide grounding that the agent needs to avoid doubling down and being confidently wrong.

In short, the sampled failure formats itself beautifully and tells no one. The symbolic failure crashes, or at worst leaves an audit trail.

The interpreter as a deterministic critic

Everything so far treats the REPL as a producer: of retrieved slices, of computed figures, etc. It is also, at no additional design cost, a critic, and that second role turns out to carry more weight than the first.

Start with what the literature says about the alternative. The appealing idea is that a model can check its own work: draft, critique, revise. Self-Refine (Madaan et al., 2023) reported gains from exactly that loop. The follow-up work has been unkind to it. Huang et al. (2024) find that intrinsic self-correction degrades reasoning performance rather than improving it, and that the gains reported earlier depended on oracle labels quietly leaking into the loop, telling the model when to stop. CRITIC (Gou et al., 2024) makes the constructive version of the argument, that LLMs cannot reliably verify themselves, but when the critique arrives from a tool (an interpreter, a search engine, an API) the verify–revise loop works, across free-form QA, mathematics, and toxicity reduction.

The mechanism is not mysterious. A verifier is only worth having if its errors are uncorrelated with the generator’s. When the critic is the same weights, conditioned on much the same context, it is drawing from the distribution that produced the mistake: the model is confidently wrong at draft time and confidently agreeable at review time. A hallucinated figure looks exactly as plausible to the reviewer as it did to the author, because the same prior scored both.

An RLM already has an uncorrelated critic sitting in the room. The property usually advertised about the Python REPL is what it unlocks on the input side — the two-orders-of-magnitude context. But every execution also returns a signal that is:

Deterministic. Same code, same verdict. Nothing is sampled, so nothing regresses to a plausible-sounding average.
Independent of the model. A traceback comes from Python, which has never seen the prompt and holds no opinion about what the answer ought to be.
Dense and localised. KeyError: 'total' at line 14 shows error provenance exactly, with an included stack trace.

It is also cheap. A failing execution costs milliseconds and one traceback’s worth of tokens, against a full critique pass.

Zhang et al. name the mechanism directly, describing the root as iteratively refining its recursion “via execution feedback from the persistent REPL”. They use it to correct strategy: a chunking scheme that returned nothing, a regex that matched too much. We are pointing the same channel at the deliverable.

With RLMs, the CRITIC loop comes for free: the model writes pandas, the pandas raises, the model reads the traceback and repairs.

Crashing errors are free. A wrong column name, a shape mismatch, a bad join key all announce themselves.

Silent-wrong errors are not. The dangerous class is code that runs cleanly and yields a wrong deliverable, formatted exactly as confidently as a right one.

Against these, the interpreter is a critic with nothing to say. It has no idea what the answer was supposed to satisfy.

Making the silent failures loud

The remedy follows from the same principle that makes a traceback useful. The critic must be external to the generator, deterministic, and specific.

Concretely, we inject a trusted prelude into the sandbox namespace before the model’s script runs. The model does not define these functions; it calls them. Three kinds of check, in increasing strength:

Structural. A null is never an intended value in a report. verified_table(frame) refuses to render a frame containing one, and a backstop outside the sandbox rejects an exported CSV with empty cells. Entirely domain-agnostic, and it catches the label-misalignment NaN above.

Metamorphic. Relations that must hold over the result, declared by the model alongside the code that produces it. reconciles(total='Total', parts=['In-store','E-commerce']) asserts that a total column equals the sum of its part columns, row-wise; column_totals(total='Total') asserts that a total row equals the column-wise sum of the rows above it. Metamorphic testing is the right frame, because in analytics there is no ground truth to compare against. Only invariants.

Provenance. The strongest of the three, and the one specific to Blended Generation. Register every figure with a second, independent expression that recomputes it from source:

facts = {
    "leader":  fact(leader, "piv['Total'].idxmax()"),
    "lead_bn": fact(piv.loc[leader, 'Total'] / 1e9, "piv['Total'].max() / 1e9"),
}
output = render("**{leader}** leads at ${lead_bn:.2f}B.", facts)

render re-evaluates each expression against the live frames and raises if it disagrees with the emitted value. It closes the gap where the fact slot alone cannot guarantee that the number in the variable holds the correct quantity.

render enforces one more discipline the fact-slot idea implies but cannot police on its own: every figure in the template must arrive through a slot. A typed numeral raises even when it happens to be correct, because “correct this time” is a property of the sample, not of the program.

When the critic will not sign off

A last consequence, easy to miss. A critic the model cannot satisfy is itself telling you something.

If the repair loop is unbounded, a model that keeps failing verification keeps resampling until something slips through, which selects for answers that pass by luck rather than by construction. Bounding it forces the third option a reliable analytical agent needs, beyond “answer” and “silently give up”: an explicit, user-visible abstention. I could not produce a verifiably correct answer; here are the checks that failed. Abstention is the only honest output when a deterministic critic refuses to sign off.

Leaning into the bitter lesson

There is an objection worth naming. Symbolic programs, interpreters, structure imposed on generation, isn’t this exactly the hand-built scaffolding that the bitter lesson (Sutton, 2019) says always loses? Seventy years of AI history argue that methods leveraging human knowledge get overtaken by general methods that leverage computation.

Let’s look at what is being hand-built here: nothing about the domain. There are no rules about e-commerce analytics, no grammar of reports, no ontology of business questions. The entire human contribution is one general affordance: an interpreter. Everything task-specific regarding which columns matter, what to aggregate, what the prose should claim, is decided by the model, per query, in code it writes itself.

The bitter lesson’s actual claim is narrower than its reputation: the methods that win are the ones that scale with computation, and Sutton names two: search and learning. An interpreter is leverage of computation in its purest form: deterministic, exact, and effectively free next to a forward pass.

Why we build this way

Standard tool calling agents ask users to trust the LLM. Blended Generation asks the users to trust pandas, a trust people already extend.

The model’s role narrows to the thing post-training actually made it good at, namely deciding which computation answers the question, and writing it down as code. Everything that must be exact is delegated to the interpreter, and everything the interpreter produces is held to checks the model does not control: structural, metamorphic, provenance. The result is a deliverable whose correctness is a property of the program, inspectable after the fact, rather than a property of the sample, gone the moment it was drawn.

Oolong Technologies is an Applied AI Lab specialising in oracle-based verifiable generation. If your agents write numbers people act on, chat to us at hello@oolongtechnologies.io