When a Tool Result Is Too Big for the Context Window

Store the full result, keep a whole-record preview in the context, and hand the model a reference it can open. How that works for list results, for the saved conversation, for the model request and across agent hand-offs.

Post #25 showed the conversation manager trimming oversized tool results, so that one long run would not drag every earlier result into every later model call. That was the right protection for the window. It also left a question I had been avoiding: what does the user get when they ask for more data than a window can hold?

The request that forced the question was an ordinary one. A user wanted a list-style tool to return up to a thousand rows, and every row when asked. A thousand structured records is far more than belongs in a prompt. Whatever the cap decided, the model would be answering from part of the data while the user believed it was answering from all of it. And the number it would report was the worse problem: a count taken from the rows that fit is the size of the preview, not the size of the result. It is the same family of error as the counter that measured writes instead of entities in post #26.

So the design question changed. It stopped being "how big should the cap be" and became "where does the full result live, and what does the model hold instead?" The answer I settled on is to pass big results by reference.

This post walks through that pattern in four places: list results, the saved conversation, the model request itself, and the hand-off between agents. It ends with a sidebar about a list read that quietly hid chats, because it is the same size problem arriving from the other direction.

Why Raising the Cap Is Not an Escape

The tempting answer to "the cap is too small" is a bigger cap. It fails for reasons that have nothing to do with the model call.

• The chat is stored as one item. Each conversation lives in a single DynamoDB item, and DynamoDB items cannot exceed 400 KB. I keep a fixed margin under that limit for attribute overhead, so a result that is merely "large" competes with every other turn for the same space. An item over the hard limit does not degrade; the write fails.
• Every later turn re-sends it. History is part of the next request. A big result is paid for again by each follow-up question, which is the quadratic bill described in post #25.
• More inline is not better inline. I kept the inline part deliberately small, on the bet that a few dozen well-chosen records give a cleaner answer than thousands skimmed, with the complete set a click away.

Export by Reference: Anatomy of a List Result

Every tool that can return a list uses one shared writer. Rows are streamed from the query (on a server-side cursor when the source is a database) into a spooled temporary file as CSV. While the rows go by, the writer also builds the inline preview. When the rows run out there is one upload, and the tool returns a payload with four kinds of information:

A result then looks like this, with N the number of matches and k the number of rows that fit inline:

{
  "download_url": "<link to the per-user CSV>",
  "total_matching": N,
  "rows_exported": N,
  "rows_inline": k,
  "truncated": false,
  "inline_truncated": true,
  "data": [ ...the first k rows... ]
}

The true total comes from the database

The total is a window function, evaluated over the full match before LIMIT is applied. Every row carries it; the writer reads it off the first row and strips it from the output.

SELECT id, score, status,
       COUNT(*) OVER () AS _total   -- over the whole match, before LIMIT
FROM   results
WHERE  ...
ORDER  BY ...
LIMIT  :n;

The agent is also told, in the tool description and in the instructions for the agent that writes the answer, to report counts from the total and never from the number of inline rows. A rule in the prompt is not enough on its own, but it is cheap, and it matches what the payload makes easy.

Contiguous means no gaps

The inline prefix stops at the first row that does not fit. It does not skip that row and keep packing smaller ones behind it.

def inline_prefix(rows, budget, serialize=json.dumps):
    kept, used = [], 0
    for row in rows:
        size = len(serialize(row)) + 2        # the ", " between list items
        if used + size > budget:
            break                              # first misfit ends the prefix
        kept.append(row)
        used += size
    return kept

Packing is tempting because it shows more rows. It also means the model sees rows 1 to 30, then 33 and 34, and happily describes a contiguous run it never had. Two details keep the budget honest: the size is measured with the same serialiser used for the final payload, so the budget is counted in the bytes the model will actually receive, and the preview is a prefix of the query order, so "the first rows" stays true.

What the two flags buy

The CSV holds up to the row limit the caller asked for, so the first flag says whether the file itself is short of the total, for example because the request asked for fewer rows than matched. The second says whether the inline preview is short of the file. With both, the model can say "these are the first few dozen rows; the download holds the rows you asked for, out of a larger total" instead of guessing which of the three numbers it is looking at.

The CSV Is a Download, So It Gets a Download's Security

Once a result becomes a file that people click on, the questions change from "is it too big" to "who may fetch it, and what happens when they open it".

• Identity comes from the request, not from the model. The writer takes the owner from the authenticated request context. The model never passes an identity argument, so it cannot be talked into exporting to someone else. With no authenticated user there is no export at all, because nobody could download it.
• The key is under the owner. The object key contains the lower-cased owner identity, and the download route builds the key from the caller's own identity. Someone else's export id simply resolves to nothing. It is a 404, not a 403, so the route never confirms that an id exists.
• The id is validated before it touches a key. A strict 32-character hex pattern, nothing else.
• The role check happens at download time. The producing tool is recorded as object metadata when the file is written. The route runs that tool through the same role gate the tool itself is subject to, using the caller's current roles. A role revoked after the file was written takes effect when the link is clicked. An unrecognised tool type is refused, so a new tool has to be added to the allow-list on purpose.
• Cells are guarded against formula injection. Data that came from external sources can start with a character a spreadsheet treats as a formula. The writer prefixes such string cells with a literal quote. Numbers stay numbers, so a negative value is not turned into text.

def formula_guard(cell: str) -> str:
    # =, +, -, @, TAB and CR start a formula in most spreadsheet apps
    if cell and cell[0] in "=+-@\t\r":
        return "'" + cell
    return cell

Giving the Conversation the Same Treatment

List tools are the easy case because they know they are producing rows. A schema dump, a search result or a status listing can be just as large, and nobody writing those tools thought about it. So the same idea is applied generically, by a hook that runs after every data tool call:

1. If the call succeeded and the result is above a small floor of about a kilobyte, the full result is written to storage under the owner and the session, with the producing tool recorded as object metadata. Images are dropped from the copy.
2. A second content block carrying a data reference is appended to the result. The first block is never touched, because several parsers read only the first block.
3. The hook is an async callback and the upload runs in a bounded worker pool. The framework awaits only async callbacks; a synchronous one runs on the event loop, and blocking storage calls there are how a service ends up killed by its own liveness probe (the failure mode post #25 describes). The hook never raises, so an export problem cannot fail a tool call.
4. The preview is computed from the full result at the moment the tool returns, not later from the message list. The message list is mutated in place by the conversation manager after the loop, so anything derived from it afterwards is derived from an already changed object.

When the turn is saved, the stored copy of a large result is the preview plus a marker block:

[full result stored: ref <id>, <bytes> bytes,
 available until <date>; open it with the analysis tool]

Two properties make the marker safe to rely on. It is written only for a copy that is confirmed in storage: persistence waits, off the event loop and for a bounded time, for the upload behind each result, and a pointer to nothing is worse than no pointer. And it carries a date, derived from the retention setting, so a marker is trusted only while that date has not passed.

Cutting a preview without breaking the data

A preview that ends in the middle of a record is another way to hand the model broken data. For JSON results the preview is cut at the last complete element of the shallowest open array that holds a comma, which is normally the list of rows. The open brackets are then closed, so the result parses and consists of whole records.

TOKEN = re.compile(r'"[^"\\]*(?:\\.[^"\\]*)*"|"|[\[\]{},]')

def cut_at_record_boundary(head: str):
    stack = []                       # [closer, position of the last comma at this level]
    for m in TOKEN.finditer(head):
        tok = m.group()
        if tok == '"':               # unterminated string: stop here
            break
        if tok in "{[":
            stack.append(["}" if tok == "{" else "]", None])
        elif tok in "}]":
            if stack:
                stack.pop()
        elif tok == "," and stack:
            stack[-1][1] = m.start()
    # prefer the shallowest open array holding a comma: the cut falls between rows
    pick = next((i for i, (c, p) in enumerate(stack) if c == "]" and p is not None), None)
    if pick is None:
        pick = next((i for i, (c, p) in enumerate(stack) if p is not None), None)
    if pick is None:
        return None
    return head[: stack[pick][1]] + "".join(c for c, _ in reversed(stack[: pick + 1]))

Opening the reference

When the model needs every row, it passes the reference to an analysis tool that runs a script in a sandbox against the stored copy, so it analyses or charts the exact data instead of retyping a preview. The reference carries the session segment, so the lookup goes straight to one object. It is parsed with a strict pattern, resolved under the caller's own prefix only, and the producing tool is re-checked against the caller's current access before the sandbox is allowed to read it. A string supplied by a model is input like any other.

A Safety Net Before the Model Call

In mid-October a single status-listing tool returned about 14.7 MB. The request carrying it was rejected as too long: roughly 8 million tokens against a 1-million-token window. Nothing sat between that tool result and the model call. The cap from post #25 runs after the loop, and the delegate agents run without a trimming conversation manager, so there was no bound to hit.

The guard that followed uses the same by-reference idea, as a wrapper around the model-invocation step:

• While the estimated request fits, it does nothing at all and returns the very same context object.
• When the estimate passes a configurable share of the window (85% by default), it writes the full text of the largest tool results to storage, confirms the writes, and hands the model a whole-element prefix plus the stored-result marker.
• If the model still raises an overflow error, it offloads more and retries the call exactly once.

Three details are worth copying.

The estimate is deliberately pessimistic. It divides bytes by a bytes-per-token ratio of 1.84, which is what the incident measured for JSON-heavy output (that 14.7 MB came to about 7.97 million tokens). Prose is closer to 4 bytes per token. Using the low JSON ratio over-counts prose, so the guard may act a little early and never late.

The replacement text is stable. Each offloaded result's replacement is computed once per run and cached, the set of offloaded results only grows, and the replacement is byte-identical on every later model call. That keeps the prefix the provider caches stable, which matters because a guard that rewrites history differently on each call would turn every call into a cache miss. Largest-first, with the tool-use id as tie-break, keeps the choice deterministic.

It works on a copy. The middleware receives a deep copy of the messages. The live message list, and everything built from it (charts and tables in the UI, the grounding check, knowledge-graph harvesting), still sees the full result.

The Link Has to Survive the Hand-Off

A big result rarely reaches the user from the agent that fetched it. In a multi-agent flow a delegate fetches, another agent analyses, and a final agent writes the answer. Each hand-off is a place for the link, the total or the flags to be paraphrased away. Four things keep them intact:

1. The delegate is told to copy the fields verbatim, at the top. The instructions for the data-fetching delegate say to pass the link, total and flags through unchanged before any table or summary. A hand-off that is later shortened for the next stage keeps its beginning, so the top is where the fields have to be.
2. The reference block is separated before any size budget is applied. The data reference is split off the result body and carried as its own field, so no later budget on the body can touch it.
3. The final writer is told what to do with it. Its instructions say to render the download link verbatim as a link, to take counts from the total, and never to imply that the inline rows are all the rows.
4. Reference rather than retelling. When the delegate's own text is shortened for the final writer, the full text is stored, under the same ownership and role checks as the CSV, and a reference to it is attached to the response next to the answer. It reaches the user without passing through the model.

Sidebar: A List Read That Hid Chats

The same size problem showed up on the read side. A user reported that chats were "not saved after a certain date". The writes were fine. The sidebar was showing 40% of one heavy user's chats.

The cause was the page size of the list call. A DynamoDB Query or Scan returns at most 1 MB per page, and that limit is measured on the items read before ProjectionExpression is applied. Projecting only the sort key does not shrink the page. Chat items can be large, so one page covered only a handful of them, and the code never looked at LastEvaluatedKey. Because the sort key was a random identifier, the hidden chats were scattered alphabetically, which looked like a gap in dates.

items, kwargs = [], {"KeyConditionExpression": "pk = :pk", ...}
while True:
    page = table.query(**kwargs)
    items.extend(page["Items"])
    if "LastEvaluatedKey" not in page:
        break
    kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"]
# filter hidden and expired chats AFTER every page has been read

Two lessons came out of it. When users say records are missing, compare a paginated count against what the endpoint returns before touching the write path. And filter after you have read every page: a FilterExpression is applied per page after the read, so a short or even empty page does not mean you are done; keep following LastEvaluatedKey either way. Filtering client-side after the last page keeps the loop simple.

Where the Limits Hide

What I'd Build Next

Today the model sees a preview and a total. The next step is a short, code-computed profile of the full result stored beside the preview: column names and types, minimum and maximum for numeric columns, distinct counts for categorical ones, and null rates. It would be accumulated in the same pass that writes the CSV, so it costs one extra accumulator per column and no second read. With it the model would know the shape of the part it cannot see, whether the first rows are typical or the tail looks different, without opening the file. It also keeps the rule from post #34: code owns every number, and the model writes the words around them.

Disclaimer: The views and opinions expressed in this post are my own and do not represent those of my employer or any organisation I am associated with.