Contextual Chunk Retrieval Without Special Embeddings

Agentic Memory Architecture — Part 1

My assistant forgot my dogs' names.

The names were stored. The recall system ran. Relevant memories existed. A focused search found them. But the actual voice prompt also contained a discussion about engineering, and the automatic retrieval returned engineering memories while missing the dogs.

Come on. I said dogs.

The question that followed was useful: why should every subject in a mixed prompt have to compete for the same retrieval slots?

And once we give each subject its own search, how do we preserve what that subject means in this particular context?

We tested a small piece of vector math using ordinary embeddings. It distinguished identical sentences stored in different contexts and improved correct-context ranking when unrelated text was added. Then we paired it with a simple retrieval rule: every query chunk gets its own top-X allowance above a relevance threshold.

I'm calling that combination contextual chunk retrieval here. It applies existing ideas with only a few moving parts. The weighted representation has been tested; the grouped retrieval policy is the architectural contract described below.

The two things a memory search needs to preserve

A conversation can contain several independent information needs. A person can discuss a deployment, ask about an appointment, and check whether the assistant remembers their dogs in the same turn.

One embedding compresses all of that into one search direction. A single top-X list then allocates the entire result allowance according to that direction. A subject can be present in the prompt and still receive no useful memory.

Splitting the query addresses coverage. Each chunk gets a search of its own.

But isolated chunks introduce a second problem: meaning. The same sentence can mean different things depending on the passage around it.

Consider this sentence:

We need to handle this email.

In one conversation, the draft is waiting for approval. In another, delivery has already been authorized. In a third, the recipient is uncertain. In a fourth, a tracking link needs correcting.

The email is the same subject. The sentence is identical. The intended work is different.

If we store only that sentence's embedding, all four occurrences have the same vector. A more detailed query cannot reconstruct the distinction we discarded at storage time.

So the representation needs two ingredients: the focused span and the passage that gives it its flavor.

What late chunking does

Jina's original late-chunking article describes the context-loss problem in document retrieval: a small passage may depend on a name, subject, or relationship established elsewhere in the document.

Late chunking encodes the larger text into contextual token representations first. It then pools the token representations belonging to each chunk. Context enters before the final chunk vector is produced. The research paper explains this change in the position of chunking relative to the transformer and mean pooling.

That is a strong mechanism: a token within the chunk can be represented in light of surrounding text before pooling.

It also requires access to that mechanism, either through the model or an API that supports it. A conventional endpoint that returns one final vector for each input doesn't expose the token representations needed to implement late chunking yourself. Jina makes that distinction in its follow-up article, and also separates boundary selection from context preservation.

Our question was narrower: can final vectors preserve enough contextual flavor to be useful, without changing the embedding service?

Blend the concept with its context

Let E(text) be an embedding from one model at one fixed dimension, and let unit(v) normalize a vector to length one.

For a stored focus span s and its surrounding passage C:

text
focus   = unit(E(s))
context = unit(E(C))
stored  = unit(alpha * focus + (1 - alpha) * context)

The context passage includes the focus span. It might be a paragraph, a short exchange, or a bounded section that establishes what the span refers to.

Apply the same construction to a query chunk q and the prompt context P:

text
query = unit(alpha * unit(E(q)) + (1 - alpha) * unit(E(P)))

We started with alpha = 0.8: 80% focus, 20% context. Both inputs are normalized before mixing so their relative magnitudes don't silently change the intended weight.

The stored result is still one vector of the original dimension. It can be searched with ordinary cosine similarity. The context embedding can be shared among several focus spans from the same passage.

The focus keeps the concept recognizable. The context distinguishes its occurrences.

This is a blend of two completed representations. It does not give the sentence's tokens another pass through the model, and it does not recreate late chunking's token-level conditioning. Whether it preserves the distinction you need is an empirical question. That's why we built the test.

There is prior work beyond late chunking

Weighted combinations of local and broader representations already exist. A particularly close example is MCRAH-GIE, published in Results in Engineering in June 2026. Section 3.3.5 describes combining chunk, paragraph, and document embeddings with weights of 50%, 30%, and 20%. Its embedding generator uses all-MiniLM-L6-v2. That is direct precedent for obtaining contextual representations by combining ordinary embeddings at different scales.

The query side also has close precedent. Korikov and colleagues' 2024 Aspect Fusion paper studies how a monolithic query can overrepresent some aspects. It explicitly seeks coverage of all relevant aspects and equal aspect frequency, and selects an equal number of reviews per aspect before fusion. Its final output is a ranked item list; our proposed output keeps the memory groups attached to their originating query chunks.

LangChain's MultiQueryRetriever generates several queries, retrieves for each, and returns their unique union. RichRAG explores sub-aspects, retrieves supporting documents, and uses a learned listwise ranker to assemble a final selection.

These are useful precedents. The application here is to combine weighted contextual representations with independent retrieval allowances for the actual chunks of an agent's incoming prompt. We don't need to claim we invented vector interpolation or multi-aspect retrieval to show why that application matters.

What the math actually adds

With both stored and query vectors using the 80/20 blend, their dot product has four contributions. Before the final normalization factors are applied, the numerator is:

text
0.64 * dot(query_focus, stored_focus)
+ 0.16 * dot(query_focus, stored_context)
+ 0.16 * dot(query_context, stored_focus)
+ 0.04 * dot(query_context, stored_context)

The dominant term matches the focused spans. Two cross terms connect each focus to the other side's context. The last term measures agreement between the surrounding passages. Final cosine similarity divides this sum by the lengths of the two mixed vectors.

That gives a small, explicit control over how much the surrounding passage can influence retrieval. It also explains why the same sentence can acquire different search directions when stored in different contexts.

This is the depth we're after: retrieving the intended relationship behind a statement. “Handle this email” can rise as an approved delivery task or as a request to keep a draft unsent, depending on the surrounding meaning.

A blend can only preserve distinctions its input embeddings capture. It still compresses the context into one vector. For memories that require following several graph relationships to discover the answer, graph traversal remains a separate capability.

Test the same sentence in different contexts

A useful test has to make the sentence alone insufficient.

We created 96 synthetic records across 24 sentence families. Each family contained the exact same focus sentence in four different contexts. Eight families were used for development; sixteen were reserved for evaluation, producing 128 context-specific evaluation queries and 16 generic queries.

The corpus had two independently searched blocks: a 64-record block including different meanings of terms such as “key,” “thread,” and “session,” and a 32-record block with different purposes for the same subject. The second block covered things like rotating, scoping, storing, or auditing an API key; compressing, resizing, redacting, or inspecting an image; and the four email contexts above.

The fixtures and target labels were fixed before embedding. Queries used independently written contextual descriptions. IDs and flavor labels were never embedded.

We used text-embedding-3-small at 1,024 dimensions and exact cosine search. There was no lexical search, graph walk, reranker, or approximate index in this experiment. We compared 15 representation variants under four conditions: clean text, distracting text in stored passages, distracting text in queries, and distractions on both sides. Each noisy input retained its relevant text and added five seeded, unrelated sentences.

The 80/20 weight was proposed before these fixtures were evaluated. A separate development-only selection slightly favored 95/5; it did not establish a universal best weight.

Here is correct-context top-1 accuracy on the 128 evaluation queries:

Representation	Clean	Stored distractions	Query distractions	Both distracted
Isolated sentence on both sides	25.0%	25.0%	25.0%	25.0%
Whole passage and whole query	100.0%	94.5%	57.8%	36.7%
80/20 blend on both sides	100.0%	96.1%	71.9%	50.8%

The isolated-sentence row has four tied candidates within each family. We assign a quarter of a top-1 success to each, rather than awarding a win based on arbitrary record order.

The meaningful improvement appears when unrelated material competes with the intended context: 92 correct first results instead of 74 with query distractions, and 65 instead of 47 when both sides were distracted. Both differences are 14.1 percentage points.

Controlled retrieval results comparing isolated sentences, whole-context embeddings, and weighted contextual blends, with a separate measure of generic-query coverage.

The clean whole-passage baseline already scored 100%. Weighting didn't improve a result that was already perfect. It made this test more resistant to distraction.

Generic queries contained only the shared focus sentence. With the 80/20 stored blend, all four contextual versions occupied the top four positions for every evaluation family, including the noisy stored passages: 64 of 64 relevant occurrences. A broad question could still find the subject across its contexts; a specific question could favor the intended occurrence.

The stored side mattered. Blending only the query, while keeping whole-passage stored vectors, scored 36.7% with query distractions—below the whole-context baseline's 57.8%. And a rich contextual query against four identical sentence-only stored vectors still produced a four-way tie in the clean condition. Query weighting is not a substitute for preserving context in the records.

The protocol, context fixtures, same-subject fixtures, and full rankings accompany this post. These numbers establish behavior on this controlled corpus. They are not a benchmark victory over late chunking, which we did not run, or a measurement of production response quality.

Give every query chunk its own group

The representation test answers whether contextual flavor can survive in ordinary vectors. The integration needs one more rule to address the original mixed-prompt failure.

For every chunk of the incoming prompt:

Build its weighted query vector using the surrounding prompt context.
Search independently, in parallel with the other chunks.
Keep up to X matches above a relevance threshold for that chunk.
Attach its selected memories under that chunk's group.

Engineering has its own group. Dogs has its own group. A higher engineering score cannot take a qualifying dog's slot.

For chunk i, the returned group is simply:

text
group_i = top_X(matches whose similarity to query_i >= threshold)

Every chunk receives the same allowance. A chunk with fewer than X qualifying memories returns fewer; we don't fill its group with weak matches. The threshold governs relevance within a group, and X governs its size.

Keep those groups through prompt assembly. A final global top-X cut would restore the competition we set out to remove. Cross-group deduplication is unnecessary for the first version; a memory that supports two chunks can appear in both.

This guarantees an independent opportunity for each chunk to contribute qualifying results. It doesn't guarantee that the chunker isolates every concept or that the relevant memory clears the threshold. Those are distinct questions to measure.

The accuracy table above measures contextual representations. Evaluating the grouped policy end to end also requires checking which selected memories actually reach the answering model.

How the approaches compare

The approaches put context into retrieval at different stages:

Approach	How context enters the representation	Required access or preprocessing	Result allocation
Isolated chunk embedding	Only the chunk's own text	Ordinary embedding endpoint	Chosen separately by the retrieval system
Late chunking	Larger-text token encoding before pooling each span	Token representations and pooling control, or a supporting API	Chosen separately by the retrieval system
Contextual Retrieval	Generated explanatory text prepended to each chunk	Context generation, then embedding and lexical indexing	Chosen separately by the retrieval system
Weighted contextual blend	Mix the focus vector with a passage vector after embedding	Ordinary final vectors from the same model and dimension	Chosen separately by the retrieval system
The application proposed here	Weighted contextual records and weighted query chunks	Batched embeddings and independent searches	Up to X qualifying memories per query chunk

Anthropic's Contextual Retrieval generates a short explanation of a chunk's place in its document and prepends it before embedding and BM25 indexing. That can make missing context explicit in text. Our blend uses the existing passage directly, without generating another explanation.

Late chunking can represent a span through tokens that have already interacted with their surroundings. Our blend can be implemented through a final-vector API. Neither choice determines whether the retrieval system preserves all the subjects of a mixed query. Independent groups can be applied to either representation.

The comparison is architectural. There is no shared dataset here that would justify ranking these approaches against each other by accuracy or speed.

Boundaries and performance

A sentence is a convenient boundary. It isn't a universal concept boundary.

“Do you remember their names?” depends on what came before it. A voice transcript can contain several subjects without a full stop. A paragraph can contain one coherent statement or three unrelated ones.

Start with small chunks based on available sentence and paragraph boundaries, with bounded windows and overlap where punctuation is weak. Use the larger passage as context. Better concept boundaries can improve the system later; they don't require another model call before every voice response. Our test used manually chosen focus spans, so it doesn't establish the quality of an automatic chunker.

At storage time, build contextual vectors in the background. For a passage containing several chunks, embed the passage once and reuse its context vector. Store the blended vectors with their source occurrences so identical text in different settings remains distinguishable.

At query time, batch the prompt context and chunk embeddings, do the arithmetic locally, and execute the searches in parallel. There is no additional generative step required to explain or rerank the chunks.

That keeps the path short, but it doesn't make extra work free. More chunks create more embedding inputs and searches. Parallel search depends on available concurrency. Returning X memories for each of m chunks can attach up to m * X memories, increasing the answering model's input cost and processing time.

A latency claim needs measurements under load, including recall p95 and time to first audio. A global result cap may control cost, but it changes the equal-representation contract. If we need smaller groups, reduce X consistently rather than letting one subject consume another's allowance.

Thresholds also need evaluation with the representation and score scale actually used. In the clean email example, the 80/20 blend scored the correct delivery record at 0.993, but the other three contextual versions scored between 0.975 and 0.984. A score close to one is not a probability of correctness. Strong focus weighting can keep a subject's variants close together while still ordering them correctly.

What we're taking forward

The most useful result is concrete: ordinary embeddings can distinguish an identical statement used in different contexts, while keeping its other contextual versions retrievable. In this controlled test, weighted blends also improved first-result accuracy when unrelated text was present.

The strongest baseline still won some individual cases. The blend mistook a conversation-sharing request for summarization, and a customer-record restriction request for correction under query noise. Neither 80/20 nor 95/5 is a universal setting.

The application remains worth building: preserve the statement's flavor at storage time, give each incoming chunk its own contextual query, and keep its qualifying results intact through delivery to the agent.

The dogs get a search of their own. And the memories that answer it reach the prompt.

Extractive Summarization with Jev

Our Most Effective Jev Use Case: Walking Agent Memory