Extractive Summarization with Jev

Jev extracts facts from text by selecting sentences out of it. It never writes a summary.

The method has two independent steps. Extraction finds every complete fact in a body of text, and it needs nothing but the text. Ranking orders those facts against a reference you choose: a question, a task, a policy, a customer's problem. You can swap the reference, or skip ranking altogether, without touching extraction.

We built this to compress our agent's context, and that's the worked example in this post. The extraction itself has no idea it's working for an agent.

Extractive vs abstractive

There are two ways to shorten text.

We measured this on our own assistant, which used to write its own facts before old conversation left its context. We spot-checked nine of those summaries against their source, and four were thin or off. Three left out an asset or task ID that later work depended on. One missed the decision the section existed for. One of those also added a detail the source didn't contain.

Extraction works on any text

Pass 1 takes text and returns the complete facts in it, verbatim. It never sees a task, a question or a user, so the same pass serves any job where you need facts out of a large body of text:

• contracts and leases, for obligations, dates and amounts;
• support tickets, for versions, error messages and reproduction steps;
• incident timelines and logs, for what happened and when;
• meeting transcripts, for decisions and owners;
• research papers and filings, for claims and figures;
• product reviews, for specific complaints.

Ranking is where the use case comes in. Rank contract facts against "what does the tenant owe on move-out", support facts against the bug you're triaging, review facts against one product feature. Or don't rank, and keep the facts in source order. The two steps share nothing but the list of facts.

What Jev is

• noul returns a number from 0 to 1 for a yes/no judgment.
• choice returns one of N labelled levels, with a probability for each level.

A call carries up to a few hundred questions against one state. That makes "ask the same question about every sentence" cheap.

Pass 1: extract facts as facts

The input is text and nothing else. The output is a list of complete facts, each one a verbatim span of the input.

text
One sentence is marked with double brackets in this passage from the chat
session, said by the assistant: "...[[The invoice went to Dana on March 3.]]
She paid it the same day..."
Is the marked sentence the first sentence of a whole and complete fact?
A fact must begin and end in a complete sentence, even if facts overlap.
Example where it is: "Ok thanks. [[The invoice went to Dana on March 3.]] She paid it the same day."
Example where it is not: "Ok thanks. The invoice went to Dana on March 3. [[She paid it the same day.]]"

Why boundaries instead of a summary

An abstractive model decides what a fact says. This method only lets Jev decide where a fact starts and where it stops. The words come from the source.

Boundary questions are also small judgments. A sentence plus its neighbours fits easily in one question, so every sentence gets asked in parallel. Jev decides each span's length too. A fact can be one sentence or four, depending on how the source wrote it, not on a fixed window.

ts
// Pass 1 — text in, verbatim facts out. No task, no question, no reference.
function extractFacts(text, { isCandidate = (s) => true } = {}) {
  const facts = []
  for (const chunk of chunkBy(text, 25_000 * 3 /* bytes */)) {
    const sentences = splitSentences(chunk).filter(isCandidate)   // the rest stays as context

    const [begin, end] = await jev.ask(chunk, sentences.flatMap((s, i) => [
      noul(`begin_${i}`, mark(s, window = 3), IS_FIRST_SENTENCE_OF_A_FACT),
      noul(`end_${i}`,   mark(s, window = 3), IS_LAST_SENTENCE_OF_A_FACT),
    ]))

    const spans = []
    for (let b = 0; b < sentences.length; b++) {
      if (begin[b] < 0.3) continue
      for (let e = b; e < b + 4 && sameBlock(b, e); e++)
        if (end[e] >= 0.3) spans.push({ b, e, text: join(sentences, b, e) })
    }

    const whole = await jev.ask(chunk, spans.map((s, i) => noul(`w${i}`, s.text, IS_A_WHOLE_FACT)))
    const scored = spans.map((s, i) => ({ ...s, score: whole[i] }))

    // Pop: best span first; a span that touches a taken sentence is skipped.
    const taken = new Set<number>()
    for (const s of scored.sort((x, y) => y.score - x.score)) {
      if (s.score < 0.5 || range(s.b, s.e).some((i) => taken.has(i))) continue
      range(s.b, s.e).forEach((i) => taken.add(i))
      facts.push(s.text)
    }

    for (const line of structuredLines(chunk))             // logs, dumps: per line
      if ((await wholeFact(line)) >= 0.5) facts.push(line)
  }
  return facts
}

Pass 2: rank against a reference

ts
// Pass 2 — reorder facts against any reference. Nothing is cut.
function rankFacts(facts, reference) {
  const state = `${reference}\n\nFacts:\n${numbered(facts)}`
  const answers = await jev.ask(state, facts.map((f, n) =>
    choice(`r${n}`, `Against this population of facts, how relevant is fact number ${n + 1} ` +
                    `to the reference: "${f}"`, RELEVANCE_1_TO_5)))
  return facts
    .map((f, n) => ({ f, score: expectedLevel(answers[`r${n}`]) }))   // Σ level × p(level)
    .sort((a, b) => b.score - a.score)
    .map(({ f }) => f)
}

The reference can be anything you can write down. You could also change the question itself, for example to importance, recency or risk. The extraction doesn't change.

Our use: compressing an agent's context

Our assistant runs in long-lived conversations. When the context gets too big, two things have to leave live context. One is a run of large tool outputs inside a single turn. The other is the oldest sections of the conversation. Before they go, we keep a list of facts from them.

• Extraction runs on the tool outputs or the section's turns. Assistant sentences and tool-output lines are the candidates.
• Our reference is the agent's current task: the latest user prompt, plus about 20k tokens of the chat history before it. Facts that matter to the task in progress come first.

We first tried a cut instead of a ranking. We kept a fact only when Jev put at least 70% on the top relevance level, then at least 50%. On tool outputs from the current turn, the cut kept the right facts. On older history sections it kept nothing. Those sections covered a different topic from the current task, so every fact scored low against it, and a section would have been sealed with an empty fact list. A ranking keeps every fact and orders them. That lesson applies to any reference: a cut tied to one reference throws away facts that a different reference would need.

There's an operational reason too. When the assistant writes its own facts, it does so inside its own loop. That costs turns, and the assistant misses the format often enough that we built a system error and a retry around it. Jev runs off the main thread, and a case takes under a second.

Numbers

We ran both passes on eight real cases from our agent, read-only, against production data: four runs of tool outputs and four history sections.

The three asset listings yield one fact, because a listing of file links contains no complete sentences. That's the result we want.

Ranked against the task, the facts the next step depended on came first: locked constraints, a run's outcome, and the IDs of what it saved. Platform boilerplate ended up at the bottom.

What's still open

• Pass 1 lets some noise through. In our material, tool descriptions and API table rows occasionally pass the whole-fact check, because they really are complete statements. Ranking pushes them down. A noise pass that asks whether a line is the platform describing itself is next. It worked in our September experiments.
• Near-duplicates survive. The same sentence can arrive twice, once with escaped line breaks and once with real ones.
• Prose. The assistant's own notes carry prose that ties facts together, and Jev's list doesn't. We log every checkpoint, whether the assistant wrote the note or Jev did. If Jev's facts hold up as well over time, extraction moves to Jev entirely, and the assistant stops doing housekeeping.

Sources

• TypeSafe AI — Introducing System One Models and Jev: https://typesafe.ai/blog/introducing-system-one-models-and-jev
• Part 1 of this series — Contextual Chunk Retrieval Without Special Embeddings: https://apifunnel.ai/blog/contextual-chunk-retrieval
• Our Most Effective Jev Use Case: Walking Agent Memory: https://apifunnel.ai/blog/our-most-effective-jev-use-case-walking-agent-memory