TokenOvenALTC v0.1
ACTIVE LLM TOKEN COMPRESSION (ALTC)|Method Family
Critical Fact Loss = 0
TokenTumbler & Verdict Ready

Bake Context Down to Pure Signal.

TokenOven is the active context-governance engine that sits between your agent and frontier LLMs. We compress bloated agent traces, multi-file codebases, and RAG bundles by 40%–80% while mathematically guaranteeing that hard constraints, file paths, identifiers, and benchmark numbers are 100% preserved.

Launch The Oven Simulator
Token Savings
40–80%

On long-running agent & RAG traces

Critical Loss
0.00%

Paths, constraints, & numbers intact

Fast Compactor
< 8ms

Deterministic dedupe & aliasing

Prefix Cache
100% Hit

Static system/tool prefix protection

The Core Inequality: Costcompression + Costcompressed_inference < Costraw_inference
INTERACTIVE OVEN SIMULATOR

See Context Compression in Action

Select a real-world trace, switch compression modes, and inspect the side-by-side prompt diff, live BakeReport, and reversible OvenHandles.

Select Trace:
Target Model:
Compression Mode:
Raw Input
52,400
Input tokens
Baked Context
19,410
Saved 62.96%
Fact Loss
0.00%
6 critical facts pinned
Cost Delta
-$0.099
per request saved
Prefill Speedup
+264ms
Faster time to first token
Raw Uncompressed Context
52,400 tokens
[SYSTEM] You are a senior runtime systems engineer inside Verdict PAS (Programmable Agent Swarm). Strict Policy: Never mutate registry.py directly. Acceptance gate: BCB-100 must exceed 50/100. Tool schemas: [34 distinct tools with verbose JSON definitions, 8,200 tokens] [ARCHITECT TO DIRECTOR] We evaluated 4 repair options for failure #402 at /Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict/src/core/router.py. Option A: Rewrite the routing table (Rejected: breaks legacy backwards compatibility). Option B: Patch the dispatch regex (Accepted). Option C: Invalidate cache keys (Rejected: causes 400ms latency spike). Option D: Full retry loop (Rejected: costly). [DIRECTOR TO PROGRAMMER] Task: Patch dispatch regex in /Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict/src/core/router.py. Hard Constraint: Keep time-to-first-token under 4.0 seconds. Benchmark history: - Run 2026-05-01: BCB-100=33/100, syntax-recovery subset +14.8pp, total gate failed by -20pp. - Run 2026-05-02: BCB-100=38/100, syntax-recovery subset +19.2pp, total gate failed by -15pp. Test failure stack trace: AssertionError: Port mismatch at /Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict/tests/test_router.py line 144. Expected 8080 got 8000. ... [420 lines of redundant stack trace and verbose build logs omitted] ... [USER CORRECTION - LATEST] User: "Make sure you do NOT touch /Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict/src/core/auth.py either, and verify port 8080 is strictly enforced."
⚠️ Bloated with redundant schemas, repeating paths & logsUnoptimized
TokenOven High-Density Context
<|to:v1 id=to_bal_01 mode=bal raw=52400 cmp=19410 sha=8fa1b2|> $ROOT=/Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict $CORE=$ROOT/src/core [SYSTEM: PROTECTED CACHED PREFIX] Runtime Systems Engineer. Gate: BCB-100 > 50/100. Pinned policy intact. [CRITICAL PINNED CONSTRAINTS] 1. Never mutate $ROOT/registry.py 2. Never mutate $CORE/auth.py 3. Port must strictly be 8080 (Failed assertion at $ROOT/tests/test_router.py:144) 4. TTFT budget: < 4.0s [CURRENT DIRECTIVE] Patch dispatch regex in $CORE/router.py to resolve port 8080 mismatch. [REVERSIBLE OVEN-HANDLE] <|to_mem id=bench_2026_05_02 kind=benchmark_summary raw=stored|> BCB-100=38/100; syntax subset +19.2pp; prior run 33/100 (+14.8pp). Total gate delta -15pp. <|/to_mem|> <|/to:v1 id=to_bal_01|>
FIDELITY VERIFIER: 100% PRESERVEDRisk Score: Low (0.00)
Never mutate registry.py directlyDo not touch auth.pyEnforce port 8080 strictlyBCB-100 gate > 50/100TTFT latency < 4.0sTarget file: src/core/router.py
CONTEXT GOVERNANCE PILLARS

Why This Is Not Just Summarization

Standard summarization optimizes for human reading prose and destroys machine task state. TokenOven is an active context governor designed for autonomous agent execution.

Rule of Zero-Loss

Operational Fact Preservation

Naive summarizers casually drop critical file paths, error codes, and acceptance gates. TokenOven treats context as structured state: hard constraints, numeric results, and identifiers survive with 100% mathematical fidelity.

Production Ready·Zero Hallucination
Reversible Memory

OvenHandles (<|to_mem|>)

Transform compression from destructive deletion into a tiered memory hierarchy. Raw traces and voluminous documents are indexed into high-speed memory and replaced with lightweight recall handles that can be rehydrated on demand.

Production Ready·Zero Hallucination
100% Cache Friendly

Cache-Aware Prefix Assembly

Standard compression breaks prompt caching by altering initial tokens. TokenOven keeps system instructions and tool schemas 100% byte-static, placing compressed context strictly past the provider's cache boundary.

Production Ready·Zero Hallucination
< 8ms Latency

Deterministic Fast Compactors

Before invoking any model-based compression, TokenOven executes sub-10ms deterministic deduplication, symbolic path aliasing ($ROOT=...), schema compaction, and log stripping with zero external compute cost.

Production Ready·Zero Hallucination
BPE & SentencePiece

Tokenizer-Aware Optimization

Different tokenizers count the same text differently. TokenOven optimizes specifically for the target model's tokenizer (OpenAI o200k, Anthropic BPE, Qwen/Llama SentencePiece) to squeeze out maximum token efficiency.

Production Ready·Zero Hallucination
Agent Multi-Turn

Context Governance vs Summarization

Built specifically for long-running agent swarms (like Verdict PAS) and enterprise RAG. TokenOven pins active task state, eliminates amnesia loops, and stops context rot during multi-agent handoffs.

Production Ready·Zero Hallucination
THE TRI-SYSTEM ARCHITECTURE

How TokenOven, TokenTumbler & Verdict Connect

Each product stands alone with its own market interface, while recursively reinforcing one another to deliver the fastest, cheapest, and highest-IQ inference pipeline in AI.

Flagship Environment

Verdict

Autonomous Coding & Swarm Platform

The battle-tested execution environment. Uses TokenOven inside PAS agent loops (Architect $\to$ Director $\to$ Programmer $\to$ Judge) to prevent context amnesia and stop context rot.

Commercial Advantage:

Exclusive and unlimited TokenOven compression is included natively with Verdict Enterprise tiers.

PAS · VAK · SAM · CrucibleActive
Broker & Router

TokenTumbler.ai

Smart Orchestration Broker (SOB)

Evaluates every request, intercepts prompts $\ge$ 4k tokens, and calls TokenOven to compress context before tumbling requests across Mode 1 (Self-SOB) through Mode 4 (Premium Apex).

Commercial Claim:

“Tokens In, Smarter Answers Out.” By compressing first, Tumbler cuts Mode 3 costs from $5.31 to under $2.00 per 1k tasks.

BYOK · Mode 1–4 Laddertokentumbler.ai
Core Engine

TokenOven

Active LLM Token Compression (ALTC)

Stands alone as a hosted API (`api.tokenoven.com`) or local daemon. Ingests context, pins non-negotiable facts, generates OvenHandles, and outputs verified high-density prompts.

Universal Tagging:

TTCP Tagged Context Protocol passes seamlessly through standard OpenAI and Anthropic compatible interfaces.

ALTC · TTCP · OvenBenchtokenoven.com
End-to-End Orchestration Flow
Client Prompt -> [TokenTumbler Broker] (Inspects tokens, calculates ROI)
                     |
                     v (If prompt >= 4,000 tokens)
              [TokenOven ALTC Engine]
                     |-- Pin Hard Constraints & File Paths (0% loss)
                     |-- Deduplicate Schemas & Logs (< 8ms)
                     |-- Replace Heavy Traces with OvenHandles (<|to_mem|>)
                     |-- Cache-Aware Stable Prefix Alignment
                     v
              [High-Density Compressed Payload (40-80% smaller)]
                     |
                     v
              [TokenTumbler Model Ladder]
                     |-- Mode 1: Local Qwen 2.5 Coder ($0.00 / LAN)
                     |-- Mode 2: Selective Rescue (Failed Assertions)
                     |-- Mode 3: Frontier-Lite (Peak ROI Hybrid)
                     |-- Mode 4: Premium Apex (Consensus Swarm)
                     v
              [Upstream Providers: Claude 3.7 / GPT-4o / DeepSeek R1]
                     |
                     v
Client Receives Accelerated Response + BakeReport Telemetry Header
OVENBENCH EVALUATION STANDARD

Proven Zero-Loss Preservation

The core difference between ordinary summarization and TokenOven: summarizers lose 14–30% of critical constraints and numbers. TokenOven maintains a strict 0.00% loss rate.

Benchmark SuiteRaw TokensNaive SummarizerTokenOven (balanced)Constraint LossAccuracy Impact
Verdict PAS Agent Handoff
50k Context		52,400
23,100 tokens
21.4% constraints dropped
19,410 (-63.0%)
Preserved all critical state
0.00%+2.8%
Signal amplified
Constraint Retention Bench
32k Context		32,000
14,200 tokens
18% constraints dropped
11,800 (-63.1%)
Preserved all critical state
0.00%+0.0%
Signal amplified
Multi-Needle Retrieval (10 needles)
64k Context		64,000
26,000 tokens
30% constraints dropped
21,500 (-66.4%)
Preserved all critical state
0.00%+4.1%
Signal amplified
Codebase Context & Stack Traces
40k Context		41,200
18,000 tokens
15.5% constraints dropped
15,100 (-63.3%)
Preserved all critical state
0.00%+3.4%
Signal amplified
Enterprise RAG Bundle (20 chunks)
28k Context		28,500
12,100 tokens
14% constraints dropped
8,820 (-69.1%)
Preserved all critical state
0.00%+5.2%
Signal amplified
Tested on LiveCodeBench, RULER, and Verdict PAS multi-agent traces.Evaluated with N=100 runs per benchmark family
TTCP SPECIFICATION

TokenOven Tagged Context Protocol

Provider-neutral tags that pass cleanly through standard OpenAI and Anthropic endpoints without requiring custom server modifications.

<|to:v1 id=to_0001 mode=bal raw=18420 cmp=7210 sha=9f8a31|> $ROOT=/Users/trentcarter/Artificial_Intelligence/AI_Projects/Verdict $CORE=$ROOT/src/core [PINNED CONSTRAINTS] - Port 8080 mandatory - Do not mutate $CORE/auth.py [ACTIVE TASK STATE] Target file: $CORE/router.py. Patch dispatch regex to fix test assertion failure. <|/to:v1 id=to_0001|>
Adds only ~10 tokens overhead per compressed blockRFC-TTCP-v1 Draft
DROP-IN INTEGRATION

One Line to Slash Your Token Bill

Wrap standard OpenAI clients, call our REST endpoint directly, or enable TokenOven inside TokenTumbler.

# pip install tokenoven openai from openai import OpenAI from tokenoven import wrap_openai # Wrap any standard OpenAI client with TokenOven ALTC compression client = wrap_openai( OpenAI(api_key="sk-..."), oven_api_key="to_live_sec_...", mode="balanced", # "safe" | "balanced" | "aggressive" cache_friendly=True # Protects static system/tool prefix ) # Call completions normally; TokenOven compresses pre-flight response = client.chat.completions.create( model="gpt-4o", messages=large_multi_agent_history, stream=True ) for chunk in response: print(chunk.choices[0].delta.content or "", end="") # Inspect live telemetry from response headers print(f"\n[BakeReport] Saved {response.bake_report.savings_pct}% tokens!")
Available on PyPI and npm · Zero breaking changesv0.1.0-alpha