Jevstiller

Drop-in proxy for Jev · v0.6.0 · alpha

Jev, distilled
on the fly.

Point your services at Jevstiller instead of Jev. It learns each repeated question from Jev's own answers, proves it agrees with Jev within a budget you set, then answers most requests itself in milliseconds.

$ get startedread the guarantee
jevstiller serve · :8080task support_router · student:v10
local 70.7%agreement 99.45%audit 2%p50 16 ms

One container, one env var

Services keep the TypeSafe SDK and their own Jev keys. Only the base URL changes.

# run it next to your services
docker run -d -p 8080:8080 -v jevstiller-data:/data ghcr.io/tomerglick57/jevstiller:0.6.0

# in each calling service; nothing else changes
export TYPESAFE_BASE_URL=http://jevstiller:8080

How it works

Same call. Same answers. Your hardware.

Every Choice question in the traffic becomes a task. At first every request is forwarded to Jev unchanged. From Jev's answers, with their full probability distributions, a small local model is trained, shadow-tested and promoted. After that the proxy answers what it is sure about, in Jev's exact response format, and forwards the rest.

your servicesunchanged SDKjevstillerone process · many tasks and tenantslocal modelmost requests~16 msJevunsure, novel, audit~300 ms
answered locallyforwarded to Jev

Measured, live

One task learned from Jev, while serving traffic

Banking77, 77 customer-support intents, replayed through Jevstiller against live Jev on CPU with the default encoder. Both lines are measured on 2,000 held-out messages labelled by Jev.

One task, learned live from JevOver 11,083 messages, the share answered locally rises from 0% to 70.7%, while agreement with Jev stays between 99.25% and 99.75%, above the 98% target.learningall to Jev0%25%50%75%100%70.7% answered locallyAnswered locally, without calling Jev97%98%99%100%target 98%99.45% agreementAgreement with Jev, on 2,000 held-out messages02,5005,0007,50010,000messages seen
70.7%
answered locally by the end of the replay
99.45%
agreement with Jev, against a 98% target
16 ms
median local answer, on CPU. Jev: ~300 ms
78.5%
accuracy on the dataset's own labels, the same as Jev's

Hover the chart for the values at each checkpoint. How it was measured →

The contract

You set one number. The threshold is derived, not tuned.

target_agreement = 0.98
disagreement budget2% of all requests
confidence95%, exact (Clopper–Pearson)
calibrationheld-out, IID, strictest threshold first
audit2% of live traffic, forever
if brokenfall back to Jev, retrain
  1. 01The routing threshold is chosen so that, with 95% confidence, the share of requests answered locally and differently from Jev stays within the budget.
  2. 02A random 2% of requests keeps going to Jev. If agreement drops, or Jev's model changes, the task falls back to Jev and retrains by itself.
  3. 03Local answers only for API keys Jev has accepted. Keys are forwarded, never stored.
Agreement with Jev is not accuracy

If Jev is wrong, the student is wrong the same way. Every number Jevstiller reports is agreement with the teacher, and the status report says so next to it.

Why

Jev is fast, cheap and typed. It is also 300 ms away.

~300 ms per answer at every load, hosted only, under rate limits set per organization. Jevstiller is for the workload where that hurts: decisions made one after another, latency budgets in milliseconds, boxes with no egress, or not wanting every classification to depend on one external API.

no code changes

Set the base URL and stop

Services keep the TypeSafe SDK and get the same response shape back. x-jevstiller-source tells you who answered.

bounded

Same answers, as a bound

A finite-sample bound on disagreement over all requests, re-verified on the audit channel for as long as the task lives.

self-healing

Checked forever

Drift in Jev's answers is detected on the audit slice. The task falls back and retrains without anyone on call.

self-hosted

One container on CPU

Many services and tenants from one process. Prometheus metrics, JSON logs, an admin CLI, a Kubernetes manifest. Three security audits.

How it compares

Online learning, with a guarantee

Research has done each half: students trained online from an LLM's answers without a bound, and bounds on agreement without a student that keeps learning. Jevstiller combines the two, with a permanent audit and automatic fallback on top. How it compares to stuntd, cloud distillation, routers and caches →