Security: threat model, what is stored, and four audits

Threat model

Jevstiller is a self-hosted proxy on a company network. Services call it with their own Jev API keys. It forwards requests to Jev, trains local models from Jev’s answers, and answers some requests itself.

Actor	Trust	What the proxy guarantees
Caller service (anything that can reach the port)	untrusted input	It never gets a local answer unless its key was accepted by Jev; it cannot create tasks, write training data, or exhaust the proxy’s disk without a key Jev accepts; it cannot reach any host but the configured upstream
Another caller / tenant	untrusted	No access to other tenants’ tasks or answers in per_key tenancy; in shared tenancy (default) all callers share tasks by design
Jev (the upstream)	trusted	Its answers are training labels; its 401/403/429 decide key state
Operator	trusted	Config, data directory, admin token, tenants map
Someone with the data directory or a backup	trusted as much as your traffic	Holds request text (unless disabled/retained), embeddings and models, but never an API key

Trusting Jev means trusting the TLS certificate that says it is Jev. The proxy checks it against certifi’s CA bundle, or the file in SSL_CERT_FILE if set.

• In October 2026, the private keys of eight eMudhra intermediate CAs turned out to be public. Until certifi 2026.10.19 removed their root, anyone on the network path could have posed as Jev to the proxy: seen callers’ keys and text, and returned answers for the students to learn.
• Jevstiller requires that certifi from 0.5.3. The image’s smoke test checks that neither CA bundle in the image trusts those roots.

What is stored

Per task, under <data_dir>/tasks/<key>/ (directories 0700, files 0600 when run with jevstiller serve):

• samples.sqlite: one row per request, kept while the loop can use it (see Controls). It holds the state’s text or canonical JSON (unless store_text = false), a 64-bit HMAC of it (keyed with the deployment salt), the embedding, Jev’s answer and distribution, the student’s answer, routing metadata, and the upstream request id. There’s no API key and no caller identity beyond the tenant.
• versions/: trained models (numpy arrays, JSON policies). They contain up to 5,000 training embeddings per version (the OOD reference).
• task.json: the question (instructions and criteria), tenant, requested model, and timestamps.

Plus <data_dir>/key-salt (32 random bytes, 0600). It is used to hash API keys and text. Losing it means tenant-map entries no longer match; leaking it makes key hashes guessable offline for known keys.

Controls:

• store_text = false: keep no request text at all.
• Stored text is cut at 8 KiB (UTF-8), past what any encoder reads. With less than min_free_disk_mb (1 GiB) free on the store’s volume, new rows keep no text.
• text_retention_days: blank text older than N days. The old text is securely deleted: secure_delete plus a WAL truncate. A task whose task.json can’t be loaded is not served, but retention and delete-tenant still cover its store.
• Rows the loop no longer reads are deleted: beyond the newest max_train_samples (50,000) training and max_calib_samples (20,000) calibration rows with a Jev answer, per lineage, and beyond the newest keep_local_rows (10,000) rows the student answered alone. They are securely deleted too, and only their counts are kept.
• jevstiller admin delete <key> / delete-tenant <tenant>: remove a task or a tenant’s tasks completely.
• Back up the data directory as you would a database of your traffic (jevstiller backup). Backups and restored data directories get the same modes (files 0600, directories 0700). Text retention doesn’t reach backups: rotate them within the retention period.

API keys

• The caller’s Authorization header is forwarded to Jev unchanged; Jevstiller has no Jev key of its own.
• Keys are held only as sha256(salt + key) in memory, never written to disk or logs.
• A key is “accepted” only after Jev answered a /v1/systemone request made with it (a 2xx whose body parses as a Jev answer), and only for key_ttl_s (1 h).
  • Until then its requests are forwarded, and are recorded as training data only after Jev’s successful answer.
  • A 401/403 from Jev on any path revokes it at once.
  • A 429 makes the proxy back off for that key.
• A request with more than one Authorization header is rejected (400).

Access to the proxy

By default anyone who can reach the port can use it, with their own Jev key. Restrict it with any of these:

Setting	Effect
allow_networks = ["10.0.0.0/8"]	Only these client networks (the TCP peer; X-Forwarded-For is honoured only from trust_forwarded_for proxies)
access_token / access_token_file	Callers must send x-jevstiller-token: <token>. The TypeSafe SDK can add it with TypeSafeClient(headers={...}), which is a one-line code change. The header is never forwarded to Jev.
TLS	ssl_certfile / ssl_keyfile, or a reverse proxy in front
admin_token / admin_token_file	Enables the admin API (/jevstiller/v1/*) and gates /metrics. Without it both are off (404).

An empty token, token file or access-control variable stops startup rather than switching the control off, and the startup log names the controls in effect.

Always enforced, for every path:

• a body limit (max_body_mb, 4 MiB), checked on the declared length and while streaming. POST /v1/systemone with Jev’s images header may send up to max_image_body_mb (32 MiB, Jev’s own limit), and at most max_large_bodies (16) bodies above 4 MiB are held at once (more get 503);
• no ./.. path segments, and no control characters or encoded ?/# in the path; forwarded paths must stay under the upstream URL’s path;
• at most max_questions (32) distinct questions routed per request;
• task caps (max_tasks 10,000, max_tasks_per_tenant 1,000).

GET/HEAD on /healthz and /readyz need no token (probes). The proxy serves /healthz, /readyz, /metrics and /jevstiller/* itself and never forwards them.

Requests with images (experimental)

With [encoder] images set, the proxy reads requests with images (Jev 1.14) instead of only forwarding them. Run 5 audited this path (below).

• Who gets read. Only a key Jev has accepted: a new key’s image request is forwarded unread, and Jev’s answer accepts the key. At most max_image_reads (2) are read at once; more are forwarded.
• Parsing. A strict multipart reader works on the body already in memory (bounded by max_image_body_mb, with at most max_large_bodies large bodies held at once, each within body_timeout_s). Nothing is written to disk. Anything it doesn’t accept, or could read differently from a standard parser (a boundary that isn’t the only, properly parsed parameter; an image with two sources or bytes after its end), is forwarded to Jev as it is. So is a JSON body with more than 50,000 items, before it is parsed.
• Decoding. An image’s header is checked against Jev’s own limits first: PNG or JPEG (the only formats Pillow tries), not animated, 280–1,568 px a side, 78,400–2,458,624 px, 16 MiB, and for a PNG at most 256 chunks before its pixels and 8 compressed ones. Only then is it decoded, in the image encoder’s worker thread. A decoding failure forwards that request; other requests in the same batch are encoded on their own.
• Storage. Image bytes are never stored. A row keeps the state with the image replaced by its SHA-256 and the image’s embedding, as with store_text = false for text.

Security audit, 2026-09-24

A multi-agent audit ran against commit 65fe64b: reconnaissance, four hunters, adversarial validation, and independent verification of every finding. All attacks were reproduced locally against mock upstreams. Twelve findings were confirmed and fixed in 1f29f42, each with a regression test in tests/test_audit_fixes.py.

#	Severity	Finding	Fix
1	Medium	Any 2xx from any forwarded path marked a key as accepted, so a made-up key could get local answers via an upstream path that doesn’t check keys	Only a parsed /v1/systemone answer accepts a key
2	Medium	/healthz / /readyz were exempt from access checks for every method and forwarded upstream; unbounded bodies there	Served locally for every method; only GET/HEAD skip the checks
3	Medium	Callers with no valid key created persistent tasks; admission counted per question name	Unaccepted keys are forwarded first and recorded after Jev’s answer; admission per distinct question; task caps
4	Medium	Unbounded questions per request multiplied encoder work and memory	≤ 32 distinct questions routed; one routing per question; state encoded once per request
5	Low	Chunked bodies bypassed the body limit	Limit enforced while streaming
6	Low	Duplicate Authorization headers: first hashed, last forwarded	Rejected
7	Low	.. in paths escaped a path prefix in upstream	Rejected
8	Low	X-Forwarded-For from loopback bypassed allow_networks	Honoured only from trust_forwarded_for
9	Low	Retained text survived in the WAL and free pages; world-readable data files; unsalted text hash	secure_delete + WAL truncate; umask 0077 / 0700; HMAC
10	Low	Task identity used 48 bits of the question hash (collision into another task)	Full SHA-256 fingerprint; old tasks re-keyed automatically
11	Info	Upstream cookies were shared across callers	No cookie jar
12	Info	Config mistakes failed open (e.g. STORE_TEXT=false ignored, key-hash creating a new salt)	Strict settings validation; key-hash never creates a salt

The audit also verified these properties:

• No injection or unsafe deserialization: SQL values are bound, np.load refuses pickles, and there is no eval/exec/subprocess.
• The upstream host can’t be changed, and there are no redirects.
• Responses never cross between concurrent callers.
• Header CR/LF injection is impossible.
• 500s carry no stack traces.

Dependency floors were raised (h11 >= 0.16, starlette >= 1.3.1) and CI actions pinned by SHA in 40bced8.

Security audit, 2026-09-25 (run 2)

A second run against commit a8b75d6 weighted toward what run 1 did not cover: the admin API, /metrics, settings, backup/restore, the CLI, the deployment files, and the proxy changes since run 1 (encoder backpressure, upstream pools and retry, keep-alive, task unloading, drift restarts, training caps). Eight reviewers, adversarial validation, and independent verification of every finding, all against local mocks. Fifteen findings were confirmed, and all are fixed with regression tests in tests/test_audit_run2.py.

No finding lets a caller read another tenant’s data, obtain keys, or reach the admin API. Keys, tenant isolation, admin authentication and request framing held under every probe.

#	Severity	Finding	Fix
1	Medium	One large request (or one stalled encode) latched encoder backpressure on: every request was forwarded unrouted, for every tenant, until restart	The encoder sees at most 32k characters of a text; an idle encoder is never “overloaded”, so the estimate recovers
2	Medium	A non-ASCII header byte made the proxy answer 500 and never release the tasks it routed: the loaded-engine cap stopped holding and the tasks could not be deleted until restart	Raw header bytes are forwarded; any forwarding failure is a 502; routed engines are released whatever happens (shielded from cancellation); retry-after must be finite and is capped at 1 h
3	Medium	Closing a task’s store while text retention (or an admin call) was using it crashed the process (SIGSEGV)	Retention and admin calls hold the engine like requests do; a store closes only after writes in progress
4	Medium	Privacy settings were silently ignored: store_text = false under [engine], quoted booleans ("false" is true), text_retention_days = 0	Every value is type-checked; either store_text = false wins; retention must be > 0; the startup log says whether text is stored
5	Low	Run 1’s dot-segment check was bypassable with a tab, CR or encoded ?/# (e.g. /.%09./x), escaping a path prefix in upstream	Control characters and encoded ?/# are rejected; the forwarded path must stay under the upstream path
6	Low	Concurrent /readyz calls (no token needed) raced on one probe file and made readiness flap	A unique probe file per check, at most once a second
7	Low	Before Python 3.14 (and in the Docker image), training workers were forked from the live server and held its sockets	Workers start with forkserver (spawn where there is none)
8	Low	An empty token file or environment variable switched caller access control off, or wiped the file’s allowlist	Empty values stop startup; the effective controls are logged
9	Low	key_ttl_s = inf made every made-up key count as accepted	An unknown key is never accepted; key_ttl_s must be finite
10	Low	trust_forwarded_for entries with host bits passed validation but were ignored by uvicorn, so allow_networks checked the load balancer	Validated strictly; uvicorn floor raised to 0.31 (CIDR support)
11	Low	The tenants-file error printed the start of a raw key pasted in place of its hash; upstream credentials were logged and printed	Errors name the tenant; credentials in upstream are redacted
12	Low	backup / restore ignored the 0600/0700 policy	Owner-only modes for backups and restored data dirs
13	Low	Caller-chosen class names put terminal escape sequences into jevstiller admin status	Class names and event values are shown with control characters escaped
14	Low	A caller could create tasks with questions Jev rejects, filling the tenant’s task cap at no cost	Only questions Jev answered count towards admission
15	Info	jevstiller admin delete-tenant 'a#b' deleted tenant a	Names are URL-escaped; the argument is required

Also fixed: duplicate server/date response headers; ci.yml runs with read-only permissions; docker build --build-arg PRELOAD_ENCODER= no longer breaks startup.

Open hardening items:

• No Host-header allowlist (DNS rebinding reaches an unauthenticated proxy from a browser; the access token stops it).
• A key Jev revokes keeps local answers until one of its requests is forwarded again (at least the audit share) or key_ttl_s expires.
• The one retry of a forwarded request can re-send a POST Jev already read (billed to the caller’s own key).
• No per-key fairness for upstream connections or engine loads; no rate limit on admin-token failures.
• The Kubernetes manifest has no NetworkPolicy, and docker-compose publishes on all interfaces: restrict both to your network.
• If Jev ever accepted a second credential (a query api_key, an x-api-key header), a 2xx earned with it would accept the bearer too; today Jev and its SDK use only the bearer.

Security audit, 2026-09-25 (run 3)

Run 3 audited 42820ab and was weighted toward availability, keys and tenants, the completeness of run 2’s fixes, and the code added since: the status page, the public API boundary, the release pipeline, and the website build. Everything was reproduced locally against mock upstreams. Regression tests are in tests/test_audit_run3.py.

No finding exposes keys, other tenants’ data or the admin API. Bearer parsing, revocation, per_key isolation, admin and status-page authentication and escaping, and the absence of keys from every log and response held. The two most important findings are about the guarantee itself, not about an attacker.

#	Severity	Finding	Fix
1	Medium	The audit re-scored its window with the current student, which had since trained on most of those rows, so it overstated agreement and could miss a broken budget	Audit rows are scored as served
2	Medium	Calibration was a sample of distinct texts, so a heavily repeated input could break the bound on the real traffic mix	Split per request
3	Medium	Request text was stored whole, once per task in the request (local answers included), with no cap; a full disk failed readiness	Stored text cut to 32,768 characters; readiness no longer depends on disk space (a metric reports it)
4	Low	One key could fill a shared tenant’s task cap (the rest of run 2 #14)	max_new_tasks_per_key (100 per window)
5	Low	A blank JEVSTILLER_CONFIG skipped the config file, access controls included; blank values reset settings (the rest of run 2 #8)	Blank values stop startup; none clears
6	Low	Two malformed requests (a non-ASCII Basic credential, a URL httpx can’t build) got a 500 and a logged traceback	401 / 400, no traceback; stricter path checks
7	Low	Release builds ran unpinned build tools (and, in the release job, unpinned test dependencies) alongside the files being published	Hash-pinned build tools, no build isolation, --require-hashes in the image, build separated from tests, hashes checked before publishing
8	Info	Settings errors could print a raw key (a reversed tenants map) or a token given in place of its file’s path	Entries named by position; paths not printed

Also fixed: docker build --build-arg PRELOAD_ENCODER= produced an image that couldn’t start.

Open, needs deployment testing against the real Jev:

• If Jev keeps the first of two duplicate JSON keys (the proxy’s parser keeps the last), a request could get an answer recorded for a different question, state or model than the one the proxy routed. Rejecting bodies with duplicate keys, and answers whose label isn’t one of the task’s classes, would close it regardless.
• If Jev accepts a credential other than the bearer (x-api-key, a cookie, a query parameter), a request could get a made-up bearer accepted. Not accepting keys from requests that carry such credentials would close it regardless.
• In shared tenancy, if Jev resolves jev-latest to different models for different keys, one service’s answers can keep flipping a shared task’s teacher lineage (and so its fallback).

Open hardening items (new):

• Admin mode/target changes can miss an engine that is being loaded at the same moment. mode auto doesn’t end a fallback for a task that isn’t loaded.
• Audit draws use a fixed seed per engine load.
• In shared tenancy the bound covers the combined traffic of all services on a task: a low-traffic service can see more disagreement than the budget while the mix is within it. Use per_key or a tenants map where a service needs its own guarantee.
• Uvicorn has no header or body read timeout and no connection limit: put a reverse proxy with those in front when the port is reachable by untrusted networks.

One area was only partly covered: HTTP path handling and forwarding had no dedicated reviewer in this run (the path checks above came from other reviewers), so a fourth run should start there.

Security audit, 2026-10-09 (run 4)

Run 4 audited 3476197 (0.5.0) and was weighted toward what changed since run 3: the yes/no (noul) path, the confidence floor, the cutoffs endpoint and CLI, the bounded store, the website and release pipelines, and HTTP path handling (run 3’s gap). Everything was reproduced locally against mock upstreams. Regression tests are in tests/test_audit_run4.py.

No finding exposes keys, other tenants’ data or the admin API. Yes/no requests that are malformed, mixed with choice questions, or carry unexpected cut-offs or fields are rejected or forwarded unchanged; an uncalibrated yes/no student never answers. Admin and status-page authentication, key acceptance and per_key isolation held.

#	Severity	Finding	Fix
1	Medium	Stored text was cut at 32,768 characters (~128 KiB per row in 4-byte characters) with no total bound: a caller with an accepted key could fill the volume with requests the student answered, free at Jev	Cut at 8 KiB of UTF-8; below min_free_disk_mb (1 GiB) free, rows keep no text
2	Low	A confidence floor on a yes/no task was saved before it was rejected: the task stopped answering, and after a restart it dropped out of text retention and tenant deletion	Changes are checked before they’re saved; an override the task can’t take stops startup; unloadable tasks stay covered by retention and delete-tenant
3	Low	The website workflow ran npm install scripts and the build in the job holding the Cloudflare token	Build and deploy jobs; the deploy job installs only wrangler, scripts off; a 7-day Dependabot cooldown
4	Low	One caller’s new questions could flush every tenant’s admission counts (one shared table)	A caller holds at most a tenth of the table
5	Low	An admin cut-off, floor, target or mode change racing an engine load was saved but never applied (run 3’s open item, now for all four settings)	Changes, loads and deletes take the same per-task lock
6	Low	A change (or the janitor’s last_seen write) racing a delete could bring the deleted task’s question back after a restart	The same lock; a change to a deleted task is a 404
7	Info	jevstiller admin --url X sent the local server’s admin token to X	The server’s token is used only without a URL

Also fixed: a caller’s Accept-Encoding: br/zstd could corrupt a relayed body; Connection-named headers were forwarded; repeated response headers were merged; credentials in upstream replaced every caller’s key (now a startup error); differently spelled yes/no criteria were counted once each; a NaN teacher confidence counted as above a floor in the audit.

Still open, needs deployment testing against the real Jev: run 3’s three items (duplicate JSON keys, non-bearer credentials, jev-latest resolving per key). On the second, run 4 also found that each made-up bearer gets its own max_new_tasks_per_caller quota.

Open hardening items (new):

• The calibration split and audit draws are seeded from public values (the config seed and the task version) and restart with each engine load, so they can be predicted. Whether an attacker in a shared task can use that to choose which of their inputs land in calibration was not measured: the reviewer was cut short.
• nan/inf are accepted in a few numeric settings (max_encoder_wait_ms, admit_window_s, max_body_mb, upstream_timeout_s).
• Release: the GitHub-release job doesn’t re-check the artifacts’ hashes; the Docker build’s cache-from and the base image and encoder model aren’t pinned by digest; no Dependabot entry for pip.
• uvicorn/h11 accept a request with both Content-Length and Transfer-Encoding and keep the connection: not reproduced as smuggling, but a reverse proxy in front normalises it.

One area was only partly covered: the reviewer of the guarantee under adversarial input stopped before measuring the item above, so a fifth run should start there. (Run 5 measured it: see finding 1 below.)

Security audit, 2026-10-24 (run 5)

Run 5 audited 82db190 (0.5.3 plus image support, before its release in 0.6.0) and was weighted toward the new image path (the multipart and base64 reader, the image check and decoding), run 4’s open question about the guarantee’s predictable draws, and resource limits beyond disk. Everything was reproduced locally against mock upstreams, then re-run by a second reviewer trying to disprove it and checked line by line by a third. Regression tests are in tests/test_audit_run5.py.

No finding exposes keys, request text, image bytes, other tenants’ data or the admin API. Key acceptance held on the image path (a key Jev hasn’t accepted never got an image task or a local image answer), image bytes are never stored, the new settings fail closed, and run 4’s fixes held. Findings 1, 2, 5, 6, 7 and 9 affect versions before 0.6.0 (images off); 3, 4 and 8 only image support, and were fixed before its first release (0.6.0).

#	Severity	Finding	Fix
1	Medium	Every engine load restarted the audit and calibration draws from public values. A task reloaded before each request (more than max_loaded active tasks, or another tenant cycling 64 questions) drew the same first number every time: never audited, no drift fallback, no shadow judged. With Jev silently changing 10% of its answers: 6.4% wrong answers against a 2% budget, no fallback	The draws continue across reloads; in the server they are keyed with the deployment salt, so callers can’t compute them
2	Medium	One failing text failed every request in the shared encoder’s batch: a lone surrogate (\udc80) in one request’s state made the tokenizer raise, and up to ~40 concurrent requests of any caller lost their local answer and training row	A failed batch is encoded call by call; lone surrogates are replaced; failed batches count in the overload estimate
3	Medium	Two request shapes let the proxy and Jev read different images in one request (a boundary hidden in another Content-Type parameter; an asset with both part and data_base64), so the proxy recorded Jev’s answer about one image against another: 59 of 59 honest requests for the targeted kind of image got the wrong local answer. Depends on how Jev parses	Ambiguous boundaries, two-source assets and bytes after an image’s end are forwarded
4	Medium	Image bodies were parsed, and their image headers checked, before the caller’s key was, with no bound on how many at once: 16 keyless 32 MiB JSON bodies went past a 4 GiB container on 5 of 8 waves; one crafted PNG cost 0.7 s of CPU	Read only for accepted keys, two at a time; JSON with more than 50,000 items refused before parsing; PNGs with too many chunks refused; PNG and JPEG only
5	Medium	A production student was never checked against the encoder it was trained with. After a change of encoder (small → base, or images = "auto" on a host that lost its GPU) every trained task silently stopped answering locally and learning; with the same dimension the old student answered from embeddings it wasn’t trained on	Such a student isn’t served; the task learns again from the new encoder’s rows; promote and rollback refuse it
6	Low	A caller could name authorization in its Connection header: the proxy took its identity from the bearer, then forwarded the request without it. With a Jev that accepts another credential, made-up bearers were accepted (per-caller quota bypassed, new tenants under per_key)	Connection can’t drop the bearer
7	Low	16 connections that sent only headers held every large-body slot (taken on the declared length), so every caller’s request over 4 MiB got a 503 for as long as they stayed open	A slot is taken once the bytes arrive; bodies must arrive within body_timeout_s (408)
8	Low	Some malformed image requests (non-string part or format, a deeply nested state) got a 500 instead of being forwarded	Forwarded; the reader type-checks those fields and walks the state iteratively
9	Low	A caller who is a question’s only one during its warm-up (about 3,500 rows by default) can steer it with text past the encoder’s 256-token window, which Jev reads but the student doesn’t: the bound passes on rows unlike later traffic. Depends on Jev answering on that text	Partly: finding 1’s fix keeps the audit running, which catches it; see the open item

Also fixed: the image encoder is closed on shutdown.

Still open, needs deployment testing against the real Jev: how Jev parses ambiguous multipart bodies and two-source assets (finding 3 now forwards them either way), whether it accepts credentials other than the bearer (finding 6, and run 3’s item), and whether text past 256 tokens changes its answers (finding 9).

Open hardening items (new):

• Finding 9 has no direct fix: one caller can still supply all of a new task’s calibration rows. Detecting rows whose embeddings match but whose full-text hashes and labels differ, or capping one caller’s share of calibration, would be the next step. Until then, a task’s first callers decide how well its bound describes later traffic (in shared tenancy, the bound covers the combined traffic: run 3’s item).
• json.loads of bodies up to 4 MiB runs on the event loop before the key check (~145 ms per 4 MiB). Standard for ASGI apps, and bounded by the body limit; a reverse proxy doesn’t help with it.
• The engine LRU is shared by all tenants: another tenant’s questions can still make a task reload often (now without effect on its audit, at ~4 ms of CPU per reload).
• After a restart without images, the admin API returns 500 for image tasks (a trusted caller; no state is changed).
• HF_HUB_OFFLINE in the Dockerfile follows PRELOAD_ENCODER only, not PRELOAD_IMAGES; Hub downloads aren’t pinned to a revision.
• The pillow, torch and transformers floors in pyproject.toml allow versions with published advisories (none reachable through the image path as far as the audit found).