Helicone: Lessons Learned

A distilled read of the Helicone open-source monorepo. Helicone is the most complete open-source reference for the category nexus sits in — AI gateway, request-scoped analytics, cost/escrow, and a tenant-facing console — so the design choices it got right are worth copying deliberately, and the ones it got wrong are worth improving on deliberately.

For what nexus has already merged (routes, processes, stores), treat Services and the Rust modules it links as the source of truth. This document compares Helicone’s implementation to nexus contracts and shared crates; it does not enumerate a delivery schedule.

Target reading budget: ≤ 45 minutes. Skim the structure block, then read the sections that touch the code you are changing.

How to use this document:

• Every bullet points at a concrete path in the upstream Helicone repository. If a citation ever breaks, fix the path in the same PR that moves or renames the target.
• “Patterns to improve on” items name product gaps relative to Helicone; close them with code + tests, then update Services if behaviour changes.
• “Do not copy” items are enforced by reviewers and the doc-lint script wherever possible.

Helicone’s structure

Area	Helicone path	Role
Gateway (proxy)	https://github.com/Helicone/helicone/tree/main/worker/	Cloudflare Worker that terminates the OpenAI-compatible edge and dispatches to providers. Entry point is worker/src/lib/ai-gateway/.
Control-plane API	https://github.com/Helicone/helicone/tree/main/valhalla/jawn/	TSOA + Express service for tenant CRUD, managers, and HQL. Public controllers live under valhalla/jawn/src/controllers/public/.
Dashboard	https://github.com/Helicone/helicone/tree/main/web/	Next.js app that surfaces tenant analytics, sessions, prompts.
Cost / registry	https://github.com/Helicone/helicone/tree/main/packages/cost/	Model registry + cost calculator + per-provider routing helpers. See packages/cost/FLOWS.md.
Analytics schema	https://github.com/Helicone/helicone/tree/main/clickhouse/migrations/	80+ migrations evolving the ClickHouse request/response schema, culminating in schema_41_request_response_replacing_merge_tree.sql.
Control-plane schema	https://github.com/Helicone/helicone/tree/main/supabase/migrations/	240+ Supabase migrations covering tenants, keys, prompts, evals.
Dev stack	https://github.com/Helicone/helicone/tree/main/docker/	One-command Docker Compose bring-up for Postgres + ClickHouse + MinIO + web + jawn + worker.

Patterns to adopt

Patterns Helicone got right. nexus mirrors them with Rust semantics.

1. Attempt-based routing

Attempt { endpoint, providerKey, authType, priority, source } is a clean abstraction. AttemptBuilder collects every candidate for a request, AttemptExecutor runs one, and SimpleAIGateway loops over the sorted list. Each attempt carries a human-readable source string (for example "gpt-4/openai/byok") that makes debugging trivial.

Cite: worker/src/lib/ai-gateway/ARCHITECTURE.md, worker/src/lib/ai-gateway/types.ts, AttemptBuilder.ts, AttemptExecutor.ts, SimpleAIGateway.ts.

2. Two-phase BYOK → PTB, cost-sorted inside each phase

The golden rule: every BYOK endpoint is exhausted (sorted by cost) before any PTB endpoint is attempted. Per-provider byok_only flag excludes a provider from Phase 2. See packages/cost/FLOWS.md for the worked scenarios.

The contract effectively reads: “BYOK is the tenant’s explicit choice; do not silently fall back to PTB unless the tenant has exhausted their own keys and has not marked the provider as BYOK-only.”

3. Registry hierarchy: template + override + resolved

ModelProviderConfig is the registry template (pricing, context length, supported parameters). EndpointConfig is a per-deployment override (region, resource name). Endpoint is the merged runtime shape with a fully constructed baseUrl. O(1) lookups with minimal duplication. Author-based folder structure (e.g. packages/cost/models/authors/anthropic/ and sibling directories) scales to dozens of authors without a central file.

Cite: packages/cost/FLOWS.md “Data Architecture & Type System”, packages/cost/models/.

4. Model string grammar

Three accepted shapes:

"claude-3.5-haiku"                        // try every provider, cost-sorted
"claude-3.5-haiku/bedrock"                // pin provider
"claude-3.5-haiku/bedrock/us-west-2"      // pin deployment

Plus a comma-joined list for explicit fallback ("a,b,c"). Copy the grammar verbatim.

Cite: packages/cost/FLOWS.md “Model String Formats”.

5. Result<T, K> everywhere

Every async operation returns a Result<T, K> with ok(data) / err(error) helpers and isErr(r) guards. Eliminates try/catch sprawl and makes error paths explicit.

Cite: worker/src/lib/util/results.ts, applied throughout worker/src/lib/ai-gateway/.

6. Async logging off the hot path

The worker enqueues a log request and returns to the caller; a separate ingest service writes to the analytics store. Single-digit-ms added latency on the hot path. nexus implements the same split: the gateway publishes request.completed envelopes to NATS JetStream and the ingest service drains them into ClickHouse.

Cite: worker/src/lib/dbLogger/, SimpleAIGateway.ts.

7. Separated body storage

Bodies go to S3-compatible object storage with a TTL; metadata goes to ClickHouse. Hot queries never touch the blob store; full-text search uses body bloom filters in ClickHouse.

Cite: clickhouse/migrations/schema_37_request_response_versioned_assets.sql, schema_40_request_response_versioned_bodies_ttl.sql.

8. ReplacingMergeTree for logs

ClickHouse’s ReplacingMergeTree (keyed on (organization_id, provider, model, user_id, request_created_at, request_id) with a version column updated_at) permits in-place row updates for post-hoc enrichment, feedback, and cost corrections without a rewrite. The schema in schema_41_request_response_replacing_merge_tree.sql is the canonical reference:

ENGINE = ReplacingMergeTree(updated_at)
PARTITION BY toYYYYMM(request_created_at)
PRIMARY KEY (organization_id, provider, model, user_id, request_created_at, request_id)
ORDER BY (organization_id, provider, model, user_id, request_created_at, request_id);

9. Bloom filters on high-selectivity columns

Helicone adds bloom_filter(0.01) indices on mapKeys(properties) and mapValues(properties) plus ngrambf_v1(4, 1024, 1, 0) indices on request_body and response_body. The result is fast WHERE and LIKE at ClickHouse scale with a modest storage cost.

Cite: schema_39_request_response_bloom_filter.sql for the exact index shape.

10. Header-driven sessions

Helicone-Session-Id, Helicone-Session-Path, Helicone-Session-Name group multi-step agent flows without coupling the SDK. The path syntax (/root/child/grandchild) lets the UI render a tree view.

Cite: docs/features/sessions.mdx.

11. Custom properties via Helicone-Property-* headers

Dimensional tagging with zero SDK impact. Every Helicone-Property-<name>: <value> header becomes a key in the properties: Map(LowCardinality(String), String) column, queryable with ClickHouse map-accessor syntax.

Cite: see properties Map(LowCardinality(String), String) in schema_41_request_response_replacing_merge_tree.sql; Helicone docs under docs/features/.

12. Stream parsers per provider family

OpenAI SSE, Anthropic SSE, Vercel streams, and Google streams each have distinct shapes. Helicone keeps one parser per family behind a shared interface.

Cite: worker/src/lib/dbLogger/streamParsers/openAIStreamParser.ts, anthropicStreamParser.ts, vercelStreamParser.ts, responseParserHelpers.ts.

13. Provider transforms as pure helpers

Functions like toAnthropic(body), enableStreamUsage(wrapper, bodyMapping), and ant2oaiResponse() keep provider-specific logic isolated so adding a provider does not touch the router.

Cite: the imports in SimpleAIGateway.ts (ant2oaiResponse, goog2oaiResponse, oaiChat2responsesResponse), plus their implementations under worker/src/lib/clients/llmmapper/.

14. Escrow for PTB

Reserve worst-case cost before the request, settle the actual charge on success, cancel on failure. Prevents over-spend on runaway streams.

Cite: reserveEscrow(wrapper, env, orgId, endpoint) referenced in ARCHITECTURE.md, WalletKVSync.ts.

15. Dockerized dev stack

One command brings up Postgres + ClickHouse + MinIO + web + jawn + worker. Massively lowers contributor setup cost.

Cite: docker/ (compose files), repo-root docker-compose.yml variants.

Patterns to improve on

Patterns Helicone got wrong or incomplete. Track fixes in code and tests; update architecture docs when behaviour changes.

1. Cloudflare Workers for the gateway

Great for SaaS edge; hostile to air-gap. Workers are a managed runtime, limited to a narrow set of crypto primitives, opaque to on-prem ops teams, and incompatible with the FIPS posture customers in regulated verticals require.

Cite: worker/src/ and its wrangler.toml.

2. Audit logging is “request logs”

Helicone has no separate audit log. Request logs serve double duty: product analytics and compliance trail. There is no immutability, no cryptographic chain, no signing, no WORM. Customers who need evidentiary audit have to build it externally.

Cite: absence of an audit/ service in valhalla/jawn/src/; the analytics schema (clickhouse/migrations/) is the only place “history” lives.

3. RBAC only, no ABAC

Helicone has organizations, projects, and roles, but no attribute-based policy. There is no request-time authorization keyed on purpose, classification, or model family.

Cite: role tables under supabase/migrations/; public controllers in valhalla/jawn/src/controllers/public/ rely on organization membership and role.

4. No data classification

Helicone treats every request identically. There is no sensitivity tier, no classification-driven redaction, no routing restriction by tier.

Cite: request schema schema_41_request_response_replacing_merge_tree.sql — no classification column.

5. Auth stack is Better-Auth + Supabase

Works for a SaaS product; lacks the features enterprise customers need: SAML polish, PIV / CAC smartcard, FIPS crypto mode, robust SCIM, break-glass with quorum.

Cite: valhalla/jawn/src/ session layer; Supabase RLS policies in supabase/migrations/.

6. Schema churn

240+ Postgres migrations and 80+ ClickHouse migrations signal reactive design. Each one is a small shape fix; there is no “we decided the ClickHouse schema would look like X from day one and evolve via ReplacingMergeTree deliberately”.

Cite: supabase/migrations/, clickhouse/migrations/.

7. TSOA-generated TypeScript types across packages

Helicone uses TSOA to generate TypeScript clients from the TSOA-decorated Express controllers. The generated files are checked in, marked “do not manually edit”, and scattered across the repo (the autogen list is in AGENTS.md). This tightly couples backend and frontend shapes, makes API evolution fragile, and is hostile to non-TS consumers.

Cite: valhalla/jawn/tsoa.json (if present), autogenerated *.generated.ts scattered across web/ and valhalla/jawn/.

8. Escrow in a Cloudflare Durable Object

The wallet / escrow lives in a Durable Object (WalletKVSync.ts). Durable Objects are opaque, hard to inspect at rest, hostile to backups and reporting, and tied to the Cloudflare runtime.

9. Redaction / PII is ad-hoc

Helicone has per-tenant property filters and some redaction utilities, but redaction is not a named pipeline phase with a contract. A classification-driven “redact spans P1, P2, P3 before log” policy is not expressible.

10. No FIPS posture

Helicone relies on whichever TLS the Cloudflare Worker runtime provides. There is no cargo feature flag, no module-identifier assertion, no deliberate TLS 1.3-only posture.

11. No formal event taxonomy

Webhooks and alerts are feature-specific: webhook for X, email for Y, Slack for Z. There is no single canonical event schema covering auth, policy, data-access, config-change, gateway-request, admin-action.

12. No air-gap install bundle

Helicone offers Docker Compose and Helm, but the install path assumes registry and image-pull access. No fully offline, signed, version-pinned bundle.

13. Body storage without Object Lock

The blob store is plain S3 / MinIO. Nothing prevents an operator from rewriting or deleting a stored body.

Cite: the s3 / MinIO wiring in Helicone dev stack docker/.

14. Supabase as app DB + auth

Using Supabase for both application data and auth couples self-hosting to Supabase’s surface area. RLS policies become a hidden second source of truth for access.

Cite: supabase/migrations/ and the supabase_auth schema.

15. Tenant scoping by rewriting caller-supplied SQL

HQL lets tenants run arbitrary SQL over the Tier 1 analytics schema, and isolation is enforced by rewriting the query text to inject an organization_id = filter. String-level rewriting cannot constrain what a subquery reads: the injected predicate applies to the outer result, while an inner SELECT with no tenant predicate still scans every tenant’s rows and can return them as an aggregated value. Tenant isolation belongs in the database — a scoped role, row policies, or parameterised queries the service builds itself — not in a filter appended to text the caller controls.

Cite: valhalla/jawn/src/managers/HeliconeSqlManager.ts.

Things to explicitly not copy

Hard no-go list. Reviewers reject PRs that introduce any of these.

• Cloudflare Worker / Cloudflare KV bindings. Files under worker/src/ that import @cloudflare/workers-types, reference env.KV.*, or use Durable Objects. nexus uses standard Tokio services and Postgres state.
• Better-Auth web session layer. The session surface is owned by the nexus auth service’s HTTP APIs; do not import Better-Auth into nexus.
• Durable Object for wallet / escrow. See improvement item 8.
• TSOA’s OpenAPI generation. See improvement item 7. Protobuf + generated clients is the replacement.
• The assumption that every operator runs Kubernetes. The air-gap path must also work on docker compose on a hardened RHEL host. Helm charts are an add-on, not the only install.
• Implicit tenant isolation via Supabase RLS. nexus enforces isolation at the application layer (tenant_id on every row, checked in every query) plus the PDP. RLS is defense-in-depth, never the sole control.
• Scattered autogenerated files marked “do not edit”. See AGENTS.md for the allowlist of generators and their output paths.