Artifact Storage

Nexus owns OpenAI-compatible file and vector-store lifecycle resources at the gateway edge. The public API is OpenAI-shaped, but Nexus stores metadata and bytes in Nexus-controlled infrastructure.

Artifact metadata lives in the control-plane Postgres database:

• files stores file identity, tenant/org/project scope, filename, purpose, byte count, checksum, content type, object key, creation time, and soft-delete time.
• vector_stores stores vector-store identity, scope, name, metadata, optional expiration policy, creation time, last-active time, and soft-delete time.
• vector_store_files stores active and soft-deleted file attachments for each vector store, with an indexing status (in_progress, completed, failed), a last_error message for failed runs, and the current indexer lease owner/timestamp while work is claimed.
• vector_store_chunks stores embedded chunks for the nexus-hosted vector index: chunk text, a pgvector embedding column, the embedding model and dimension used when the chunk was indexed, token count, and full scope columns. The compose Postgres image is pgvector/pgvector:pg16 so CREATE EXTENSION vector is available.

File bytes live in an object store through nexus-object-store. When NEXUS_S3_ENDPOINT is set, the gateway and control-plane artifact REST use S3-compatible storage with the bucket named by NEXUS_ARTIFACTS_BUCKET (default nexus-artifacts). When no S3 endpoint is configured, they use a filesystem-backed store rooted at NEXUS_ARTIFACTS_ROOT_DIR; that variable is required for filesystem mode.

Object keys use the scoped layout:

<tenant_id>/<org_id>/<project_id>/<file_id>

Compose creates the nexus-artifacts MinIO bucket and grants the gateway/control-plane artifact user read/write access to that bucket. The same gateway user keeps read-only access to request-body storage.

Gateway file and vector-store operations derive an artifact scope from the authenticated API-key principal. Console operations go through control-plane REST and derive the same scope from the request path plus the verified session principal. The scope contains tenant_id, org_id, and project_id. Gateway handlers reject API keys that are not scoped to both an organization and a project; control-plane REST requires project visibility for reads and project-manager authority for writes.

Every metadata query filters by the full scope and excludes soft-deleted rows. Cross-project lookups behave as not found, and list routes return only resources in the caller’s project.

POST /v1/files parses multipart bytes, validates the file purpose, writes bytes to the object store, and inserts a durable files row. If metadata insertion fails after the object write, the gateway attempts to delete the orphaned object.

GET /v1/files, GET /v1/files/{id}, and GET /v1/files/{id}/content read from Postgres and the object store. Because metadata and bytes are durable, uploaded files remain readable after a gateway restart. File list responses support cursor pagination through after and limit.

DELETE /v1/files/{id} sets deleted_at on the file row and on any vector-store attachments of that file in one transaction, then deletes the object bytes. Object deletion is best-effort: a failed object delete is logged and does not fail the request, because the metadata soft-delete has already committed. Subsequent file metadata and content reads return HTTP 404, as do all artifact lookups for unknown or out-of-scope resource IDs.

Vector-store lifecycle routes persist metadata in Postgres. File attachments are stored in vector_store_files; attach inserts rows with status in_progress, and the indexer settles each row to completed or failed (with a user-facing last_error). The gateway and control-plane REST report per-status file counts and usage_bytes from active attachment rows. Vector-store list and modify routes support OpenAI-style expiration policy JSON such as {"anchor":"last_active_at","days":30}; reads compute expires_at from the current last_active_at. Reading or listing vector stores does not update last_active_at; mutating use such as attach/detach and search updates activity.

A background worker inside the gateway leases in_progress vector_store_files rows with FOR UPDATE SKIP LOCKED and runs chunk → embed → store per attachment:

1. File bytes are read through the artifact store; plain-text and PDF content is extracted (other MIME types fail with a clear last_error).
2. Extracted text is checked against indexing budgets before any provider call: max extracted bytes (NEXUS_INDEXER_MAX_TEXT_BYTES, default 1 MiB), max chunks (NEXUS_INDEXER_MAX_CHUNKS, default 512), and max estimated tokens (NEXUS_INDEXER_MAX_TOKENS, default 250,000).
3. Text is chunked with a sliding character window (defaults ≈800 tokens with ≈200-token overlap; NEXUS_INDEXER_CHUNK_CHARS / NEXUS_INDEXER_CHUNK_OVERLAP_CHARS).
4. Chunks are embedded through the same provider/registry path as POST /v1/embeddings — in-process, never through the public edge — using the model named by NEXUS_DEFAULT_EMBEDDING_MODEL.
5. Chunks replace any prior chunk set for that file atomically within the vector index backend, then the attachment status flips. Chunk writes and the metadata status flip never share a transaction, so a crash leaves the row in_progress for an idempotent redo after the lease expires (NEXUS_VECTOR_INDEXER_LEASE_SECONDS, default 300 seconds).

Chunk operations go through the VectorIndex trait in nexus-artifact-store; the default PgVectorIndex backend stores chunks in the control-plane database.

POST /v1/vector_stores/{id}/search embeds the query with NEXUS_DEFAULT_EMBEDDING_MODEL, rejects live chunks indexed with a different model or dimension, runs an exact nearest-neighbour scan (cosine distance) scoped to the store, and returns OpenAI-shaped ranked results (file_id, filename, score, content snippet). Search joins through active vector_store_files, files, and vector_stores rows, so detached or deleted content is not returned even if chunk cleanup failed. The route accepts API-key principals or the internal service token plus explicit scope headers (the console proxy path). Detach, file delete, store delete, and expiry paths delete the corresponding chunks best-effort.

The control-plane exposes membership-authorized artifact REST under:

/api/orgs/:org_id/projects/:project_id/files
/api/orgs/:org_id/projects/:project_id/vector-stores

The console BFF proxies only the project-scoped files and vector-store artifact routes (including the per-store search sub-route) through /api/artifacts/* with the user’s bearer token. It preserves multipart upload bodies and byte-stream downloads, and rejects unrelated control-plane paths.

Console search is proxied: the control-plane route enforces membership authz, then calls the gateway search route at NEXUS_GATEWAY_INTERNAL_ENDPOINT with the shared NEXUS_SERVICE_TOKEN and explicit scope headers.

Indexing and search cover vector-store contents only. Request-time retrieval injection (a hosted file_search tool inside chat/responses) is not part of the artifact path.

Artifact lifecycle audit events record metadata only: resource IDs, org/project scope, purpose, filename, byte count, checksum, status, and action. Indexing provider attempts use the synthetic subject svc_vector_indexer; search query embedding attempts use svc_vector_search. Indexing emits gateway.vector_store_file.indexed (chunk count, embedding model, token estimate) and search emits gateway.vector_store.searched (result count only; never query text or chunk content). Console artifact reads emit data_access.storage_viewed for list/detail/content/search access so sensitive file and vector-store inspection is visible in the audit stream. Raw file bytes are not copied into audit events or request analytics. Callers that need bytes use the file content route under the same authorization checks.