Data Model Reference

Entities

• Resource — id, name, source_type, source_uri, type, status, is_enabled, failure_reason, pipeline_id, artifacts[]. status: PENDING → PROCESSING → (COMPLETED | FAILED). is_enabled (default true) gates whether an MCP server exposes this resource's artifacts to an LLM — it has no effect on pipeline processing, which runs the same regardless of this flag.
• Pipeline — id, name, trigger_type (unique), steps[]. At most one pipeline per trigger_type.
• PipelineStep — position, plugin_id, max_attempts, backoff_seconds, timeout_seconds.
• Plugin — id, name, description, author, version, is_active, config. config is an optional, arbitrary JSON object a plugin defines and reads for itself — see "Storage as a plugin" below.
• Artifact — id, resource_id, type, producing_plugin_id, external_ref. Unique per (resource_id, type) — reprocessing overwrites in place rather than versioning.

Separation of storage

Postgres holds only the tables above — never raw file content or vector floats. Artifact.external_ref is an opaque pointer string (e.g. s3://bucket/key, qdrant://collection/point, or anything a plugin defines) into whichever external system actually stores the content.

Storage as a plugin

Storage isn't a special mechanism bolted onto a processing plugin — it's just another plugin, chained as an ordinary pipeline step. qdrant-register is the worked example: vector-embedder computes an embedding and writes it to a temporary spot in the shared object store, then qdrant-register (the next step) reads it, pushes it into Qdrant, deletes the temporary copy, and reports the final qdrant://... artifact. Core needs no special-casing for this — the final artifact is a completely ordinary qdrant:// ref, handled by the same built-in viewing/deletion logic as any other.

A plugin's config is how you point it at the right instance of its backend without redeploying — e.g. qdrant-register reads config.qdrant_url (falling back to its QDRANT_URL env var if unset) to decide which Qdrant deployment to write to. This is set via PUT /plugins/{id}/config, or the Settings button on a plugin's card in the Web UI. See Building a Plugin for the full pattern and how to write your own storage plugin for a different backend.

Why a resource keeps its own step snapshot

Resource.step_snapshot is a denormalized copy of the pipeline's steps taken the moment a run starts. This is why editing or even deleting a pipeline never affects a resource already PROCESSING under it — the resource doesn't re-read the live Pipeline/PipelineStep rows at all once it's running.