Data Model Reference

Full detail (including exact field types and state-transition diagrams) lives at specs/001-datacore-knowledge-warehouse/data-model.md.

Entities

• Resource — id, name, source_type, source_uri, type, status, is_enabled, failure_reason, pipeline_id, artifacts[]. status: PENDING → PROCESSING → (COMPLETED | FAILED). is_enabled (default true) gates whether an MCP server exposes this resource's artifacts to an LLM — it has no effect on pipeline processing, which runs the same regardless of this flag.
• Pipeline — id, name, trigger_type (unique), steps[]. At most one pipeline per trigger_type.
• PipelineStep — position, plugin_id, max_attempts, backoff_seconds, timeout_seconds.
• Plugin — id, name, description, author, version, is_active.
• Artifact — id, resource_id, type, producing_plugin_id, external_ref. Unique per (resource_id, type) — reprocessing overwrites in place rather than versioning.

Separation of storage

Postgres holds only the tables above — never raw file content or vector floats. Artifact.external_ref is a pointer (s3://bucket/key or qdrant://collection/point) into whichever external system actually stores the content. Deleting a resource cascades: Core reads each artifact's external_ref, deletes the underlying object from its store, then removes the rows.

Why a resource keeps its own step snapshot

Resource.step_snapshot is a denormalized copy of the pipeline's steps taken the moment a run starts. This is why editing or even deleting a pipeline never affects a resource already PROCESSING under it — the resource doesn't re-read the live Pipeline/PipelineStep rows at all once it's running.