Architecture

Components

• Core Warehouse (backend/) — Node.js/TypeScript + Express REST API under /api/v1/. Owns relational metadata (PostgreSQL via Prisma), publishes/consumes lifecycle events (RabbitMQ), and hosts the Pipeline Router that dispatches steps, handles retries/backoff, and sweeps for step timeouts.
• Plugin Workers (plugins/*/) — one independent Docker container per plugin. Each subscribes to the event broker for its own routing key and reports results via a single internal HTTP callback.
• Web UI (frontend/) — React + Vite + Tailwind SPA, three primary sections (Resources, Pipelines, Plugins) plus a global search, served via nginx which also reverse-proxies /api and /registry to their respective backend services.
• Plugin Registry (registry/ + community-site/) — registry is a standalone service (its own SQLite-backed API, with user accounts) that any DataCore deployment's frontend can browse or post to; community-site is a standalone website that fronts the same API as its own destination, independent of the rest of the stack. It's deployed separately from the main app (live at community.datacore.wakewakanai.com, not part of this repo's docker-compose.yml) — see Community Plugin Registry & Bundles for self-hosting it. Hosts two kinds of public listings: plugin listings (metadata + a link to the plugin's own repo/Docker image — not a code-hosting or auto-deploy system) and DataCore Bundles (a shareable list of resource definitions + the pipeline that processed them, for reprocessing elsewhere — see Sharing & Importing Bundles). Posting either requires a verified account; browsing doesn't.
• Storage tiers: PostgreSQL (relational metadata only), MinIO (raw files + text/JSON artifacts, S3-compatible), and whatever store a plugin's artifacts belong in (e.g. Qdrant for vectors) — Postgres never stores file content or vectors directly, only reference pointers.

Guiding principles

These shape every design decision:

1. Event-Driven Architecture — all state transitions are broker events; no synchronous cross-service calls except the one explicitly-allowed internal artifact callback.
2. Plugin Isolation — each plugin is its own container; a crash, hang, or bug in one plugin cannot affect Core or any other plugin.
3. Schema-First Data Models — Resource/Pipeline/Plugin/Artifact schemas are defined before any endpoint that reads or writes them.
4. API Contract Discipline — every REST endpoint lives under /api/v1/ with one consistent JSON error envelope ({ "error": { "code", "message" } }).
5. Separation of Storage — relational metadata, raw files, and generated artifacts each live in their own tier; nothing is duplicated across tiers.
6. UI Consistency — the Web UI reflects live API state; no hardcoded mock data once a view is wired up.
7. Test Discipline — every REST endpoint and event handler has an integration test run against real Postgres/RabbitMQ/MinIO/Qdrant — nothing is mocked.
8. No Silent Failures — a failed step surfaces a specific failure_reason on the resource; nothing is left Processing forever without either completing, failing, or (for a non-responding step) being caught by the timeout sweep.

Request flow: registering a resource

User → POST /api/v1/resources
     → Core persists Resource (PENDING), matches a Pipeline by trigger_type
     → Core publishes RESOURCE_CREATED
     → Pipeline Router consumes it, snapshots the pipeline's steps onto the resource, dispatches step 0
     → Plugin Worker consumes PIPELINE_STEP_DISPATCHED, does its work
     → Plugin calls back POST /api/v1/internal/artifacts/{resource_id}
     → Core upserts the Artifact, advances to the next step (or marks COMPLETED)

Full detail, including the retry/backoff/timeout state machine, is in the Events reference.