Building a Plugin

A plugin is any process that:

1. Connects to RabbitMQ and binds a queue to the routing key pipeline.step.dispatched.<your-plugin-id> on the datacore.resource-lifecycle topic exchange.
2. On each message, does its work, then calls back: POST {CORE_API_URL}/api/v1/internal/artifacts/{resource_id}.
3. Nothing else. It never calls Core synchronously except that one callback, and it never talks to another plugin directly (Plugin Isolation).

The three shipped samples (plugins/markdown-summarizer, plugins/vector-embedder, plugins/github-profile-scanner) are ~100-150 lines of TypeScript each and are the best reference — copy whichever is closest to what you're building.

The dispatch message you'll receive

{
  "event": "PIPELINE_STEP_DISPATCHED",
  "resource_id": "uuid",
  "occurred_at": "ISO-8601",
  "payload": {
    "pipeline_id": "uuid",
    "step_position": 0,
    "plugin_id": "your-plugin-id",
    "attempt_count": 0,
    "source_uri": "s3://bucket/key or https://...",
    "upstream_artifacts": [{ "type": "SUMMARY", "external_ref": "...", "producing_plugin_id": "..." }]
  }
}

upstream_artifacts gives you whatever earlier steps in the pipeline produced, in case your step needs them as input (e.g. the Vector Embedder plugin embeds the Markdown Summarizer's output rather than re-fetching the raw source).

The callback you must send

// success
{ "plugin_id": "your-plugin-id", "step_position": 0, "outcome": "SUCCESS", "artifact": { "type": "SUMMARY", "external_ref": "s3://bucket/key" } }

// failure (Core will retry per that step's configured policy, or mark the resource FAILED once exhausted)
{ "plugin_id": "your-plugin-id", "step_position": 0, "outcome": "FAILURE", "error": "a specific, human-readable reason" }

artifact.type can be one of the built-in types (VECTOR, GRAPH, SUMMARY) or a new type your plugin defines — adding a new artifact type is a one-line Prisma schema change plus a migration on the Core Warehouse side (see github-profile-scanner's REPO_ANALYSIS type for a worked example).

external_ref is a reference into wherever you stored the actual content — Core never asks you to send raw artifact bytes over the callback. Store your output in your own bucket/collection (or reuse Core's MinIO/Qdrant if you're deploying alongside it) and just report back a locator string.

Where your artifact's content gets viewed

If you want your artifact to be viewable in the Web UI (via the artifact chip → "view result" modal), Core needs to know how to fetch its content given external_ref. Today that means either an s3:// prefix (fetched as text or JSON) or a qdrant:// prefix (fetched as a vector + payload) — see backend/src/routes/resources.ts's GET /:id/artifacts/:artifactId handler. Using a different storage scheme means your artifact's content won't render there (but everything else — status tracking, retries, deletion cleanup — still works via external_ref alone).

Packaging & deploying

Each plugin is its own Dockerfile + docker-compose.yml service, with its own environment variables (RABBITMQ_URL, CORE_API_URL, PLUGIN_ID, plus whatever storage credentials it needs). It does not need to live in this repository — a plugin can be built, hosted, and deployed entirely independently, as long as it can reach your RabbitMQ broker and your Core API.

Sharing it with others

Once your plugin works, you can list it on the Community Plugin Registry so other DataCore operators can discover it. This posts metadata and a link to your repo — not your code — so anyone interested still clones your repo, reviews it, and deploys it themselves.