Building a Plugin

A plugin is any process that:

1. Connects to RabbitMQ and binds a queue to the routing key pipeline.step.dispatched.<your-plugin-id> on the datacore.resource-lifecycle topic exchange.
2. On each message, does its work, then calls back: POST {CORE_API_URL}/api/v1/internal/artifacts/{resource_id}.
3. Nothing else. It never calls Core synchronously except that one callback, and it never talks to another plugin directly (Plugin Isolation).

The four shipped samples (plugins/markdown-summarizer, plugins/vector-embedder, plugins/qdrant-register, plugins/github-profile-scanner) are ~100-150 lines of TypeScript each and are the best reference — copy whichever is closest to what you're building.

The dispatch message you'll receive

{
  "event": "PIPELINE_STEP_DISPATCHED",
  "resource_id": "uuid",
  "occurred_at": "ISO-8601",
  "payload": {
    "pipeline_id": "uuid",
    "step_position": 0,
    "plugin_id": "your-plugin-id",
    "attempt_count": 0,
    "source_uri": "s3://bucket/key or https://...",
    "upstream_artifacts": [{ "type": "SUMMARY", "external_ref": "...", "producing_plugin_id": "..." }]
  }
}

upstream_artifacts gives you whatever earlier steps in the pipeline produced, in case your step needs them as input (e.g. Vector Embedder embeds the Markdown Summarizer's output rather than re-fetching the raw source, and Qdrant Register reads the vector Vector Embedder just computed).

The callback you must send

// success
{ "plugin_id": "your-plugin-id", "step_position": 0, "outcome": "SUCCESS", "artifact": { "type": "SUMMARY", "external_ref": "s3://bucket/key" } }

// failure (Core will retry per that step's configured policy, or mark the resource FAILED once exhausted)
{ "plugin_id": "your-plugin-id", "step_position": 0, "outcome": "FAILURE", "error": "a specific, human-readable reason" }

artifact.type can be one of the built-in types (VECTOR, GRAPH, SUMMARY) or a new type your plugin defines — adding a new artifact type is a one-line Prisma schema change plus a migration on the Core Warehouse side (see github-profile-scanner's REPO_ANALYSIS type for a worked example).

external_ref is a reference into wherever you stored the actual content — Core never asks you to send raw artifact bytes over the callback. Store your output in your own bucket/collection (or reuse Core's MinIO/Qdrant if you're deploying alongside it) and just report back a locator string.

Where your artifact's content gets viewed

If you want your artifact to be viewable in the Web UI (via the artifact chip → "view result" modal) and cleaned up on delete, Core needs to know how to fetch and delete its content given external_ref. Out of the box, that means either an s3:// prefix (fetched from/deleted in MinIO) or a qdrant:// prefix (fetched from/deleted in Qdrant) — see backend/src/routes/resources.ts. Report an external_ref in one of these two forms and viewing and cleanup just work, with zero extra code.

Storage as a plugin

A different storage backend (Pinecone, Weaviate, a different Qdrant/S3-compatible instance, anything) isn't a special mode bolted onto a processing plugin — it's just another plugin, chained as an ordinary pipeline step. qdrant-register is the worked example:

1. vector-embedder computes an embedding and writes the raw vector to a temporary spot in the shared object store (s3://bucket/vectors-pending/{resource_id}.json), reporting that as an intermediate VECTOR artifact.
2. qdrant-register (the next step) reads that upstream artifact via upstream_artifacts, pushes the vector into Qdrant, deletes the temporary object, and reports the final VECTOR artifact with external_ref: qdrant://collection/point-id — overwriting the intermediate one (Artifact is unique per (resource_id, type), so this is a normal upsert, not a special case).

Core needs no new code for this — the final artifact is a completely ordinary qdrant:// ref, viewed and cleaned up by the exact same built-in logic as any other. Writing your own storage plugin for a different backend means following the same two-step shape: a compute/produce step that hands off a temporary artifact, and a register step that owns actually writing to (and deleting from) your backend.

Making it configurable: a plugin's config

Hardcoding a connection URL means redeploying every time it changes. Instead, a plugin can read its own config — an arbitrary JSON object stored on its Plugin record, set via PUT /plugins/{id}/config or the Settings button on its card in the Web UI (see REST API reference):

// qdrant-register resolves its own Qdrant URL like this:
const res = await fetch(`${CORE_API_URL}/api/v1/plugins/${PLUGIN_ID}`);
const plugin = await res.json();
const qdrantUrl = plugin.config?.qdrant_url || process.env.QDRANT_URL || 'http://localhost:6333';

config is entirely up to the plugin to define and interpret — Core just stores and returns whatever JSON object you PUT. A future "Pinecone Register" plugin might read config.api_key/config.index_name instead; Core doesn't need to know or care about the difference.

Packaging & deploying

Each plugin is its own Dockerfile + docker-compose.yml service, with its own environment variables (RABBITMQ_URL, CORE_API_URL, PLUGIN_ID, plus whatever storage credentials it needs). It does not need to live in this repository — a plugin can be built, hosted, and deployed entirely independently, as long as it can reach your RabbitMQ broker and your Core API.

Sharing it with others

Once your plugin works, you can list it on the Community Plugin Registry so other DataCore operators can discover it. This posts metadata and a link to your repo — not your code — so anyone interested still clones your repo, reviews it, and deploys it themselves.