Introduction

DataCore is a Knowledge Warehouse: a team registers raw resources (PDFs, GitHub repos, CSVs, audio, Markdown), the system automatically routes each one through a configurable pipeline of plugin steps, and the pipeline produces reusable artifacts (vector embeddings, knowledge graphs, summaries, or custom analysis data) that downstream LLMs can query.

Core concepts

Concept	What it is
Resource	A raw piece of data a user registers — a name, a source (uploaded file or URL), and a type. Every resource has a status: Pending → Processing → Completed/Failed.
Pipeline	An ordered sequence of processing steps that runs automatically for a given resource type. At most one pipeline exists per trigger type.
Plugin	An independent worker (its own Docker container) that performs one pipeline step and reports back an artifact — or a failure — over the event broker.
Artifact	A reusable output produced for a resource by a plugin step — a vector embedding, a graph, a text summary, or any custom analysis data type a plugin defines.

Why event-driven?

Core (the Warehouse API) and Plugin Workers never call each other synchronously. Instead:

1. Core publishes RESOURCE_CREATED when a resource matches a pipeline.
2. The Pipeline Router dispatches the first step as a PIPELINE_STEP_DISPATCHED event, routed to the plugin responsible for it.
3. The plugin does its work and reports back via a single internal HTTP callback (POST /api/v1/internal/artifacts/{resource_id}) — the only synchronous call in the whole system.
4. Core advances to the next step, or marks the resource Completed/Failed.

This means a plugin crashing, hanging, or being slow can never take down Core or any other plugin — see Architecture for the full reasoning.

Ready to run it yourself? Head to Getting Started.