Introduction

DataCore is a Knowledge Warehouse: a team registers raw resources (PDFs, GitHub repos, CSVs, audio, Markdown), the system automatically routes each one through a configurable pipeline of plugin steps, and the pipeline produces reusable artifacts (vector embeddings, knowledge graphs, summaries, or custom analysis data) that downstream LLMs can query.

Core concepts

Concept	What it is
Resource	A raw piece of data a user registers — a name, a source (uploaded file or URL), and a type. Every resource has a status: Pending → Processing → Completed/Failed.
Pipeline	An ordered sequence of processing steps that runs automatically for a given resource type. At most one pipeline exists per trigger type.
Plugin	An independent worker (its own Docker container) that performs one pipeline step and reports back an artifact — or a failure — over the event broker.
Artifact	A reusable output produced for a resource by a plugin step — a vector embedding, a graph, a text summary, or any custom analysis data type a plugin defines.

Use cases

DataCore is reusable RAG infrastructure — instead of writing a bespoke ingestion pipeline every time an LLM needs to know about some new content, you register a resource and let a pipeline of plugins turn it into queryable artifacts once, automatically, for every future resource of that type. Some concrete examples:

• Personal portfolio / activity summarizer — try the live demo (also in Japanese) before reading anything else. It's real, end-to-end DataCore: github-profile-scanner scans a GitHub profile — repo metadata, READMEs, and even the actual package.json/pom.xml/Dockerfile content for its top repos — Gemini writes portfolio copy grounded in that real data, and a semantic "ask about my work" search box answers visitor questions via retrieval over the same data. The demo's own DataCore Resources tab shows every resource and artifact that actually ran behind the scenes — nothing about it is mocked. Source: examples/github-portfolio — see Building a Plugin for how the underlying scanner plugin and its storage step work.
• Engineering team knowledge base — point pipelines at your repos, exported docs, and PDFs; embeddings and summaries stay current as new resources are added, queryable by any MCP-compatible LLM client (Claude, Cursor, etc.) without each tool re-implementing its own retrieval.
• Support / onboarding assistant — ingest scattered CSVs, call transcripts, and runbooks once; any downstream chatbot gets grounded answers via MCP instead of re-implementing retrieval per app.

Why event-driven?

Core (the Warehouse API) and Plugin Workers never call each other synchronously. Instead:

1. Core publishes RESOURCE_CREATED when a resource matches a pipeline.
2. The Pipeline Router dispatches the first step as a PIPELINE_STEP_DISPATCHED event, routed to the plugin responsible for it.
3. The plugin does its work and reports back via a single internal HTTP callback (POST /api/v1/internal/artifacts/{resource_id}) — the only synchronous call in the whole system.
4. Core advances to the next step, or marks the resource Completed/Failed.

This means a plugin crashing, hanging, or being slow can never take down Core or any other plugin — see Architecture for the full reasoning.

Ready to run it yourself? Head to Getting Started.