DataCore

English

日本語

English

日本語

Appearance

DataCoreEvent-Driven Knowledge Warehouse

Turn PDFs, GitHub repos, CSVs, audio, and Markdown into vector embeddings, summaries, and queryable knowledge your LLMs can actually use — through a pipeline of independent, pluggable workers you fully control.

See the Live Demo
Get Started
Architecture
API Reference

Event-Driven

Core and plugins never call each other directly — every state transition is a RabbitMQ event, so a misbehaving plugin can't take down the platform.

Pluggable Pipelines

Define an ordered sequence of plugin steps per resource type. Each step has its own retry, backoff, and timeout policy.

Storage as a Plugin

Where an artifact actually lives (Qdrant, S3, anything) is its own pluggable, configurable step — not hardcoded into Core. Point it at a different backend from the Web UI, no redeploy needed.

Community Plugins

Browse and share plugins across DataCore deployments through the public plugin registry — discover what others have built, or post your own.

Why DataCore? ​

Every team eventually needs the same thing: take some raw content — code, documents, audio, spreadsheets — and turn it into something an LLM can actually query and reason about. Usually that means writing a one-off ingestion script per project, per content type, per storage backend. DataCore is that plumbing, built once: register a resource, a pipeline of small independent plugins processes it automatically, and the result is a reusable artifact (an embedding, a summary, a graph, anything a plugin defines) that any MCP-compatible LLM client can query — without every downstream tool reimplementing its own retrieval logic.

See it in action ​

Don't just read about it — try the live demo (日本語版). It's a real portfolio site generated entirely through DataCore's own pipeline, not a mockup:

  • github-profile-scanner scans a real GitHub profile — repo metadata, actual README content, and even the real package.json/pom.xml/Dockerfile content for its top repos, not just a guess from the language field.
  • Gemini writes the portfolio copy, grounded only in that real scanned data.
  • A semantic "Ask About My Work" search box answers visitor questions via retrieval-augmented generation over the same data — a genuine RAG feature, not a chatbot improvising from training data.
  • The demo's own DataCore Resources tab shows every resource and artifact DataCore actually produced behind the scenes, so you can see the real pipeline output, not just the polished result.

Full source in examples/github-portfolio — see Building a Plugin for how the scanner plugin and its Qdrant storage step actually work.

Use cases ​

  • Personal portfolio / activity summarizer — the live demo above. A small, complete example of the "bring your own plugin" story end to end.
  • Engineering team knowledge base — point pipelines at your repos, exported docs, and PDFs; embeddings and summaries stay current as resources are added, queryable by any MCP client (Claude, Cursor, etc.) without each tool reimplementing retrieval.
  • Support / onboarding assistant — ingest scattered CSVs, call transcripts, and runbooks once; any downstream chatbot gets grounded answers via MCP instead of reimplementing retrieval per app.

Ready to dig in? Start with the Introduction for the core concepts, or jump straight to Getting Started to run it yourself.