The Open-Source AI Repos Worth Knowing, By Layer
Open-source AI projects cluster into six layers: model runtimes that execute models locally, embedding models that turn text into vectors, vector stores that search them, retrieval and agent frameworks that orchestrate calls, protocol servers that connect agents to systems, and evaluation tools that tell you whether any of it works. Pick by layer, not by popularity.
Layer 1: model runtimes
These execute a model on hardware you control.
llama.cpp made local inference on ordinary machines practical, with quantized models that fit in consumer memory. It is the substrate under a large fraction of the local AI ecosystem, and worth reading even if you never invoke it directly.
Ollama wraps local model management into something you can install and use in a minute, which is why it is usually the fastest way to get a local setup running for experimentation.
vLLM solves a different problem: serving many concurrent requests efficiently on GPUs. If you are building a service rather than a workstation setup, this is the layer you care about.
Hugging Face transformers remains the reference implementation for loading and running models across architectures, and the place you go when you need to do something the convenience tools do not expose.
Choose by deployment shape. One user on a laptop, many users on a GPU, or research flexibility are three different answers.
Layers 2 and 3: embeddings and vector stores
sentence-transformers is the practical entry point for embedding models: turning text into vectors you can compare by meaning. This is the layer that most determines retrieval quality, and it is chronically under-attended because it feels like a solved dependency.
FAISS is the similarity search library that most vector infrastructure is built on or measured against. Understanding it clarifies what a hosted vector database is actually doing for you.
Qdrant and Chroma occupy the database tier: persistence, filtering, and metadata alongside vectors. Chroma optimizes for getting started, Qdrant for running something in production with filtering that has to be fast.
The common mistake at this layer is spending a week choosing a vector database and an hour choosing an embedding model. Retrieval quality moves far more with the second decision. If your retrieved passages are plausible but not useful, that is nearly always an embedding, chunking, or query formulation problem rather than a storage one.
Chunking is not a library decision
No repo will chunk your documents well by default, because good boundaries depend on what your documents are. A retrieved chunk must be understandable alone, which means carrying its section title and enough surrounding context to identify what it describes. This is the highest leverage hour of work in most retrieval systems.
Layers 4 and 5: orchestration and connection
LangChain and LlamaIndex sit at the orchestration layer: chaining calls, wiring retrieval into generation, managing prompts and memory. Both are genuinely useful for getting something working quickly and both introduce abstractions you will eventually want to see through. The recommendation that has aged well is to write your loop by hand once, then adopt a framework knowing exactly which part of it you are outsourcing.
Model Context Protocol servers are the connection layer. The open collection of reference servers is the fastest way to understand the protocol, because reading three server implementations teaches more than the specification does. This is where you go when an agent needs access to a system rather than knowledge about one.
open-webui deserves a mention as the interface layer for local models, since a usable chat surface over your own runtime is often what makes a local setup something you actually use rather than something you configured once.
Layer 6: evaluation, and the layer everyone skips
promptfoo and Ragas cover evaluation, the former for comparing prompts and models on your own test cases, the latter aimed at retrieval pipelines specifically. This layer is where projects become improvable. Without it, changes are judged by impression, which reliably improves whatever you are currently looking at and quietly degrades everything else.
The layer nobody lists as a repo is the one that determines whether the rest is worth running: the knowledge your agents retrieve from. A perfect stack over a corpus that does not contain your team's actual knowledge produces confident, well engineered answers to the wrong question.
RDK addresses that layer directly. Files from local vaults, docs, and code are indexed as encrypted private chunks that agents search before querying a model, which cuts token spend 80 to 90 percent because the answer is retrieved instead of regenerated. Stacked retrieval sets the proportions: private material answers 40 to 65 percent of queries, the public network adds 15 to 20 percent, and the model handles the remaining 5 to 10 percent. Authors of public chunks earn USDC per retrieval, which is what makes publishing good knowledge worth doing rather than merely generous.
How to actually pick
Name the layer where your current problem lives before you evaluate anything. Slow responses under load is a runtime problem. Irrelevant retrieved passages is an embedding and chunking problem. An agent that acts wrongly is a tool design problem. Adopting a repo from the wrong layer is how teams end up with an impressive stack and an unchanged failure rate.
Frequently asked questions
- Which open-source AI repo should I learn first?
- Whichever sits at the layer of your current problem. If you want local inference, start with Ollama or llama.cpp. If retrieval quality is your problem, start with sentence-transformers and your own chunking. Learning a popular repo from a layer you are not working at produces familiarity without progress.
- Do I need a vector database?
- Not at first. A similarity search library over a modest corpus is enough to learn whether your retrieval works at all, and it removes a moving part while you debug embeddings and chunking. Move to a database when you need persistence, metadata filtering, or concurrent access, which is a scaling concern rather than a quality one.
- Are agent frameworks worth adopting?
- After you have written a loop by hand. The core loop is around a hundred lines, and building it once shows you exactly which decisions a framework is making on your behalf. Adopt one knowing what you are outsourcing, and be willing to drop it when its abstraction stops matching what you need.
- What determines retrieval quality most?
- Embeddings and chunking, not the vector store. Most teams spend a week choosing a database and an hour on the embedding model, which is backwards. If retrieved passages are plausible but unhelpful, the problem is almost always chunk boundaries, missing context inside chunks, or how the query is formulated.