TL;DR: AnythingLLM paired with Ollama gives you private RAG your whole team can use: chat with internal documents through a familiar chat interface, without any document, prompt, or answer ever leaving your own network. The setup takes four pieces: Ollama running a local LLM and an embedding model, AnythingLLM as the document ingestion and chat layer, a local vector database, and workspace permissions to control who can see what. Here is the full setup, end to end.
What Private RAG Is, and Why Offline Matters
Retrieval-augmented generation, or RAG, lets a language model answer questions using your own documents instead of only what it learned during training: a question comes in, the system retrieves the most relevant chunks of your documents from a vector database, and passes them to the model as context before it generates an answer. That is what makes it possible to ask a model about last quarter’s contract terms or an internal policy document and get a grounded answer instead of a guess.
The “private” part matters as much as the “RAG” part for most businesses. Contracts, HR records, financials, and client files are exactly the documents a team wants an assistant to search, and exactly the documents that should not be uploaded to a third-party AI service as part of that process. Running the entire pipeline offline, on infrastructure you control, means the LLM, the embedding model, the vector database, and the documents themselves never touch an external server.
The Four Pieces of an AnythingLLM and Ollama RAG Stack
Everything in this walkthrough fits into four moving parts:
Ollama
Runs the local LLM and a dedicated embedding model that AnythingLLM calls over its local API on port 11434.
AnythingLLM
Handles document ingestion, chunking, retrieval, and the chat interface. Desktop app for one user, Docker for a team.
Vector database
AnythingLLM ships with LanceDB running locally by default, enough for most teams with no extra setup.
Workspace permissions
One workspace per department keeps each team’s documents and chat history isolated from every other team.
Step 1: Install Ollama and Pull an LLM and an Embedding Model
Ollama is the local runtime that serves both the chat model and the embedding model AnythingLLM will call. After installing Ollama, pull a capable instruction-tuned model sized to your hardware, for example “ollama pull llama3.1:8b” or “ollama pull qwen2.5:14b” for stronger reasoning if you have the VRAM. Then pull a dedicated embedding model, which is what converts document text into vectors for retrieval: “ollama pull nomic-embed-text” is a solid, lightweight default. Using a purpose-built embedding model rather than asking the chat model to double as an embedder produces noticeably better retrieval quality.
Confirm both models are available by running “ollama list”, and leave the Ollama service running in the background, since AnythingLLM will connect to it over its local API on port 11434.
Step 2: Install and Configure AnythingLLM
AnythingLLM is the layer that handles document ingestion, chunking, retrieval, and the chat interface itself. It is available as a desktop application for a single-user setup, or as a Docker container for anything you want reachable by a team over the network. For a team deployment, running the Docker image on a server or VM inside your own network is the right choice, since it puts the chat interface on an internal URL every teammate can reach without installing anything locally.
On first launch, AnythingLLM walks through an initial setup wizard where you choose the LLM provider, the embedding provider, and the vector database. This is the point where the whole stack gets pointed at your local Ollama instance instead of a cloud API.
Step 3: Connect Ollama as the LLM and Embedding Provider
In AnythingLLM’s settings, set the LLM provider to Ollama and point it at your Ollama base URL, typically “http://localhost:11434” if AnythingLLM runs on the same machine, or the server’s internal IP address if it runs elsewhere on your network. Select the chat model you pulled earlier from the model dropdown. Then, separately, set the embedding provider to Ollama as well and select your embedding model, since AnythingLLM treats the chat model and embedding model as two independent settings even though both run through the same Ollama instance.
For the vector database, AnythingLLM ships with LanceDB built in and running locally by default, which is enough for most teams and requires no extra setup. Larger deployments with heavier document volumes can point AnythingLLM at a dedicated vector database instead, but that is an optimization to consider later, not a requirement to start.
Step 4: Ingest Your Documents
Documents are uploaded per workspace, either through the web interface or by pointing AnythingLLM at a folder to bulk-import. Supported formats cover the common business document types: PDF, Word, plain text, CSV, and even scraped web pages. On upload, AnythingLLM chunks each document into smaller passages and embeds each chunk using the embedding model you configured, storing the resulting vectors for retrieval.
Chunk size and overlap are configurable and worth revisiting once you see real query results: smaller chunks improve precision for short factual lookups, while larger chunks preserve more context for questions that need a fuller passage to answer well. Start with the defaults, test with real questions your team will ask, and adjust from there rather than tuning blind.
Step 5: Workspaces and Permissions for Teams
AnythingLLM organizes documents and chat history into workspaces, and this is the main tool for controlling who sees what. A common structure is one workspace per department, HR, finance, legal, sales, each populated only with the documents relevant to that team, so a query in the sales workspace cannot surface HR files even accidentally. Enabling multi-user mode adds login accounts and role-based access, so employees only see the workspaces they have been granted access to, which matters as soon as more than one or two people are using the system.
Keeping It Fully Local
Every piece of this stack, Ollama, the LLM, the embedding model, AnythingLLM, and the vector database, can run entirely inside your own network with no outbound calls to a third-party AI provider once configured correctly. That is the entire point for a business handling confidential documents: employees get the chat-with-your-documents experience they expect from consumer AI tools, without any of those documents leaving the building. Getting this configured correctly the first time, including firewall rules, backups, and model selection for your actual hardware, is exactly what private, local Ollama deployment for business is built to handle.
Scaling to a Team
A pilot with a handful of documents on one laptop and a rollout to fifty employees across five departments are different engineering problems. Scaling private RAG to a full team means sizing the underlying GPU for concurrent chat and embedding requests, deciding whether a single shared Ollama instance or multiple instances behind a load balancer fits your usage pattern, and building out workspace and permission structures before broad rollout rather than after. This is the point where a structured private RAG deployment pays for itself: getting the architecture, document governance, and hardware sizing right before a hundred employees are depending on it daily, rather than rebuilding under pressure once it breaks.
AnythingLLM and Ollama get a private, document-aware assistant running quickly, but a setup that holds up for a whole team, with the right hardware, permissions, and document governance, takes more planning. Our private RAG, chat-with-your-documents solution and private Ollama deployment service handle that build for you.
Want private RAG built over your documents?
Book an AI strategy session →The bottom line
Ollama plus AnythingLLM gets private, document-aware chat running on infrastructure you control in an afternoon, but scaling it to a full team means sizing GPU capacity, workspace permissions, and document governance up front. That is what a private RAG deployment built on private, local Ollama is designed to handle.