Private LLM Stack
Local, offline AI · Ollama + Open WebUI on a GPU
Summary
A fully self-hosted AI stack. Ollama serves local language models on an NVIDIA GPU, with Open WebUI as the chat front end. Nothing leaves the network, so it's a private
sandbox for running models, testing prompts, and providing offline reasoning to the
agent team.
01How it works
Ollama pulls and runs open models locally and exposes an API; Open WebUI gives it a clean ChatGPT-style interface with conversations, model switching and prompt management. Because it's all on-network, I can work with sensitive or lab data without sending anything to a third-party API. The same Ollama endpoint also doubles as a private inference backend for automation.
02The stack
| Component | Image | Role |
|---|---|---|
| Ollama | ollama/ollama | Local model server with NVIDIA GPU passthrough; API bound to localhost (11434). |
| Open WebUI | ghcr.io/open-webui/open-webui | Web chat front end, talks to Ollama over an internal Docker network. |
| Shared network | ai-net (Docker) | Private bridge so the two containers: and the agent services: reach Ollama by name, not exposed port. |
03Why run it locally
- Self-hosted AI: running and serving LLMs locally with GPU acceleration.
- Data privacy by design: an offline alternative to cloud AI, so nothing sensitive leaves the network.
- Integration: the local endpoint backs other projects (the agent team) instead of a paid API.
- Container networking: a private shared network keeps the API off any public port.