How to run local LLMs with Ollama

Ollama is the simplest way to run AI models (LLMs) on your server. The data stays with you, and the cost is fixed, not per token.

1. Install

curl -fsSL https://ollama.com/install.sh | sh

2. Download and run a model

ollama run llama3.2

The first run downloads the model; then you can chat directly in the terminal.

3. Use it through the API

Ollama exposes a local API on port 11434:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Hello"
}'

4. Choose the right model

Small models (1-3 billion parameters) run on CPU; large ones need a GPU. See how to choose a model for your hardware.

Do you need a GPU?

For large models and fast responses, a server with a GPU makes the difference. See the GPU plans and when you need a GPU. Add a chat interface with Open WebUI.