How to set up your PC to run local AI models

Step 1 — Check what your PC can run

Model size is limited by memory before anything else. A model that fits in your graphics card's VRAM runs fast; one that does not has to be split with system RAM, which works but is considerably slower; and one that fits in neither will not load at all.

Finding this out by downloading 20 GB of weights is the slow way round, so check first.

Step 2 — Install Ollama

Download the installer for your operating system and run it. On Windows it installs a background service and adds the ollama command; there is nothing to configure afterwards.

Windows, macOS and Linux. Downloading only from ollama.com is worth the habit — mirrors of popular AI tools are a common way to distribute something else.

Step 3 — Download and run your first model

Open a terminal — Command Prompt or PowerShell on Windows, Terminal on macOS or Linux — and run one command. It downloads the model the first time and starts a chat with it:

ollama run gemma3

When it finishes you get a prompt. Type a question, press Enter, and the answer is generated on your own hardware. Type /bye to leave the chat.

Start small even if your PC can handle more — a 3B or 4B model confirms the whole setup works in a few minutes, and you can move up once you know it does. Replace the name with any model from the catalog, which lists what each one needs.

Step 4 — The commands worth knowing

These five cover almost everything day to day:

ollama pull qwen3:8b

ollama list

ollama rm qwen3:8b

ollama show gemma3

ollama ps

A name like qwen3:8b is the model followed by the build. The part after the colon picks the size and quantization; leaving it off gives the default, which is usually a mid-sized 4-bit build. Each model page on this site lists the builds available and the memory each one needs.

Where the models are stored

Weights are several gigabytes each and go on your system drive by default:

If your system drive is tight, set the OLLAMA_MODELS environment variable to a folder on another disk and restart Ollama. Models already downloaded stay where they are unless you move them yourself.

Using it from your own applications

Ollama serves a local API on port 11434 while it is running. That is what desktop chat front-ends and editor extensions connect to, and it is how you would call a model from your own code — nothing leaves your machine:

curl http://localhost:11434/api/generate -d "{\"model\":\"gemma3\",\"prompt\":\"Why is the sky blue?\"}"

If it runs slowly

• Check it is using the GPU. Run ollama ps while a model is loaded — it reports how much sits on the GPU. A model shown entirely on the CPU is the usual reason for slow replies.
• Use a smaller build. A model that does not fit in VRAM is split with system RAM, and the offloaded part runs at a fraction of the speed. Dropping one size is often dramatically faster.
• Shorten the context. The KV cache grows with the context window and competes with the weights for the same memory, so a long context can push a model that otherwise fits out of VRAM.
• Update your GPU drivers. Acceleration depends on them, and an old driver can mean the GPU is not used at all.

Frequently asked questions

Do I need a graphics card to run AI models locally?

No, but it changes what is practical. Ollama runs on the CPU alone using system RAM, which works for small models and is slow for large ones. A dedicated GPU holds the model in VRAM instead and is typically many times faster, so the amount of VRAM you have decides which models are comfortable rather than merely possible.

How much RAM and VRAM do I need?

As a rule of thumb a 4-bit quantized model needs roughly 0.6 GB per billion parameters, plus memory for the context window and runtime overhead. An 8B model therefore wants about 6 GB of VRAM at a short context. That is only a guide — the scan on this site calculates it from your actual hardware and the exact build you plan to run.

Is Ollama free, and does it work offline?

Ollama is free and open source, and no account is required. Downloading a model needs an internet connection, but once the weights are on your disk the model runs entirely on your machine, offline, and your prompts are never sent anywhere.

Where does Ollama store the models?

On Windows the weights go in %USERPROFILE%\.ollama\models, on macOS in ~/.ollama/models, and on Linux in /usr/share/ollama/.ollama/models. Models are several gigabytes each, so if your system drive is short of space you can point Ollama elsewhere by setting the OLLAMA_MODELS environment variable to another folder.

What model should I start with?

Start with a small one — a 3B or 4B build downloads in a few minutes, runs on modest hardware, and is enough to confirm everything works. Once you know the setup is fine, move up to the largest model your hardware supports comfortably.

Can I use Ollama without the command line?

Yes. Ollama exposes a local API on port 11434, and several desktop and browser front-ends connect to it to give you a chat window. The command line is only the quickest way to get the first model running.