How to choose an AI model for your hardware

AI models come in different sizes, measured in billions of parameters (B). The bigger they are, the more capable, but they need more memory.

The basic memory rule

A quantized (compressed) model needs roughly as many GB of memory as it has billions of parameters:

• 3B: ~3-4 GB, runs on CPU with enough RAM.
• 7-8B: ~6-8 GB, ideal on an entry GPU.
• 13B: ~10-14 GB, needs a decent GPU.
• 70B: ~40+ GB, needs a powerful GPU or several.

CPU or GPU

On CPU, small models work, but slowly. On a GPU, everything is tens of times faster. See when you need a GPU.

Quantization

Quantized versions (Q4, Q5) reduce the memory needed with little quality loss. For most cases, a Q4 quantization is an excellent compromise.

How to choose in practice

1. Start with a small model and test whether it solves your task.
2. Move to a bigger one only if you need more quality.
3. Match the size to the available memory; see the GPU plans for large models.

With hourly billing you can test a large model on a GPU for a few hours, to see if it is worth it, with no commitment.