RTX 5060 Ollama Benchmarks: Best GPU for 8GB LLMs Below
The NVIDIA RTX 5060 GPU with 8GB of VRAM is an affordable yet surprisingly capable option for running open-source large language models (LLMs) locally. With the help of Ollama—a user-friendly framework for running quantized LLMs—it’s now easier than ever to deploy models like LLaMA, DeepSeek, Mistral, and Phi on consumer-grade hardware.
Test Overview
Server Configs:
- CPU: 24-Core Platinum 8160
- RAM: 64GB RAM
- Storage: 120GB SSD + 960GB SSD
- Network: 1Gbps
- OS: Ubuntu 22.0
A Single 5060 Details:
- GPU: Nvidia GeForce RTX 5060
- Microarchitecture: Blackwell 2.0
- Compute Capability: 12.0
- CUDA Cores: 3840
- Tensor Cores: 144
- Memory: 8GB GDDR7
We benchmarked 18 popular models on the RTX 5060 8GB using Ollama 0.9.5, and here are the results.
Benchmarking Ollama Results on Nvidia RTX5060
| Models | deepseek-r1 | deepseek-r1 | deepseek-r1 | gemma3n | gemma3n | gemma3 | gemma3 | qwen3 | qwen3 | qwen3 | qwen3 | llama3.1 | llama3.2 | llama3.2 | phi4-mini | phi3.5 | phi | mistral |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Parameters | 1.5b | 7b | 8b | e2b | e4b | 1b | 4b | 0.6b | 1.7b | 4b | 8b | 8b | 1b | 3b | 3.8b | 3.8b | 2.7b | 7b |
| Size (GB) | 1.1 | 4.7 | 5.2 | 5.6 | 7.5 | 0.815 | 3.3 | 0.523 | 1.4 | 2.6 | 5.2 | 4.9 | 1.3 | 2.0 | 2.5 | 2.2 | 1.6 | 4.4 |
| Quantization | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 |
| Running on | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 | Ollama0.9.5 |
| Downloading Speed(mb/s) | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 | 114 |
| GPU UTL | 54% | 72% | 79% | 68% | 70% | 59% | 77% | 67% | 77% | 85% | 90% | 75% | 57% | 60% | 61% | 80% | 68% | 88% |
| Eval Rate(tokens/s) | 111.06 | 58.46 | 53.91 | 56.11 | 39.51 | 146.64 | 80.72 | 210.97 | 156.39 | 89.59 | 61.70 | 58.07 | 133.49 | 96.41 | 75.00 | 113.89 | 131.47 | 72.88 |
Analysis & Insights
1. Surprisingly High Token Speeds
2. Excellent Fit for 8GB VRAM
3. Ollama 0.9.5 Compatibility Is Strong
4. Best Speed-to-Quality Trade-offs
5. Download Bottleneck Irrelevant
6. Best Models for Low-Latency Inference
7. Real-Time LLM on Budget GPUs Is Viable
RTX 5060 GPU Hosting for 10B LLMs Below
Basic GPU Dedicated Server - RTX 5060
- 64GB RAM
- GPU: Nvidia GeForce RTX 5060
- 24-Core Platinum 8160
- 120GB SSD + 960GB SSD
- 100Mbps-1Gbps
- OS: Windows / Linux
- Single GPU Specifications:
- Microarchitecture: Blackwell 2.0
- CUDA Cores: 4608
- Tensor Cores: 144
- GPU Memory: 8GB GDDR7
- FP32 Performance: 23.22 TFLOPS
Enterprise GPU Dedicated Server - RTX 4090
- 256GB RAM
- GPU: GeForce RTX 4090
- Dual 18-Core E5-2697v4
- 240GB SSD + 2TB NVMe + 8TB SATA
- 100Mbps-1Gbps
- OS: Windows / Linux
- Single GPU Specifications:
- Microarchitecture: Ada Lovelace
- CUDA Cores: 16,384
- Tensor Cores: 512
- GPU Memory: 24 GB GDDR6X
- FP32 Performance: 82.6 TFLOPS
Enterprise GPU Dedicated Server - A100
- 256GB RAM
- GPU: Nvidia A100
- Dual 18-Core E5-2697v4
- 240GB SSD + 2TB NVMe + 8TB SATA
- 100Mbps-1Gbps
- OS: Windows / Linux
- Single GPU Specifications:
- Microarchitecture: Ampere
- CUDA Cores: 6912
- Tensor Cores: 432
- GPU Memory: 40GB HBM2
- FP32 Performance: 19.5 TFLOPS
Enterprise GPU Dedicated Server - RTX 5090
- 256GB RAM
- GPU: GeForce RTX 5090
- Dual 18-Core E5-2697v4
- 240GB SSD + 2TB NVMe + 8TB SATA
- 100Mbps-1Gbps
- OS: Windows / Linux
- Single GPU Specifications:
- Microarchitecture: Blackwell 2.0
- CUDA Cores: 21,760
- Tensor Cores: 680
- GPU Memory: 32 GB GDDR7
- FP32 Performance: 109.7 TFLOPS
Conclusion: Best LLMs for RTX 5060
The RTX 5060 (8GB) paired with Ollama 0.9.5 supports dozens of quantized models, delivering surprisingly high throughput.
If you need to run lightweight chatbots, coding assistants, or inference pipelines on a budget GPU, the RTX 5060 is more than capable with these optimized LLMs.
Nvidia RTX 5060 Hosting, Ollama benchmarks, LLM performance, RTX 5060 AI, Ollama 0.9.5, local LLM hosting, DeepSeek RTX, Qwen3 Ollama, Phi LLM, Mistral Ollama, llama3 benchmarks, Gemma3 inference speed, Ollama GPU benchmark, RTX 5060 AI server