Best Open Source LLM: Pick by VRAM First, Benchmark Second

Every open-model leaderboard sorts by benchmark score, which is the wrong first sort — because a model that does not fit in your available VRAM scores zero on every benchmark that matters to you, no matter what the table says.

Choose in the opposite order. Start with the hardware you have or are willing to rent, work out what fits at what quantisation, and only then compare quality among the models that are actually runnable. That inverted order eliminates most of the leaderboard immediately and turns a sprawling decision into a short one. Here is how to do it, without naming models whose rankings will have moved by the time you read this.

The VRAM arithmetic, which is simple and unforgiving
Quantisation: what you are actually trading
Licences, which are the part people skip and regret
The serving stack, and why it decides your throughput
Should you self-host at all
How this fits the rest of the stack
FAQ

The VRAM arithmetic, which is simple and unforgiving

Weights have to be in memory to run inference. The rough rule: parameters multiplied by bytes-per-parameter, plus overhead for the context window and the runtime.

FP16 / BF16 — about 2 bytes per parameter. A 7B model needs roughly 14GB for weights alone. A 70B needs roughly 140GB, which is multiple GPUs.
INT8 — about 1 byte per parameter. Roughly halves it, with quality loss that is small for most tasks.
INT4 — about 0.5 bytes per parameter. A 7B fits in around 4GB, a 70B in around 35GB. Quality loss is noticeable but frequently acceptable.
KV cache — additional memory that grows with context length and batch size. At long contexts this stops being a rounding error and becomes a significant fraction of the total.

So a 24GB consumer GPU comfortably runs a mid-size model at 4-bit or a smaller one at 8-bit. A 16GB card is more constrained than people expect once context is accounted for. Anything approaching 70B at reasonable precision means either multiple GPUs or renting.

This arithmetic, done first, usually cuts the candidate list from dozens to a handful — and it is the same handful regardless of which leaderboard you were reading.

Quantisation: what you are actually trading

Quantisation stores weights at lower precision. It is what makes local inference practical, and it is not free.

The degradation is not uniform across tasks. Conversational fluency and summarisation hold up well at 4-bit — most people cannot reliably distinguish the output. Code generation, mathematical reasoning and precise instruction-following degrade more noticeably, because they depend on exactness rather than plausibility.

The practical heuristic that has held up: a larger model quantised harder generally beats a smaller model at full precision, within the same memory budget. A 4-bit 13B typically outperforms an 8-bit 7B in the same VRAM. Test this on your own workload rather than taking it on faith, but it is the right starting hypothesis.

Not all 4-bit is the same. Quantisation methods differ meaningfully in how much quality they preserve at the same bit width, and the gap between a naive and a good method is larger than the gap between adjacent model sizes. Use a well-regarded quantisation of a model rather than the first file you find.

Licences, which are the part people skip and regret

Open source is used loosely in this space. Models described as open span genuinely permissive licences and licences with restrictions that would surprise you if you found out during a commercial review.

Categories worth distinguishing:

Genuinely permissive — Apache 2.0 or MIT. Commercial use, modification and redistribution, no strings. This is what open source means everywhere else.
Custom open-weight licences — weights published under bespoke terms. Common restrictions include usage thresholds above which separate permission is required, prohibitions on using outputs to train competing models, and acceptable-use clauses.
Research-only — no commercial use at all. Perfectly legitimate, and fatal if discovered after you built a product on it.

Read the actual licence file before building anything commercial on a model. Not the announcement post, not the README summary, not a comparison article. The licence is a short document and reading it takes ten minutes.

This matters more than it did, because several widely-used families sit in the middle category with terms that are fine for most companies and not fine for some.

The serving stack, and why it decides your throughput

The model is half the decision. What serves it determines whether you get usable throughput or a demo that falls over at four concurrent users.

Broadly there are two kinds of runtime. Single-user tools optimised for running a model locally are excellent for development and evaluation, and they are not built to serve concurrent traffic. Production inference servers implement continuous batching, paged attention and similar techniques, and deliver dramatically better throughput under concurrency.

The difference is not marginal. A production server can serve many times the requests per second of a development runtime on identical hardware, because it keeps the GPU busy rather than processing one request at a time. If you are serving an application, this choice matters as much as the model choice.

Also budget for what surrounds the model: a queue for bursts, timeouts, a fallback for when the GPU is saturated, and monitoring on latency percentiles rather than averages. Inference latency distributions have long tails, and an average hides the experience of the users having a bad time.

Should you self-host at all

The honest answer for most teams is no, at least not initially, and the reasoning is arithmetic rather than ideology.

Self-hosting makes sense when at least one of these is true: data cannot leave your infrastructure for regulatory or contractual reasons; your volume is high and steady enough that a continuously-utilised GPU beats per-token pricing; you need a model fine-tuned on proprietary data; or you require guarantees about model versioning that an API cannot give you.

It makes less sense when volume is low or bursty, because a GPU idle 90% of the time is an expensive way to buy tokens. It makes less sense when you have no one to operate it, because inference infrastructure is infrastructure and it needs the same care as anything else in production.

The pattern that works: start with an API, measure real usage for a month, and calculate the crossover point against a rented GPU at your actual utilisation. The crossover is usually further away than enthusiasm suggests, and when you reach it the decision is a calculation rather than an argument.

Either way, the application around the model is ordinary infrastructure — a service handling requests, a database holding conversations and results, storage for documents, and a queue for anything slow. That part exists regardless of where the weights live, and it is where most of the engineering actually goes.

How this fits the rest of the stack

Whether the model runs on your hardware or behind someone’s API, the application around it is the same shape: a service taking requests, a database holding state, storage for documents, and bandwidth. The RunxBuild hosting calculator puts those lines together so the recurring cost is visible before the build. RunxBuild hosts that layer — Python, Node, Go and Docker services from a repository, managed Postgres or MySQL beside them, and autoscaling between a floor and ceiling plan for traffic that arrives unevenly.

Useful related references:

FAQ

How much VRAM do I need to run an open source LLM?

Roughly 2 bytes per parameter at FP16, 1 byte at INT8, and 0.5 bytes at INT4, plus overhead for the KV cache which grows with context length and batch size. A 7B model needs about 14GB at full precision or about 4GB at 4-bit. Work out what fits before comparing benchmark scores.

Is a quantised larger model better than a smaller full-precision one?

Usually, within the same memory budget — a 4-bit 13B typically outperforms an 8-bit 7B. The degradation from quantisation is uneven though: conversation and summarisation hold up well at 4-bit, while code generation and mathematical reasoning suffer more. Test on your own workload.

Are open source LLMs really free to use commercially?

Not all of them. Some are Apache 2.0 or MIT with no restrictions. Many are published under custom open-weight licences with usage thresholds, restrictions on training competing models, or acceptable-use clauses, and some are research-only. Read the licence file before building a commercial product on any of them.

What should I use to serve an open source LLM in production?

A production inference server that implements continuous batching, rather than a single-user local runtime. The throughput difference under concurrency is large on identical hardware, because the production server keeps the GPU busy instead of handling one request at a time.

Is self-hosting an LLM cheaper than using an API?

Only at high, steady utilisation. A GPU idle most of the time is an expensive way to buy tokens. Start with an API, measure a month of real usage, and calculate the crossover against a rented GPU at your actual utilisation — it is usually further away than expected. Data residency requirements can override the arithmetic entirely.