Prometheus: Combining Labels Without Losing the Data You Need

Q: What is the difference between by and without in PromQL?

by names the labels to keep and drops the rest; without names labels to drop and keeps the rest. Use without when stripping infrastructure labels like instance, so new labels are kept automatically, and by when you want an explicitly bounded result.

Q: How do I combine two metrics with different names?

Use or to union them, which is useful during a metric rename so dashboards keep working. If the label names differ, normalise them first with label_replace so the two sides share a matching key.

Q: Should I aggregate before or after rate()?

Always take the rate first, then aggregate: sum(rate(x[5m])). Summing counters before computing the rate breaks the counter-reset detection that rate relies on and produces wrong numbers.

Aggregating in PromQL discards every label except the ones you name in by(). That is the single most important thing to know, and the cause of most queries that return nothing or return one useless number.

PromQL treats labels as the dimensions of your data, and aggregation collapses dimensions. sum(http_requests_total) gives you one number across every service, endpoint, status code and instance — technically correct and almost never what you wanted.

The useful queries are about controlling exactly which dimensions survive, and about joining series whose labels do not line up.

by and without
The empty-result problem
Many-to-one joins with group_left
Combining metrics that do not overlap
Cardinality, and the query that kills your Prometheus
Metrics you actually get to query
How this fits the rest of the stack
FAQ

by and without

Two ways to say the same thing from opposite directions:

# Keep ONLY these labels
sum by (service, status) (rate(http_requests_total[5m]))

# Keep everything EXCEPT these labels
sum without (instance, pod) (rate(http_requests_total[5m]))

by is an allowlist, without is a denylist. Prefer without when you are aggregating away infrastructure detail — instance, pod, container — because a new label added by your service later is automatically kept rather than silently dropped.

Prefer by when you want an explicitly bounded result, such as a dashboard panel that must show exactly one line per service regardless of what labels appear later.

Note the ordering rule: with rate, always take the rate first and aggregate second. sum(rate(x[5m])) is correct; rate(sum(x)[5m:]) produces wrong numbers because summing counters across series breaks the reset detection that rate depends on.

The empty-result problem

A binary operation between two metrics only matches series whose label sets are identical. This is where most “my query returns nothing” cases come from:

# Returns empty if the two metrics carry different labels
sum by (service) (rate(http_errors_total[5m]))
/
sum by (service) (rate(http_requests_total[5m]))

That particular one works, because both sides were aggregated down to the same single label. Aggregating both sides to a common label set is the standard fix and the reason you see by (service) on both operands of a ratio.

When the label sets genuinely differ and you cannot aggregate them away, use on or ignoring:

# Match only on the labels they share
rate(http_errors_total[5m]) / on (service, endpoint) rate(http_requests_total[5m])

# Match on everything except these
rate(http_errors_total[5m]) / ignoring (status) rate(http_requests_total[5m])

Debug this by running each side alone and comparing the label sets in the output. Nine times out of ten one side has an extra label — status, code, le — that prevents the match.

Many-to-one joins with group_left

The other common case: one series on the left for every N on the right, or vice versa. Prometheus refuses ambiguous matches unless you say which side is which.

# Attach a per-service quota to per-instance usage
rate(requests_total[5m])
  * on (service) group_left (tier)
  service_metadata

group_left means the left side has many series per matching right-side series, and the extra labels named in the parentheses are copied from the right onto the result. group_right is the mirror image.

This is the standard pattern for enriching metrics with metadata — attaching a team, tier, or environment label from an info-style metric onto operational data. It is worth learning because the alternative is duplicating that metadata as labels on every metric, which bloats cardinality for no reason.

The classic form uses an info metric that is always 1:

sum by (team) (
  rate(http_requests_total[5m])
    * on (service) group_left (team)
    service_info
)

Combining metrics that do not overlap

or unions two vectors, taking the right side only where the left has no series at that label set:

# Old and new metric names during a migration
rate(http_requests_total[5m]) or rate(http_server_requests_total[5m])

# Fill in a zero where a metric is absent
sum by (service) (rate(errors_total[5m])) or vector(0)

The migration case is genuinely useful: a dashboard keeps working across a rename without a flat spot in the graph.

When the label names differ but mean the same thing, normalise with label_replace:

label_replace(old_metric, "service", "$1", "job", "(.*)")

That copies job into a new service label so the two sides can match. Read the arguments as: source vector, destination label, replacement (with capture references), source label, regex.

There is also label_join for concatenating several labels into one, which is occasionally the cleanest way to build a matching key out of parts.

Cardinality, and the query that kills your Prometheus

Every unique combination of label values is a separate time series stored separately. A metric with 10 services, 50 endpoints, 8 status codes and 100 instances is 400,000 series from one metric name.

The rules that keep this manageable:

Never put an unbounded value in a label. User IDs, request IDs, full URL paths, email addresses, timestamps. Each unique value is a permanent new series.
Template your paths. /users/{id}, not /users/8f3a-11.
Aggregate away instance labels in dashboards. You rarely need per-pod detail on an overview panel, and querying it is expensive.
Check before you add. count(count by (label_name) (metric)) shows how many series a metric has.

For queries you run constantly — dashboard panels, alert expressions — precompute them with a recording rule rather than recalculating on every evaluation:

groups:
  - name: http
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service, status) (rate(http_requests_total[5m]))

The naming convention — level:metric:operation — is worth following. It tells a reader what has already been aggregated away without opening the rule file.

Metrics you actually get to query

All of this presumes metrics exist and are retained. Getting there is its own work: instrumenting the application, running a Prometheus server, sizing its storage, and deciding retention — and Prometheus stores locally by default, so a restart without persistent storage loses history.

That is a reasonable amount of infrastructure for a small team, which is why the first useful step is usually not a full Prometheus deployment but simply having per-service metrics and logs available at all.

On RunxBuild, services carry runtime metrics and logs per deploy alongside the build that shipped them, with autoscaling driven by CPU thresholds you set. It is not a replacement for Prometheus on complex infrastructure, and for a handful of services it answers most of the questions a dashboard would.

How this fits the rest of the stack

Aggregation drops every label you do not name, binary operations need identical label sets to match, and group_left is how you attach metadata to operational data. Take rates before aggregating, use without when stripping infrastructure labels, and keep unbounded values out of labels unless you enjoy cardinality incidents. For a handful of services, per-deploy metrics and logs answer most of it — the RunxBuild hosting calculator shows what running those services costs.

Useful related references:

FAQ

What is the difference between by and without in PromQL?

by names the labels to keep and drops the rest; without names labels to drop and keeps the rest. Use without when stripping infrastructure labels like instance, so new labels are kept automatically, and by when you want an explicitly bounded result.

Why does my Prometheus query return no data?

Usually because a binary operation is comparing two vectors with different label sets, which cannot match. Run each side separately and compare labels — typically one side has an extra label. Fix it by aggregating both sides to a common set or using on / ignoring.

What does group_left do?

It resolves a many-to-one match, telling Prometheus the left side has multiple series per right-side series, and copies the named labels from the right onto the result. It is the standard way to enrich metrics with metadata from an info-style metric.

How do I combine two metrics with different names?

Use or to union them, which is useful during a metric rename so dashboards keep working. If the label names differ, normalise them first with label_replace so the two sides share a matching key.

Should I aggregate before or after rate()?

Always take the rate first, then aggregate: sum(rate(x[5m])). Summing counters before computing the rate breaks the counter-reset detection that rate relies on and produces wrong numbers.