Configure tail sampling#

This guide shows you how to tune which traces the Alloy gateway keeps. It assumes tracing is already enabled; if not, see Enable tracing first.

The gateway tail-samples: it waits for a trace to complete, then decides whether to store it. You control that decision through the tempo.tailSampling fields. For the reasoning behind tail sampling, see Tracing architecture.

The sampling policy#

The gateway keeps a trace when any of these hold.

  • The trace contains an error span. This rule is always on and is not configurable.

  • The trace is slower than tempo.tailSampling.latencyThresholdMs milliseconds.

  • The trace is in the tempo.tailSampling.probabilisticPercent percent of the remaining traces kept as a representative sample.

Every error and every slow trace is kept regardless of the probabilistic percentage. The percentage applies only to the traces that are neither errors nor slow.

Set the thresholds#

Override the two fields in your mergeConfig input. The values below show the defaults.

const config = mergeConfig({
  // other required cluster values ...
  tempo: {
    enabled: true,
    bucket: 'traces-tempo-prod',
    tailSampling: {
      latencyThresholdMs: 1000,
      probabilisticPercent: 10,
    },
  },
});

Tune for your traffic#

If you want to keep more of your slow traces, lower latencyThresholdMs. For example, set it to 500 to keep every trace slower than half a second. To keep only the genuinely slow ones, raise it.

If you want a denser sample of ordinary traffic, raise probabilisticPercent. To cut storage and trace volume, lower it; errors and slow traces are still kept in full.

Set probabilisticPercent to 0 to keep only errors and slow traces. At 0 the probabilistic policy is omitted entirely, so every fast, non-error trace is dropped at the gateway and never reaches Tempo. This is the most aggressive volume reduction and pairs well with a latency threshold set to where “good enough” ends for your service (for example 300).

Important

The gateway runs as a single replica so that one instance sees every span of a trace, which tail sampling requires. Changing the sampling thresholds does not change this; do not scale the gateway up to increase throughput, because that would split a trace’s spans across replicas and corrupt the sampling decision.

Verify#

The thresholds are baked into the gateway’s Alloy configuration at synth time. After deployment, confirm the gateway restarted with the new values.

kubectl rollout status deployment/alloy-traces -n monitoring

Send traffic through an instrumented application, then confirm in Grafana that error and slow traces appear while ordinary traces are present only as a sample.

See also#