Alerts and routing#

This page lists the alert rules the library ships, the stock rules it changes, and the Alertmanager configuration it generates. Your own application alerts are not part of the library; see Add application dashboards and alerts.

Stock kube-prometheus-stack rules#

The stack keeps the default rule set of the kube-prometheus-stack chart, with these exceptions.

Change

Effect

defaultRules.disabled.KubeJobFailed: true

The stock KubeJobFailed rule is off. The job-health group below replaces it.

defaultRules.disabled.KubeContainerWaiting: true

The stock KubeContainerWaiting rule is off. The job-health group below replaces it.

defaultRules.disabled.NodeMemoryMajorPagesFaults: true

The stock NodeMemoryMajorPagesFaults rule is off. The node-memory group below replaces it.

kubeProxy.enabled: false

No kube-proxy scrape target and no stock KubeProxyDown alert.

kubeEtcd.enabled: false

No etcd scrape target and no stock etcd alerts.

kubeControllerManager.enabled: false

No kube-controller-manager scrape target and no stock KubeControllerManagerDown alert.

kubeScheduler.enabled: false

No kube-scheduler scrape target and no stock KubeSchedulerDown alert.

Shipped rule groups#

The kube-prometheus-stack HelmChart carries three additional rule groups through additionalPrometheusRulesMap. They are always present.

job-health#

Alert

Severity

For

Fires when

KubeCronJobLastRunFailed

warning

15m

The last scheduled run of a CronJob is newer than its last successful run, and the CronJob has no active Job.

KubeJobFailed

warning

15m

A Job that no CronJob owns has the condition Failed.

KubeContainerWaiting

warning

1h

A container waits for a reason other than CrashLoopBackOff, and its pod is in phase Pending or Running.

KubeCronJobLastRunFailed resolves with the next successful run. KubeJobFailed resolves when the Job is deleted or re-run successfully. KubeContainerWaiting ignores pods in phase Failed or Succeeded.

node-memory#

Alert

Severity

For

Fires when

NodeMemoryMajorPagesFaults

warning

15m

A node has more than 500 major page faults per second over 5 minutes, and its tasks spent more than 1 % of the last 15 minutes waiting for memory (node_pressure_memory_waiting_seconds_total).

The memory pressure condition needs the pressure collector of the node exporter, which is on by default on Linux 4.20 and later.

pdb-drain-safety#

The group evaluates every minute. Both rules select PodDisruptionBudgets that allow zero voluntary disruptions and expect at least one pod.

Alert

Severity

For

Labels

PodDisruptionBudgetUndrainable

warning

15m

component: cluster-upgrade

PodDisruptionBudgetUndrainableCritical

critical

1h

component: cluster-upgrade

Both rules exclude these PodDisruptionBudgets:

  • every PodDisruptionBudget in the namespace longhorn-system

  • every PodDisruptionBudget whose name ends in -primary

Opt-in rule groups#

longhorn.volume-health#

The longhorn-volume-health PrometheusRule exists only when longhorn.enabled and longhorn.alerts are both true.

Alert

Severity

For

Expression

LonghornVolumeFaulted

critical

1m

longhorn_volume_robustness == 3

LonghornVolumeUnhealthy

warning

10m

longhorn_volume_robustness == 0 or longhorn_volume_robustness == 2

The metric longhorn_volume_robustness takes the values 0 (unknown), 1 (healthy), 2 (degraded), and 3 (faulted).

Alert labels#

Prometheus adds the external label cluster with the value of clusterName to every alert. Prometheus drops the external label replica from alerts through additionalAlertRelabelConfigs. Metrics keep the replica label for Thanos deduplication.

Routing tree#

The root route has these settings.

Setting

Value

group_by

['job']

group_wait

30s

group_interval

5m

repeat_interval

alertRouting.repeatInterval, default 12h

receiver

email

Alertmanager evaluates the child routes in this order and stops at the first match.

Order

Matcher

Receiver

1

alertname: Watchdog

null, or deadmansswitch when deadMansSwitch.enabled is true

2

alertname: CPUThrottlingHigh

null

3

alertname: KubeMemoryOvercommit

null

4

alertname: KubeCPUOvercommit

null

5

alertname: KubeletTooManyPods

null

6

alertname: InfoInhibitor

null

7

namespace matches ^(<stagingAlerts.namespaces>)$

staging

Route 7 exists only when stagingAlerts.enabled is true. With deadMansSwitch.enabled, route 1 also sets group_wait: 0s, group_interval: 1m, and repeat_interval from deadMansSwitch.repeatInterval. Every alert that matches no child route goes to email.

Receivers#

null

Discards the notification.

email

Sends to every address in smtp.recipients, or to smtp.from when smtp.recipients is unset. Sends resolved notifications as well. Adds every entry of smtp.headers as a mail header.

staging

Exists only when stagingAlerts.enabled is true. Sends to the same addresses as email, with resolved notifications. Sets the subject to [STAGING] followed by the default subject, and adds every entry of smtp.headers.

deadmansswitch

Exists only when deadMansSwitch.enabled is true. Posts the alert to the URL read from /etc/alertmanager/secrets/<deadMansSwitch.urlSecret.name>/<deadMansSwitch.urlSecret.key>. Sends no resolved notifications.

SMTP settings#

The Alertmanager global block takes its values from the smtp configuration.

Alertmanager setting

Source

smtp_from

smtp.from

smtp_smarthost

smtp.host and smtp.port

smtp_require_tls

smtp.requireTls

smtp_auth_username

smtp.username, or smtp.from when unset

smtp_auth_password_file

/etc/alertmanager/secrets/<name>/<key> from smtp.passwordSecret

smtp_auth_password

smtp.password, only when smtp.passwordSecret is unset

Inhibition rules#

Source

Target

Equal labels

severity: critical

severity: warning

alertname, namespace

alertname: PodDisruptionBudgetUndrainableCritical

alertname: PodDisruptionBudgetUndrainable

namespace, poddisruptionbudget

See also#