Tune alert notifications#

This guide shows you how to adjust the alert mails once delivery works. You fix the links in the mails, change how often a firing alert repeats, and add mail headers for your relay. To set up delivery first, see Configure SMTP alerting.

Make the links in the mails open#

Every notification links to Alertmanager, and every alert links to the Prometheus expression that fired it. By default, both links point at in-cluster Service addresses that no browser can open.

Set externalUrls to the addresses under which you reach Alertmanager and Prometheus. A Thanos Query in front of Prometheus understands the same /graph links, so you can point prometheus at it.

const config = mergeConfig({
  // required cluster values ...
  externalUrls: {
    alertmanager: 'https://alertmanager.example.com',
    prometheus: 'https://metrics.example.com',
  },
});

The setting changes only the links. The pods keep serving under the path /, so you do not need to change your ingress or proxy.

Change how often a firing alert repeats#

Alertmanager mails a still-firing alert group again after alertRouting.repeatInterval, which defaults to 12h. Lower it if an alert must not go unnoticed overnight, or raise it if long-standing alerts flood the mailbox.

const config = mergeConfig({
  // required cluster values ...
  alertRouting: { repeatInterval: '4h' },
});

The interval does not delay changes. A new group goes out after 30 seconds, and a new or resolved alert in an existing group goes out after the group interval of five minutes.

Add mail headers#

Some relays rewrite links for click tracking or insert tracking pixels. A rewritten link breaks the Alertmanager and Prometheus links in the mail. Most relays accept a header that turns this off for a single message.

Add the headers to smtp.headers. The stack passes the names through as given.

const config = mergeConfig({
  // required cluster values ...
  smtp: {
    host: 'in-v3.mailjet.com',
    port: 587,
    from: 'monitoring@example.net',
    username: 'relay-api-key',
    passwordSecret: { name: 'alertmanager-smtp', key: 'password' },
    requireTls: true,
    headers: {
      'X-Mailjet-TrackClick': '0',
      'X-Mailjet-TrackOpen': '0',
    },
  },
});

The headers go on every notification, including those of the staging receiver.

Verify#

Synthesize and deploy your integration chart, then wait for the next notification or trigger a test alert. Check that both links in the mail open in your browser, and that the link targets are not rewritten to a tracking domain.

To read the configuration that Alertmanager loaded, query its status endpoint.

kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093
curl -s http://localhost:9093/api/v2/status | jq -r .config.original

Tune alert notifications#

Make the links in the mails open#

Change how often a firing alert repeats#

Add mail headers#

Verify#

See also#