New: temperature, fans, power supplies and SSD health per server. View hardware health

Try it free
Incidents in the portal: header “1 open incident”, below the list with three acknowledged, resolved incidents and the ongoing incident of the inference API with the “I’m on it” button.

An alert only helps if someone reads it. PeakNode sends every notification to where your team is already looking: email, Teams, Slack, Telegram, Discord or your ticket system. An outage stays open until someone takes it over with “I'm on it”. Until then, the portal sends reminders and, after the configured time, brings in additional recipients.

Screenshot from the portal with sample data (German interface).

What is Alerts and escalation?

PeakNode alerting sends outages, threshold breaches and hardware events to the channels you set per monitor: email, Microsoft Teams, Slack, Telegram, Discord or webhook, that is, a call to your ticket system. An outage stays open until someone acknowledges it with “I'm on it”. Until then, PeakNode sends a reminder every 15, 30 or 60 minutes and, after the configured time, notifies additional channels once (escalation). The all-clear goes out through the monitor’s channels.

Dashboard of the PeakNode portal: on the left the sidebar with all modules, at the top the state of the fleet (2 critical items, 61% GPU utilization, 12.5 kW power) with the history over 24 hours, below four key figures, the availability of the services, the NodeGuard panel with one critical item and the “Needs attention” list.
Screenshot from the portal with sample data (German interface).

Alerts in the portal

  • Six channels, several at once per monitor. Your plan sets how many channels a resource may have.
  • You can acknowledge in the portal, with the button in the alert email or, for incidents of uptime monitors and heartbeats, via the API.
  • You switch reminders and escalation on per monitor. By default, both are off.

From alert to all-clear

Acknowledgement, reminders and escalation apply to outages. All other notifications go to the channels once.

Every monitor has its own recipients

Every uptime monitor, heartbeat, server monitor, DNS monitor and every domain has its own selection of channels. The training cluster can alert the ML team, the inference endpoint the on-call engineer and, in addition, the customer’s IT.

“I'm on it”

An outage shows “Open: nobody has acknowledged yet” in the portal. Whoever takes over clicks “I'm on it”, in the portal or with the button in the alert email, which works without signing in. The portal then shows when, how and by whom it was acknowledged. With the button from the email, the name of the channel the alert went to is shown instead of the person.

Reminders and escalation when nobody responds

As long as nobody acknowledges, PeakNode can send a reminder every 15, 30 or 60 minutes to the same channels, at most twelve times per outage. In addition, after 15, 30 or 60 minutes exactly one notification goes to further channels, for example to a second person. PeakNode does not support multi-level escalation chains or on-call schedules.

All-clear the same way

When the check succeeds again, the heartbeat arrives or the server reports data again, PeakNode sends the all-clear to the resource’s channels. For uptime monitors and heartbeats it states the start, end and duration of the incident.

Delivery with retries

If a channel is unreachable or responds with a server error, PeakNode retries delivery at increasing intervals over about 4 hours. A late chat message carries the note “Delivered late” with the time of the event. The webhook is an exception: if the connection is established and only the response is missing, there is no retry, so that your system does not trigger anything twice.

Every outage stays traceable

Every outage of an uptime monitor or heartbeat becomes an incident with start, end, duration and reason. The events in between are listed with it: first failure detected, failure confirmed by the second check, notified via which channel, resolved. A comment records the root cause.

One outage, five notifications

The course of one night

TimeEventSent toWording
03:14AlertMicrosoft Teams, Slack, emailYour “gpu-train-01” server monitor stopped reporting data (offline).
03:29Escalationadditionally the second person’s TelegramEscalation: Server monitor “gpu-train-01” has been offline since 03:14 and nobody acknowledged the alert within 15 minutes.
03:44ReminderMicrosoft Teams, Slack, emailServer monitor “gpu-train-01”: still offline since 03:14 (reminder 1 of 12). Nobody has acknowledged yet.
03:52Acknowledgementshown in the portalAcknowledged. Reminders and escalation have been stopped.
04:05All-clearMicrosoft Teams, Slack, emailYour “gpu-train-01” server monitor is reporting data again (online).

Example with the wording as in the portal, times shortened. Configured here: escalation after 15 minutes, reminder every 30 minutes.

Six channels and what arrives there

Every channel can be checked with “Test integration” before it is attached to a monitor, email as soon as the address is confirmed.

ChannelWhat arrivesSetup
EmailSubject, for example: Alert: “api-prod” is offline. In the body the target, start and reason, the link to the incident and the “I'm on it” buttonEnter the address and verify it with the link in the confirmation email
Microsoft TeamsText message in the channel, for an uptime monitor: notification, target, reason, start and link to the incidentWebhook URL of the Teams channel
SlackText message in the channel with the same detailsSlack webhook URL
DiscordText message in the channel with the same detailsDiscord webhook URL
TelegramMessage from your own bot with the same detailsAPI token of the bot and chat ID
WebhookPOST request to your URL with the fields of the event, times in UTC. Fields and format under API and webhooksURL of your ticket system or your own service

The URLs of the webhook, Microsoft Teams, Slack and Discord channels must be publicly reachable and use http or https. Only the email carries a button for acknowledging. From Teams, Slack, Telegram and Discord, the link to the incident leads to the portal, where you acknowledge. During an active maintenance window, reminders and escalation are paused.

Set up in three steps

You create channels once and use them on as many monitors as you like.

  1. Create channels

    Under “Notification handlers”, create one channel per destination: the on-call address, the webhook of the Teams channel, the URL of the ticket system. Check them with “Test integration”.

  2. Select channels on the monitor

    When creating or editing a monitor, tick the channels in the notifications section. Several at once are possible, up to the number your plan allows per resource.

  3. Set reminders and escalation

    Below that, set “Reminder while the outage lasts” and “Escalation”, and select the additional channels for the escalation. Both can be set separately per uptime monitor, heartbeat and server monitor.

Frequently asked
questions about Alerts and escalation

Continue to:

Which channels does PeakNode use for alerts?

PeakNode sends alerts via email, Microsoft Teams, Slack, Telegram, Discord and webhook. For each monitor you choose which channels receive the alert, several at once if your plan allows it. With the webhook you reach ticket systems and your own services.

What does “I'm on it” mean?

With “I'm on it”, a person confirms that they are taking over an ongoing outage. The acknowledgement stops reminders and escalation for this outage, and the portal shows that someone is on it. It does not end the outage itself: the incident only closes when the service is running again.

What happens if nobody responds to an alert?

Then PeakNode sends reminders to the same channels at the configured interval and, after the configured time, notifies further channels once. You have to switch both on for the monitor; by default they are off. Reminders stop on acknowledgement or when the service is running again.

Can I acknowledge an alert from the email?

Yes. The “I'm on it” button in the alert email opens a page where you complete the acknowledgement without signing in to the portal. The link applies only to this specific outage and stops working once the outage has ended.

Which notifications have acknowledgement and escalation?

Outages: an uptime monitor is offline, a heartbeat was missed, a server monitor stopped reporting data. Threshold alerts, hardware events, DNS changes and reminders about expiring domains or certificates go to the channels as a single notification, without acknowledgement, without escalation and without an all-clear.

How do alerts get into our ticket system?

Through the webhook channel. On an alert and on the all-clear, PeakNode sends a POST request with the fields of the event to your URL. Which fields these are and how a ticket system acknowledges an incident via the API is described under API and webhooks.

What happens if Slack or the mail server is unreachable at the moment?

PeakNode retries delivery after 1, 5, 15, 30, 60 and 120 minutes. Only then is the notification discarded. The outage itself is stored in the portal regardless, and the channel shows the reason for the last failed delivery.

What is an escalation level?

An escalation level defines who is notified when an alert has not been acknowledged after a certain time. PeakNode has one such level: after 15, 30 or 60 minutes without acknowledgement, exactly one notification goes to additional channels. PeakNode does not model multi-level chains or on-call schedules.

How do I avoid alert fatigue?

With the settings PeakNode provides for this: threshold alerts only fire after 1 to 10 consecutive checks, an uptime monitor only alerts after a failed recheck, and maintenance windows suppress the expected alerts of planned work. Hardware events of one report arrive bundled in one message, and NodeCare collects plannable tasks in a summary at 08:00.

Send alerts to where someone is looking

Create an account, set up channels, switch on reminders and escalation on your first monitor.