Mobile alerting with tracking & escalation for Prometheus Alertmanager
Why SIGNL4
When critical systems fail, SIGNL4 is the fastest way to alert your staff, engineers, IT admins on call and “in the field”. SIGNL4 provides reliable notifications via mobile app push, text and voice calls with tracking, escalations and duty scheduling.
Prometheus is an open-source monitoring tool that monitors availability and performance. It gives you simple access to relevant data for example using Grafana. SIGNL4 easily enables Prometheus to notify mobile teams in the field or on call in real-time. This speeds up their response significantly and frees resources in the operations.
Pairing Prometheus Alertmanager with SIGNL4 can enhance your daily operations with an extension to your team wherever it is. The integration does not only allow you to know when a critical issue has occurred but also when it was resolved no matter where you are.
How it Works
Prometheus compatibility allows you to distribute operational alerts to a mobile SIGNL4 team by using a ready-made plugin. Persistent notifications and acknowledgement requirements ensure that issues will be handled before it is too late.
Integration Capabilities
- IT staff is alerted via mobile push, SMS text and voice call
- Staff can acknowledge and take ownership for critical events that occur
- Alerts are escalated in case of no response
- Communicate within an alert to address a particular problem
- Inbound webhook or email establishes connection from Prometheus Alertmanager to SIGNL4
- On-call scheduling to alert the right people at the right time
Scenarios
- System Monitoring
- Application Monitoring
- Network Monitoring
- Database Monitoring
Tickets and incidents in Prometheus Alertmanager are sent to SIGNL4
Event categorization, routing and automated delivery
Persistent Notifications by push, text and voice call with Tracking, Escalation and Confirmation to Staff on Duty
HOW TO INTEGRATE
The integration of Prometheus and SIGNL4 is done using the Alertmanager that is part of Prometheus.
Prerequisites
A SIGNL4 account (https://www.signl4.com)
A Prometheus installation (https://prometheus.io)
An Alertmanager installation (https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
Alertmanager Webhook
The integration is done using the standard webhook in Alertmanager. You can find more information here:
https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
A sample configuration in the configuration file might look like this:
receivers:
- name: prometheus-signl4
webhook_configs:
- url: https://connect.signl4.com/webhook/
send_resolved: true
route:
group_by:
- job
group_interval: 5m
group_wait: 30s
receiver: prometheus-signl4
repeat_interval: 10m
routes:
- match:
alertname: Watchdog
receiver: prometheus-signl4
You need to replace with your SIGNL4 team secret.
Resolved Alerts
In order to close alerts in SIGNL4 when the alerts are closed in Prometheus you can use this SIGNL4 URL.
https://connect.signl4.com/webhook/{teamSecret}?ExtIDParam=groupKey&ExtStatusParam=status&ResolvedStatus=resolved
You can find more information about the resolve feature here:
https://www.signl4.com/blog/update-july-2020-resolve-alerts/
Test it
That is it and now you can test the alert. You can for example simulate an alert in Prometheus and you should then receive an alert in your SIGNL4 app.
ALERT OPTIMIZATION
SIGNL4 can further increase the visibility of alerts through its Signals and Services section. Augmenting the color and icon of alerts will provide more relevant information at a glance without having to open the alert.
Change alert color and override title + text
Override text, colors and titles are triggered by keywords set within the mobile app.