Do you know this situation? You are on-call and in the middle of the night you get a phone call. Loud enough to wake you up. Loud enough to wake your wife up, as well. You stand up and check your emails to see what the problem is. OK, you got it. Then you log on to the console of your monitoring tool and – green. Green? False alert? Why did you get the call then? After double-checking, still a bit sleepy, you recognize that the problem has been recovered automatically. There was an automatic restart of the system and all is back to normal again. You go back to bed and try to fall asleep again. There must be a better way.
With SIGNL4 you get the signal, not the noise
Surly this alert call was not necessary. It is great to have an auto-recovery feature but then there is no need to wake somebody up a night. In these cases it is sufficient to see what was going on, in the next morning.
You can solve this easily with SIGNL4. You just need to delay the notification for an alert and when there is a recovery within this time you will not get the push message, phone call or text message.
That means if a Problem or Down event is received from your monitoring tool, the alert will show up in SIGNL4 but you won’t get a wake-up call. You can configure to delay notifications for five minutes or so. If during this time a Resolve or Up event is received from your monitoring tool, there will be no wake-up call. If the problem persists after five minutes you will receive the call because in this case we really have a problem.
How it works
In your SIGNL4 app under Settings -> Signaling you can configure a delay for certain notification types. For example you can delay the push notification for five minutes and the voice call (just in case you do not hear the push sound) for six minutes. If, within this time, there is an event from the monitoring tool indicating that the problem is resolved, then no notification is sent and you just see the resolved alerts in your alert history.
This Resolve functionality is already implemented for the SIGNL4 add-ons in Zabbix, Checkmk, n8n or Node-RED. If you would like to implement if by yourself you can just use the parameter “X-S4-Status” and set it to “resolved” in your Resolve email or webhook request.
Bonus tip 1
If you would like to get an alert but not necessarily a wake-up call in the middle of the night, you can use the power of the SIGNL4 categories. For minor incidents you can configure a category (under Services and Systems) with a soft ring tone or with just a beep. It can even be a silent alert and you can see it whenever you check your app.
Bonus tip 2
To prevent waking up your wife or your whole family in case you get a (relevant) alert at night you might want to use a smart watch or a simple fitness tracker. You can configure the notification settings in a way for the bracelet to vibrate for SIGNL4 alert notifications. In this case your bracelet will softly wake you up before (or instead of) a loud wake-up call.
That’s it and you see there are some nice ways to handle incidents that are not to critical or that are resolved automatically. It can prevent you from getting unnecessary calls in the middle of the night.