If you are using AWS CloudWatch for monitoring your AWS assets 24/7, an effective routing of alarms to IT personnel during after-business-hours is essential for a swift handling of any problems and incidents occurring any given time of the day. This post explains how to route AWS CloudWatch alarms with SIGNL4 to improve your after hour incident response while using Slack for your daytime services.
Slack for business hours and SIGNL4 for after-hours
Operating IT assets in the age of cloud or edge computing is a question of efficient teamwork. Temwork can be observed when services are built up from the ground or when they are maintained. But it is certainly essential for incident response and resolution activities. Or have you ever seen a fire truck with only one fire fighter inside?
A key question for an effective incident response process for the team is how it deals with monitoring alarms during business hours and in after hours assuming all members reside in the same time zone and cannot organize their teamwork in a “follow-the-sun” approach.
During business hours these teams usually route their alarms to a group messaging system which they also use to trigger control commands or to query various metrics from their IT assets. This work approach was called “chatops” and is a typical characteristic of devops teams.
While this approach can be very effective for business hour activities, the question is how to organize teamwork in after hours? Required work in after hours is usually narrowed down to incident response and first-response activities. There are factors that require differentiate communication during after-business hours. People are less attentive or might even sleep. Persistent notifications, i.e. “nagging”, for instance becomes a top requirement. Tracking and transparency of alert confirmations is another.
Step1 – Get started with SIGNL4
SIGNL4 is a lightweight mobile alerting app for reliably notifying personell during after business hours. It is available on GooglePlay and in iTunes AppStore. Download the app, sign up to create your SIGNL4 account. Once logged in, invite your peers and build your operations team in SIGNL4.
Step 2 – Adjust your Amazon SNS Settings to route AWS CloudWatch Alarms to SIGNL4
Monitoring alarms created by CloudWatch are notified by Amazons Simple Notification Service (SNS). SNS is based on so called Topics and Subscriptions. Topics are targeted by monitoring alarms from CloudWatch. Subscriptions on Topics take care for sending notifications once the Topic got new content. It is most likely and assumed that you already have some Topics setup and that there is at least one Subscription that sends Topic contents to an AWS Lamda function which will post it to your Team’s Slack Channels.
The idea now is to send all your Topic contents to SIGNL4 in parallel. To do that, setup an additional Subscription in SNS which sends notifications directly to the webhook of your SIGNL4 team. The webhook is automatically provisioned for your team during sign up, thus it doesn’t need to be created manually as you may have done in Slack. Even better: unlike with your SNS/Slack integration, there is no need for an intermediate AWS Lambda function to get the data from SNS to SIGNL4.
The SIGNL4 webhook is intelligent and flexible enough to handle AWS SNS webhook calls directly. Follow the below steps to create the Subscription:
- Log on to SNS and open your Subscriptions. It should list your AWS Lambda Sunscription(s) for Slack.
- Click “Create Subscription”
- Enter the ARN of the Topic from which all contents should be sent to your team in after-hours
- Select HTTPS as protocol
- Enter the webhook of your SIGNL4 team in the endpoint field. You can find that URL e.g. in the SIGNL4 app under Settings (gear) -> Team -> APIs
- Click “Create subscription” to finish
Your new SIGNL4 Subscription is now in a pending status. AWS SNS has sent a confirmation message to your SIGNL4 webhook which you have to confirm in order to activate the subscription:
- In the SIGNL4 mobile app you’ll find a new Signl from AWS
- In the details of that Signl you’ll find a “Subscribeurl” that you simply need to open to activate the Subscription in SNS
After you have opened the Url in your mobile device browser, a refresh of your SNS subscriptions will now show an active status for your new SIGNL4 subscription:
You will now start seeing AWS CloudWatch alarms in your SIGNL4 team..
Step 3 – Fine tune after-hour teamwork with SIGNL4
SIGNL4 team members will only get push notifications when they are punched in for duty. To automate duty management with SIGNL4, log in to your SIGNL4 account and select “My Team”. Enable “Duty Assistant” and set handover time to 10 pm and duty length to 8 hours. Afterwards SIGNL4 will start taking care at 9.30 pm each day and ensure that someone from your team punches in for “night shift”. The duty assistant will also automatically release on-duty people next morning when your team continues working with Slack anyways. The beauty here is that punched in team members get persistent alert notifications in after hours, so that they wake up and can troubleshoot the issue.
Get started with SIGNL4 today
As you’ve learned above there is nothing else to be done than getting the SIGNL4 mobile app and slightly adjusting your existing SNS configuration. There is no backend policies, rules, accounts or APIs that need to be configured in addition. Very lightweight and extremely easy to setup within 15 minutes:
