On-Call Alerting and Management for Datadog

Respond faster and from anywhere to critical incidents

5stars

Mobile Alerts

Adds real-time alert notifications by push, text and voice calls to your Datadog

Scheduling

Integrated on-call duty and shift scheduling allows for automated routing of ticket alerts to the right people at the right time

Mobile App

User-friendly mobile app for Android and iPhone provides incident and ticket management on-the-go

2-way Integration

Integrates with a few mouse-clicks into Datadog. 2-way integration provides for anywhere acknowledgement and closure.

datadoglogo
S4_screenshot_datadog

 Why SIGNL4

Datadog’s monitoring tools have many metrics and graphs.  The limitation is the delivery methods on which that critical information is delivered. Datadog can send an email but that seems to be where the delivery methods stop. SIGNL4 can receive those emails (or HTTP requests) sent from Datadog and reliably notifies active staff through persistent mobile push, text and voice calls with acknowledgement, tracking and escalation. For incidents updates are supported, too. Alerts that are acknowledged or closed in the SIGNL4 app will result in a status update in Datadog.

 How it Works

SIGNL4 notifies teams on their mobile devices when custom alert conditions are met in Datadog. The integration is done using the SIGNL4 webhook and you are instantly connected to Datadog’s monitors via your smartphone. You can also add custom options in the configurations section. For incident updates the Datadog Connector App in SIGNL4 needs to be configured.

 Integration Capabilities

  • IT staff is alerted via mobile push, text and voice
  • Staff can acknowledge and take ownership for critical events that occur
  • Alerts are escalated in case of no response
  • Communicate within an alert to address a particular problem
  • Inbound webhook and email establishes connection from a Datadog dashboard to a SIGNL4 team
  • Alerts that are closed in Datadog are automatically closed in the SIGNL4 app as well
  • On-call scheduling to alert the right people at the right time
  • Alerts acknowledged or closed in the SIGNL4 app will result in a status update in Datadog
  • When an incident or alert is resolved in Datadog, the corresponding SIGNL4 alarm is also closed

Scenarios

  • APM Monitoring
  • DevOps
Mobile Alerting and Anywhere Incident Management

Alarms and incidents in Datadog are sent to SIGNL4

Event categorization, routing and automated delivery

Persistent Notifications by push, text and voice call with Tracking, Escalation and Confirmation to Staff on Duty

Integration with Datadog

datadog-alerts-webhook

HOW TO INTEGRATE
You can find a detailed description on how the webhook integration works here. This one supports sending alerts / incidents as well as alert resolutions in case the issue has been resolved on Datadog side.

For incidents status updates in Datadog are supported directly from your SIGNL4 app.

Datadog Alerts

You can alert your SIGNL4 team about new Alerts in Datadog. Alerts that get resolved in Datadog will automatically close the alert in SIGNL4. In order to do so you need to configure the following.

1. In Datadog, go to Integrations and create a new Webhook integration.

2.Navigate to the Webhooks Integration tile.

3. On the “Configuration” tab, go to “Webhooks” and click “New”.

4. Under “New Webhook”, enter a meaningful “Name” and use this SIGNL4 Webhook “URL” including your team secret, for example:

https://connect.signl4.com/webhook/[team-secret]?ExtIDParam=alertId&ExtStatusParam=alertTransition&ResolvedStatus=Recovered

Replace [team-secret] with your SIGNL4 team secret here.

5. Copy-paste the following JSON in the “Payload” text box:


{
	"title": "$EVENT_TITLE",
	"message": "$TEXT_ONLY_MSG",
	"link": "$LINK",
	"priority": "$ALERT_PRIORITY",
	"host": "$HOSTNAME",
	"alertScope": "$ALERT_SCOPE",
	"alertStatus": "$ALERT_STATUS",
	"alertId": "$ALERT_ID",
	"alertTransition": "$ALERT_TRANSITION",
	"X-S4-SourceSystem": "Datadog",
	"date": "$DATE",
	"org": {
		"id": "$ORG_ID",
		"name": "$ORG_NAME"
	},
	"id": "$ID"
}

You can adapt the parameters according to your need but please leave alertId, alertTransition and X-S4-SourceSystem unchanged.

6. Click on “Save” to complete the webhook creation.

That’s it and now you can use your webhook as a notification channel in your Monitor. You can choose @webhook-SIGNL4 here if SIGNL4 is the name of your webhook. When the conditions of the monitor apply your team will receive a new SIGNL4 alert.

datadog-incidents-webhook

Datadog Incidents

You can alert your SIGNL4 team about new Incidents in Datadog. Incidents that get resolved in Datadog will automatically close the alert in SIGNL4. In order to do so you need to configure the following.

1. In Datadog, go to Integrations and create a new Webhook integration.

2.Navigate to the Webhooks Integration tile.

3. On the “Configuration” tab, go to “Webhooks” and click “New”.

4. Under “New Webhook”, enter a meaningful “Name” and use this SIGNL4 Webhook “URL” including your team secret, for example:

https://connect.signl4.com/webhook/[team-secret]?ExtIDParam=s4ExternalId&ExtStatusParam=incidentStatus&ResolvedStatus=resolved

Replace [team-secret] with your SIGNL4 team secret here.

5. Copy-paste the following JSON in the “Payload” text box:


{
	"title": "$EVENT_TITLE",
	"message": "$TEXT_ONLY_MSG",
	"link": "$LINK",
	"priority": "$ALERT_PRIORITY",
	"host": "$HOSTNAME",
	"alertScope": "$ALERT_SCOPE",
	"alertStatus": "$ALERT_STATUS",
	"alertId": "$ALERT_ID",
	"incidentPublicId": "$INCIDENT_PUBLIC_ID",
	"incidentStatus": "$INCIDENT_STATUS",
	"alertTransition": "$ALERT_TRANSITION",
	"s4ExternalId": "DATADOG-INCIDENT-$INCIDENT_PUBLIC_ID",
	"X-S4-SourceSystem": "Datadog",
	"date": "$DATE",
	"org": {
		"id": "$ORG_ID",
		"name": "$ORG_NAME"
	},
	"id": "$ID"
}

You can adapt the parameters according to your need but please leave incidentStatus, s4ExternalId and X-S4-SourceSystem unchanged.

6. Click on “Save” to complete the webhook creation.

datadog-incidents-rule

Incidents Rule

Now you can create a rule in Datadog under Monitors -> Settings (Incidents) Rules. In the you specify the criteria, i.e. the severity, service, etc. For the “Other attributes” we recommend to use “state:active” that will trigger a new alert and “state:resolved” that will close the alert. Under “Notify” you select your SIGNL4 incident webhook as created above and that’s it.

If you create a new incident now, your SIGNL4 team will receive an alert. If you acknowledge / close the alert in the SIGNL4 app, the incident status will be set to Stable / Resolved.

Also, if you set the incident status to Resolved in Datadog, this will close the alert in SIGNL4.

 

signl4-datadog-connector-app

Status Updates for Datadog Incidents

For Datadog incidents you can update the status to Stable / Resolved directly from your SIGNL4 app by acknowledging / closing the respective alert.

In order to configure this back channel, go to your SIGNL4 web portal and then to Teams -> Apps. Search for the Datadog connector app and click “Create” there. You can find more information directly within the app settings.

You need to configure the following.

Datadog URL: The URL of your Datadog instance, e.g. https://app.datadoghq.com/ or https://app.datadoghq.eu/.

Datadog API Key: Your Datadog API key. You can find or create this one in Datadog under your account, Organization Settings -> API Keys.

Datadog Application Key: Your Datadog application key. You can find or create this one in Datadog under your account, Organization Settings -> Application Keys.

Acknowledgement as Stable: Acknowledgements set the incident status to Stable.

datadog_message

Alternative Method (Email)

Email: Inside each monitor there is a section to include a message for an alert.  Here is where you add the SIGNL4 email address.

Integration with Datadog

Tips & Tricks

alertoptimization_frames

ALERT OPTIMIZATION
SIGNL4 can further increase the visibility of alerts through its Signals and Services section. Augmenting the color and icon of alerts will provide more relevant information at a glance without having to open the alert.

Change alert color and override title + text 

Override text, colors and titles are triggered by keywords set within the mobile app.

Related Integrations

AppGyver

IoT

Arduino

IoT

AT&T IoT Platform

IoT

AWS IoT Button

IoT

Axonize (now Platon)

IoT

Building Management Systems

IoT

Camunda BPM

IoT

CloudFit

IoT

Crosser

Industry 4.0 / IoT

Cumulocity from Software AG

Industry 4.0 / IoT

Datacake

IoT

elastic.io

IoT / IT Ops

ELEMENT IoT

IoT

Flic

IoT

Gimasi Service Button

IoT

Hilscher

Industry 4.0 / IoT

ifm IO-Link

Industry 4.0 / IoT

IXON Cloud

IoT

Kaa IoT

IoT

Losant

IoT

Melrose Labs SMS

Industry 4.0 / IoT

Azure IoT

IoT

n8n

IoT

Node-RED

IoT

Ready for a free 30-days trial?