datadoglogo

Mobile alerting with tracking & escalation for Datadog

S4_screenshot_datadog

 Why SIGNL4

Datadog’s monitoring tools have many metrics and graphs.  The limitation is the delivery methods on which that critical information is delivered. Datadog can send an email but that seems to be where the delivery methods stop. SIGNL4 can receive those emails (or HTTP requests) sent from Datadog and reliably notifies active staff through persistent mobile push, text and voice calls with acknowledgement, tracking and escalation. For incidents updates are supported, too. Alerts that are acknowledged or closed in the SIGNL4 app will result in a status update in Datadog.

 How it Works

SIGNL4 notifies teams on their mobile devices when custom alert conditions are met in Datadog. The integration is done using the SIGNL4 webhook and you are instantly connected to Datadog’s monitors via your smartphone. You can also add custom options in the configurations section. For incident updates the Datadog Connector App in SIGNL4 needs to be configured.

 Integration Capabilities

  • IT staff is alerted via mobile push, text and voice
  • Staff can acknowledge and take ownership for critical events that occur
  • Alerts are escalated in case of no response
  • Communicate within an alert to address a particular problem
  • Inbound webhook and email establishes connection from a Datadog dashboard to a SIGNL4 team
  • Alerts that are closed in Datadog are automatically closed in the SIGNL4 app as well
  • On-call scheduling to alert the right people at the right time
  • Alerts acknowledged or closed in the SIGNL4 app will result in a status update in Datadog
  • When an incident or alert is resolved in Datadog, the corresponding SIGNL4 alarm is also closed

Scenarios

  • APM Monitoring
  • DevOps
FS_neu_2

Alarms and incidents in Datadog are sent to SIGNL4

Event categorization, routing and automated delivery

Persistent Notifications by push, text and voice call with Tracking, Escalation and Confirmation to Staff on Duty

HOW TO INTEGRATE
You can find a detailed description on how the webhook integration works here. This one supports sending alerts / incidents as well as alert resolutions in case the issue has been resolved on Datadog side.

For incidents status updates in Datadog are supported directly from your SIGNL4 app.

datadog-alerts-webhook

Datadog Alerts

You can alert your SIGNL4 team about new Alerts in Datadog. Alerts that get resolved in Datadog will automatically close the alert in SIGNL4. In order to do so you need to configure the following.

1. In Datadog, go to Integrations and create a new Webhook integration.

2.Navigate to the Webhooks Integration tile.

3. On the “Configuration” tab, go to “Webhooks” and click “New”.

4. Under “New Webhook”, enter a meaningful “Name” and use this SIGNL4 Webhook “URL” including your team secret, for example:

https://connect.signl4.com/webhook/[team-secret]?ExtIDParam=alertId&ExtStatusParam=alertTransition&ResolvedStatus=Recovered

Replace [team-secret] with your SIGNL4 team secret here.

5. Copy-paste the following JSON in the “Payload” text box:


{
	"title": "$EVENT_TITLE",
	"message": "$TEXT_ONLY_MSG",
	"link": "$LINK",
	"priority": "$ALERT_PRIORITY",
	"host": "$HOSTNAME",
	"alertScope": "$ALERT_SCOPE",
	"alertStatus": "$ALERT_STATUS",
	"alertId": "$ALERT_ID",
	"alertTransition": "$ALERT_TRANSITION",
	"X-S4-SourceSystem": "Datadog",
	"date": "$DATE",
	"org": {
		"id": "$ORG_ID",
		"name": "$ORG_NAME"
	},
	"id": "$ID"
}

You can adapt the parameters according to your need but please leave alertId, alertTransition and X-S4-SourceSystem unchanged.

6. Click on “Save” to complete the webhook creation.

That’s it and now you can use your webhook as a notification channel in your Monitor. You can choose @webhook-SIGNL4 here if SIGNL4 is the name of your webhook. When the conditions of the monitor apply your team will receive a new SIGNL4 alert.

datadog-incidents-webhook

Datadog Incidents

You can alert your SIGNL4 team about new Incidents in Datadog. Incidents that get resolved in Datadog will automatically close the alert in SIGNL4. In order to do so you need to configure the following.

1. In Datadog, go to Integrations and create a new Webhook integration.

2.Navigate to the Webhooks Integration tile.

3. On the “Configuration” tab, go to “Webhooks” and click “New”.

4. Under “New Webhook”, enter a meaningful “Name” and use this SIGNL4 Webhook “URL” including your team secret, for example:

https://connect.signl4.com/webhook/[team-secret]?ExtIDParam=s4ExternalId&ExtStatusParam=incidentStatus&ResolvedStatus=resolved

Replace [team-secret] with your SIGNL4 team secret here.

5. Copy-paste the following JSON in the “Payload” text box:


{
	"title": "$EVENT_TITLE",
	"message": "$TEXT_ONLY_MSG",
	"link": "$LINK",
	"priority": "$ALERT_PRIORITY",
	"host": "$HOSTNAME",
	"alertScope": "$ALERT_SCOPE",
	"alertStatus": "$ALERT_STATUS",
	"alertId": "$ALERT_ID",
	"incidentPublicId": "$INCIDENT_PUBLIC_ID",
	"incidentStatus": "$INCIDENT_STATUS",
	"alertTransition": "$ALERT_TRANSITION",
	"s4ExternalId": "DATADOG-INCIDENT-$INCIDENT_PUBLIC_ID",
	"X-S4-SourceSystem": "Datadog",
	"date": "$DATE",
	"org": {
		"id": "$ORG_ID",
		"name": "$ORG_NAME"
	},
	"id": "$ID"
}

You can adapt the parameters according to your need but please leave incidentStatus, s4ExternalId and X-S4-SourceSystem unchanged.

6. Click on “Save” to complete the webhook creation.

datadog-incidents-rule

Incidents Rule

Now you can create a rule in Datadog under Monitors -> Settings (Incidents) Rules. In the you specify the criteria, i.e. the severity, service, etc. For the “Other attributes” we recommend to use “state:active” that will trigger a new alert and “state:resolved” that will close the alert. Under “Notify” you select your SIGNL4 incident webhook as created above and that’s it.

If you create a new incident now, your SIGNL4 team will receive an alert. If you acknowledge / close the alert in the SIGNL4 app, the incident status will be set to Stable / Resolved.

Also, if you set the incident status to Resolved in Datadog, this will close the alert in SIGNL4.

 

signl4-datadog-connector-app

Status Updates for Datadog Incidents

For Datadog incidents you can update the status to Stable / Resolved directly from your SIGNL4 app by acknowledging / closing the respective alert.

In order to configure this back channel, go to your SIGNL4 web portal and then to Teams -> Apps. Search for the Datadog connector app and click “Create” there. You can find more information directly within the app settings.

You need to configure the following.

Datadog URL: The URL of your Datadog instance, e.g. https://app.datadoghq.com/ or https://app.datadoghq.eu/.

Datadog API Key: Your Datadog API key. You can find or create this one in Datadog under your account, Organization Settings -> API Keys.

Datadog Application Key: Your Datadog application key. You can find or create this one in Datadog under your account, Organization Settings -> Application Keys.

Acknowledgement as Stable: Acknowledgements set the incident status to Stable.

datadog_message

Alternative Method (Email)

Email: Inside each monitor there is a section to include a message for an alert.  Here is where you add the SIGNL4 email address.

ALERT OPTIMIZATION
SIGNL4 can further increase the visibility of alerts through its Signals and Services section. Augmenting the color and icon of alerts will provide more relevant information at a glance without having to open the alert.

alertoptimization

Change alert color and override title + text 

Override text, colors and titles are triggered by keywords set within the mobile app.

Related Integrations

Camunda BPM

Business / IoT

Datacake

IoT

ELEMENT IoT

Industry 4.0 / IoT

Flic

Business / IoT

Google IoT Core

Industry 4.0 / IoT

Icinga2

IoT / IT Ops

ifm IO-Link

Industry 4.0 / IoT / production

InfluxDB

IoT / IT Ops

Kaa IoT

Industry 4.0 / IoT

Elasticsearch / Kibana

IoT / IT Ops

MariaDB

Industry 4.0 / IoT / IT Ops

Melrose Labs SMS

Industry 4.0 / IoT

MySQL

Industry 4.0 / IoT / IT Ops

Particle

Industry 4.0 / IoT

PLC / S7

Industry 4.0 / IoT

PRTG Network Monitor

IoT / IT Ops

PTC ThingWorx

IoT / IT Ops

Pycom

IoT

Raspberry Pi

IoT

ServiceNow

IoT / IT Ops

Widgelix

IoT

Workato

IoT / IT Ops

Ready for a free 30-days trial?