Azure Monitor Alerts

An alert rule runs a query on the workspace every few minutes. When the query finds the condition, an alert is fired; when the condition is gone, the alert is resolved. A rule can notify an action group. This page describes the rules of a replication, and the two rules that the template of Install the Monitoring Function creates.

How the rules look at the rows

Every rule looks at the newest rows that have reached the workspace, not at a time window: it fires when the newest 3 rows (the parameter pollsInAlarm) of the process are all bad. The rows reach the workspace a few seconds to several minutes after the function wrote them, so a rule that counted the rows of the last five minutes would see only some of them. A rule runs every 5 minutes. So an alert fires up to about ten minutes after a problem began, and it is resolved about fifteen to twenty minutes after the problem is gone (an alert is resolved after three runs of its rule in a row that do not find the condition).

The alerts of a replication

One deployment creates the rules for one Capture Process and one Apply Process of a server. Deploy it for every replication that you want to watch. The template is in the file azuredeploy-alerts.json (see Install the Monitoring Function for the download). It needs the workspace and the identity that the first deployment created; their resource ids are its outputs:

It creates up to five rules: the name of a rule is the namePrefix, the kind and the id of the process, and what it watches.

A process that has failed is also not running, so both alerts fire. The failed alert tells the reason. A stopped process has no lag value, so it raises not-running and not lag. When the server does not answer, the processes have no new rows and these alerts do not fire: the rule api-down of the first deployment tells it.

The list shows Fired alerts. The affected resource is the workspace, because the rules query it; the name of the rule tells which process it is about. Choose an alert for its details: the rule, the severity, the time it fired, and a link to the query and its result.

The rules of the first deployment

Be notified

A rule that has no action group only changes the state of its alert. Give the first deployment an e-mail address (alertEmail) and it creates an action group with that address for its two rules. For more than that, create an action group of your own: it can send e-mail, text messages and voice calls, push messages to the Azure mobile app, call a webhook, an Azure Function or a Logic App (to write to Microsoft Teams or Slack), or open a ticket. Give its resource id to the deployments as actionGroupResourceId:

The action group notifies when an alert fires and when it is resolved (tested with a webhook).

What the messages look like, and what is normal

When an alert fires, every receiver of the action group gets a message. An e-mail receiver gets an e-mail with the rule, the severity, the time and a link to the alert. A webhook gets the alert as JSON in the common alert schema. The JSON below is a real one, shortened and with the names replaced: it is the alert apply-1-not-running of a replication whose Apply Process had failed, and it reached a webhook 3 seconds after the alert fired.

The fields that matter:

• alertRule is the name of the rule, severity is Sev1 for api-down, Sev2 for not-running, failed and function-errors, and Sev3 for lag, monitorCondition is Fired or Resolved, and description is the sentence of the rule.
• firedDateTime is the time that the alert fired (UTC); a message that says Resolved also has resolvedDateTime.
• searchQuery is the query that found the condition, and metricValue 1.0 is what it returned. windowStartTime and windowEndTime are the 15 minutes of rows that it looked at.
• investigationLink opens the alert in the portal, and linkToSearchResultsUI opens the query with its result.

How long it takes. The times are from a test with a replication of two tables between two Azure SQL databases, rules that run every 5 minutes, and pollsInAlarm 3 (the lag alert was tested with maxLagSeconds 30). The time is counted from the moment that I did the first thing in the table to the moment that the alert fired:

When the cause was gone, the alerts were resolved after 13 to 18 minutes, and the alert function-errors after 28 minutes, because its rule looks back 15 minutes: an alert is resolved after three runs of its rule in a row that do not find the condition. The action group got the message Resolved 2 to 3 seconds after each of the three alerts that I watched.

These things look strange the first time. They are not faults:

• No first messages. A rule starts without a state: an alert exists only when a condition has been found, so a new deployment sends nothing until there is a problem.
• An alert at the start. If the token, the address, the fingerprint or the network of the first deployment are wrong, the function cannot read the server and api-down fires after the third poll. That is a true alert, not a false one: the column Error of RepstanceServer_CL says why (see Install the Monitoring Function). It is resolved about 15 minutes after the settings are right.
• Two alerts for one failure. A process that has failed is not running, so failed and not-running both fire.
• A server that does not answer. api-down fires. The rules of the processes look at the newest rows of the processes, which are the rows from before the server stopped, so they stay as they were.
• Give the rule its action group before the problem. In a test I gave the rules an action group while one of their alerts was fired. That alert sent its Resolved message, but another one, resolved four minutes after the change, did not. Deploy the action group with the rules, or while no alert is fired.

The parameters

When you stop a replication on purpose

A stopped process is not running, so its alert fires. Delete the deployment of the replication (delete its rules), or switch its rules off while the replication is stopped:

and switch them on again with --disabled false.