We have a release pipleline which deploys a web site to an Azure App Service and then runs a bunch of Selenium tests against the website.
However that does not test the Azure Function we also deploy. Nor does it check for any errors being reported by the server.
What we will do is make use of a Post Deployment Gate, which will go and look inside Application Insights and check if any Server Exceptions were logged in the time since the web site was released.
If any server exceptions are found then the release will not progress to the next environment.
1. Enable API Access in Application Insights
Create an API Key if you do not already have one e.g.
2. Add a ‘Post Deployment’ gate
In the release definition click ‘Post-deployment conditions’ then enable ‘Gates’.
Set the delay before evaluation = 2 minutes (this allows extra time for the application insights data to come in before the gates are evaluated, although data should be there because Selenium tests will be running before this step)
Add an Invoke REST API task
Connection type: Generic
Generic service connection: Choose the one for the application insights account or Click ‘Manage’ and add this:
New Service Connection -> Generic
Connection name: give it a name e.g. Application Insights Connection
Server URL: https://api.applicationinsights.io/v1/apps/
Username & password can remain empty
e.g.
Method: Post
Headers (replace with this):
{
"Content-Type":"application/json",
"x-api-key": "$(ApplicationInsightsAPIKey)"
}
Body:
For all server exceptions use:
{
"query": "exceptions | where client_Type != 'Browser' | where timestamp > ago(10m) | count"
}
To filter exceptions to within an assembly you can use (e.g. for all Unity server exceptions):
{
"query": "exceptions | where client_Type != 'Browser' | where assembly contains 'Unity' | where timestamp > ago(10m) | count"
}
Or to filter on trace messages
e.g. If you have an Azure Function that sends emails and writes a trace message when it’s running you can ensure emails are not failing by checking the trace messages.
{
"query": "traces | where operation_Name == 'SendEmailFunction' | where message contains 'Success' | where client_Type != 'Browser'| where timestamp > ago(10m)| count"
}
Note: The timestamp is something to play around with, it basically needs to cover:
- The time the automated Selenium tests take to run (start to finish) - for us approx. 2 minutes
- + ‘the delay before evaluation’ (2 minutes we set above)
- + If the Gates fail they re-evaluate until the 6 minute timeout so we need to ensure the time range allows for this too.
The reason for this is once the Selenium tests stop they will not generate application insights data so by the time it re-evaluates the gate again there may be no data in app insights and the gate would incorrectly pass.
Therefore we need to be checking around the last 10 minutes worth of Application Insights data
Url suffix and parameters:
$(ApplicationInsightsApplicationID)/query
Advanced
Completion event:
ApiResponse
Success criteria (The below checks for a count of 0 records to be returned):
eq(jsonpath('$.tables[0].rows[0]')[0][0],0)
Or this makes sure there are records being returned (e.g. > 0)
ne(jsonpath('$.tables[0].rows[0]')[0][0],0)
Evaluation options
Time between re-evaluation of gates: 5 minutes
Minimum duration for steady results after a successful gate evaluation: 0 minutes
The timeout after which all gates fail: 6 minutes
Gates and approvals: On successful gates, ask for approvals
Example:
3. Set Release variables
Add these keys with a scope to the Environment you are releasing:
ApplicationInsightsAPIKey - This one should be a secret so hide the value
ApplicationInsightsApplicationID
4. Save and test it by creating a release
Useful Websites
The API Explorer web site was very useful in constructing the correct message
https://dev.applicationinsights.io/apiexplorer/postQuery
Application Insights ‘Analytics’ was useful in constructing the query
