Setting up an Azure DevOps Post Deployment Gate that checks Application Insights for any Server Exceptions in the last 10 minutes.

We have a release pipleline which deploys a web site to an Azure App Service and then runs a bunch of Selenium tests against the website.

However that does not test the Azure Function we also deploy. Nor does it check for any errors being reported by the server.

What we will do is make use of a Post Deployment Gate, which will go and look inside Application Insights and check if any Server Exceptions were logged in the time since the web site was released.

If any server exceptions are found then the release will not progress to the next environment.

1. Enable API Access in Application Insights

Create an API Key if you do not already have one e.g.

2. Add a ‘Post Deployment’ gate

In the release definition click ‘Post-deployment conditions’ then enable ‘Gates’.

Set the delay before evaluation = 2 minutes (this allows extra time for the application insights data to come in before the gates are evaluated, although data should be there because Selenium tests will be running before this step)

Add an Invoke REST API task

Connection type: Generic

Generic service connection: Choose the one for the application insights account or Click ‘Manage’ and add this:

New Service Connection -> Generic

Connection name: give it a name e.g. Application Insights Connection

Server URL: https://api.applicationinsights.io/v1/apps/

Username & password can remain empty

e.g.

Method: Post

Headers (replace with this):

{

     "Content-Type":"application/json",

     "x-api-key": "$(ApplicationInsightsAPIKey)"

}

Body:

For all server exceptions use:

{

    "query": "exceptions | where client_Type != 'Browser' | where timestamp > ago(10m) | count"

}

To filter exceptions to within an assembly you can use (e.g. for all Unity server exceptions):

{

    "query": "exceptions | where client_Type != 'Browser' | where assembly contains 'Unity' | where timestamp > ago(10m) | count"

}

Or to filter on trace messages

e.g. If you have an Azure Function that sends emails and writes a trace message when it’s running you can ensure emails are not failing by checking the trace messages.

{

    "query": "traces | where operation_Name == 'SendEmailFunction' | where message contains 'Success' | where client_Type != 'Browser'| where timestamp > ago(10m)| count"

}

Note: The timestamp is something to play around with, it basically needs to cover:

• The time the automated Selenium tests take to run (start to finish) - for us approx. 2 minutes
• + ‘the delay before evaluation’ (2 minutes we set above)
• + If the Gates fail they re-evaluate until the 6 minute timeout so we need to ensure the time range allows for this too.

The reason for this is once the Selenium tests stop they will not generate application insights data so by the time it re-evaluates the gate again there may be no data in app insights and the gate would incorrectly pass.

Therefore we need to be checking around the last 10 minutes worth of Application Insights data

Url suffix and parameters:

$(ApplicationInsightsApplicationID)/query

Advanced

Completion event:

ApiResponse

Success criteria (The below checks for a count of 0 records to be returned):

    eq(jsonpath('$.tables[0].rows[0]')[0][0],0)

Or this makes sure there are records being returned (e.g. > 0)

    ne(jsonpath('$.tables[0].rows[0]')[0][0],0)

Evaluation options

Time between re-evaluation of gates: 5 minutes

Minimum duration for steady results after a successful gate evaluation: 0 minutes

The timeout after which all gates fail: 6 minutes

Gates and approvals: On successful gates, ask for approvals

Example:

3. Set Release variables

Add these keys with a scope to the Environment you are releasing:

ApplicationInsightsAPIKey - This one should be a secret so hide the value

ApplicationInsightsApplicationID

4. Save and test it by creating a release

Useful Websites

The API Explorer web site was very useful in constructing the correct message

https://dev.applicationinsights.io/apiexplorer/postQuery

Application Insights ‘Analytics’ was useful in constructing the query

Zero downtime Azure App Service deployments with EF6 Code First Migrations and MVC5

This post is about how we deploy our production web sites to an Azure Web App Service and execute Entity Framework 6 code first migrations as part of a VSTS Release process.

A little bit of history:

What we were doing was running dbMigrator.Update() on the start of the website.

So in Startup.cs we had this:

var efConfiguration = new Configuration();

var dbMigrator = new System.Data.Entity.Migrations.DbMigrator(efConfiguration);

dbMigrator.Update();

This worked well, it upgraded the database, and applied any seed data. The downside was the site became unusable until all this had finished which for us was about 1 minute.

This also had a major downside when we turned on Azure auto scaling on the App Plan, when it scaled to more than 1 instance, all instances started up, and all of them called dbMigrator.Update(). Resulting in a fair number of exceptions and the site failing to start.

What we now do:

We needed to remove the database logic from Startup.cs and do it as part of the VSTS Release process instead. So we deploy the site to a staging slot, execute database migrations via Migrate.exe into the production database (on the build server), and then swap the staging slot to production.

This does mean the production web site code is running against a newer database schema until the site swaps over, but this is fine as long as the developers code to handle current and current-1 database versions.  This is how production now looks, sharing the same database.

How to implement this:

1. The build Process

As well as packaging up the website into its own Artifact we now package up as another Artifact all the files we need in order to run migrate.exe, so we have 2 extra build tasks as part of our Main branch build:

In the contents section:

line 1 : Copies our dll’s containing our entity framework migrations

line 2: Copies the DeployDatabase.ps1 PowerShell script below

line 3: Copies the migrate.exe provided by Entity Framework in the packages folder.

The DeployDatabase.ps1 PowerShell file contains:

#

# DeployDatabase.ps1

#

[CmdletBinding()]

Param(

[Parameter(Mandatory=$True,Position=1)]

[string]$webAppName,

[Parameter(Mandatory=$False,Position=2)]

[string]$slotName,

[Parameter(Mandatory=$False,Position=3)]

[string]$slotResourceGroup

)

cls

$ErrorActionPreference = "Stop" # Stop as soon as an error occurs

if($slotName -ne $null -and $slotName -ne '') {

$isSlot = $True

}

else {

$isSlot = $False

}

Write-Host "PSScriptRoot : " $PSScriptRoot

Write-host "Web app: " $webAppName

Write-Host "Using slot: " $isSlot " " $slotName

Write-Host "SlotResourceGroup: " $slotResourceGroup

$dll = "SiteDataAccess.Extended.Customer.dll"

Write-Host "Using dll: " $dll

if($isSlot) {

$GetWebSite = Get-AzureRmWebAppSlot -Name $webAppName -Slot $slotName -ResourceGroupName $slotResourceGroup

}

else {

$GetWebSite = Get-AzureRmWebApp -Name $webAppName

}

$Connection = $GetWebSite.SiteConfig.ConnectionStrings | Where {$_.name -eq "ExtendedSiteDBContext"}

$ConnectionString = $Connection.ConnectionString

Write-host "Executing Database migrations and seeding with Migrate.exe"

& "$PSScriptRoot\migrate.exe" $dll /connectionString=$ConnectionString /connectionProviderName="System.Data.SqlClient" /verbose

if ($LastExitCode -ne 0) {

throw 'migrate.exe returned a non-zero exit code...'

}

Write-host "Finished executing Database migrations and seeding with Migrate.exe"

Write-host "Finished"

What that script does when called from an Azure Powershell task in a Release definition is lookup the connectionString from the web app service in Azure and then uses that to call migrate.exe. We did this so we did not have to store any connectionStrings in VSTS. If the sql user in the connectionString used when executing migrate.exe needs different permissions to that of the website you could change this to use VSTS release variables instead.

2. The Release definition

When we release a site our process is:

· Stop the deployment slot ‘stage’

· Deploy the website zip to slot ‘stage’

· Update the database using the PowerShell script

· Start the Stage site

· Swap Stage with production

· Ping production site

· Stop the stage site (to save resources)

Our release definition looks like this:

Release definition	Task Groups

The Update Database task is just an Azure PowerShell task containing:

In Azure the App Service Plan is configured so that the production site and the stage slot are almost identical (baring a few appSetting values). They share the same connectionString and every appSetting and ConnectionString value has the ‘Slot Setting’ checkbox ticked.

Our big gotcha

Was during the swap slots task the website will be warm started automatically by that process hitting localhost under http, or it will try the domain name of the site but again under http. We had this in our filter.config

// Ensure all http connections are redirected to https

filters.Add(new RequireHttpsAttribute());

Which meant the warm start process instantly failed to start the site, the swap process continued and the site then started up from cold in production. We could see that because our ping task that pinged the production site, was taking a minute to respond.

What we had to do was create a custom version of the RequireHttpsattribute, once we did this our ping task responds in 1-3 seconds:

// Ensure all http connections are redirected to https

filters.Add(new CustomRequireHttpsAttribute());

And the code for the attribute is:

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, Inherited = true, AllowMultiple = false)]

public class CustomRequireHttpsAttribute : RequireHttpsAttribute

{

    protected override void HandleNonHttpsRequest(AuthorizationContext filterContext)

{

    string ipAddress = filterContext.RequestContext.HttpContext.Request.UserHostAddress;

    var userAgent = filterContext.RequestContext.HttpContext.Request.UserAgent.ToLower();

    // 0.0.0.0 and 127.0.0.1 and ::1 are used by the Azure App Service Swap Slot process to Warm up the site before swapping the slot to production

    if(ipAddress == "0.0.0.0" || ipAddress == "::1" || ipAddress == "127.0.0.1")

    //if(userAgent.Contains("sitewarmup")) // doesnt work

    {

        return;

    }

    base.HandleNonHttpsRequest(filterContext);

    }

}