Setting up an Azure DevOps Post Deployment Gate that checks Application Insights for any Server Exceptions in the last 10 minutes.

We have a release pipleline which deploys a web site to an Azure App Service and then runs a bunch of Selenium tests against the website.

However that does not test the Azure Function we also deploy. Nor does it check for any errors being reported by the server.

What we will do is make use of a Post Deployment Gate, which will go and look inside Application Insights and check if any Server Exceptions were logged in the time since the web site was released.

If any server exceptions are found then the release will not progress to the next environment.

1. Enable API Access in Application Insights

Create an API Key if you do not already have one e.g.

2. Add a ‘Post Deployment’ gate

In the release definition click ‘Post-deployment conditions’ then enable ‘Gates’.

Set the delay before evaluation = 2 minutes (this allows extra time for the application insights data to come in before the gates are evaluated, although data should be there because Selenium tests will be running before this step)

Add an Invoke REST API task

Connection type: Generic

Generic service connection: Choose the one for the application insights account or Click ‘Manage’ and add this:

New Service Connection -> Generic

Connection name: give it a name e.g. Application Insights Connection

Server URL: https://api.applicationinsights.io/v1/apps/

Username & password can remain empty

e.g.

Method: Post

Headers (replace with this):

{

     "Content-Type":"application/json",

     "x-api-key": "$(ApplicationInsightsAPIKey)"

}

Body:

For all server exceptions use:

{

    "query": "exceptions | where client_Type != 'Browser' | where timestamp > ago(10m) | count"

}

To filter exceptions to within an assembly you can use (e.g. for all Unity server exceptions):

{

    "query": "exceptions | where client_Type != 'Browser' | where assembly contains 'Unity' | where timestamp > ago(10m) | count"

}

Or to filter on trace messages

e.g. If you have an Azure Function that sends emails and writes a trace message when it’s running you can ensure emails are not failing by checking the trace messages.

{

    "query": "traces | where operation_Name == 'SendEmailFunction' | where message contains 'Success' | where client_Type != 'Browser'| where timestamp > ago(10m)| count"

}

Note: The timestamp is something to play around with, it basically needs to cover:

• The time the automated Selenium tests take to run (start to finish) - for us approx. 2 minutes
• + ‘the delay before evaluation’ (2 minutes we set above)
• + If the Gates fail they re-evaluate until the 6 minute timeout so we need to ensure the time range allows for this too.

The reason for this is once the Selenium tests stop they will not generate application insights data so by the time it re-evaluates the gate again there may be no data in app insights and the gate would incorrectly pass.

Therefore we need to be checking around the last 10 minutes worth of Application Insights data

Url suffix and parameters:

$(ApplicationInsightsApplicationID)/query

Advanced

Completion event:

ApiResponse

Success criteria (The below checks for a count of 0 records to be returned):

    eq(jsonpath('$.tables[0].rows[0]')[0][0],0)

Or this makes sure there are records being returned (e.g. > 0)

    ne(jsonpath('$.tables[0].rows[0]')[0][0],0)

Evaluation options

Time between re-evaluation of gates: 5 minutes

Minimum duration for steady results after a successful gate evaluation: 0 minutes

The timeout after which all gates fail: 6 minutes

Gates and approvals: On successful gates, ask for approvals

Example:

3. Set Release variables

Add these keys with a scope to the Environment you are releasing:

ApplicationInsightsAPIKey - This one should be a secret so hide the value

ApplicationInsightsApplicationID

4. Save and test it by creating a release

Useful Websites

The API Explorer web site was very useful in constructing the correct message

https://dev.applicationinsights.io/apiexplorer/postQuery

Application Insights ‘Analytics’ was useful in constructing the query

Zero downtime Azure App Service deployments with EF6 Code First Migrations and MVC5

This post is about how we deploy our production web sites to an Azure Web App Service and execute Entity Framework 6 code first migrations as part of a VSTS Release process.

A little bit of history:

What we were doing was running dbMigrator.Update() on the start of the website.

So in Startup.cs we had this:

var efConfiguration = new Configuration();

var dbMigrator = new System.Data.Entity.Migrations.DbMigrator(efConfiguration);

dbMigrator.Update();

This worked well, it upgraded the database, and applied any seed data. The downside was the site became unusable until all this had finished which for us was about 1 minute.

This also had a major downside when we turned on Azure auto scaling on the App Plan, when it scaled to more than 1 instance, all instances started up, and all of them called dbMigrator.Update(). Resulting in a fair number of exceptions and the site failing to start.

What we now do:

We needed to remove the database logic from Startup.cs and do it as part of the VSTS Release process instead. So we deploy the site to a staging slot, execute database migrations via Migrate.exe into the production database (on the build server), and then swap the staging slot to production.

This does mean the production web site code is running against a newer database schema until the site swaps over, but this is fine as long as the developers code to handle current and current-1 database versions.  This is how production now looks, sharing the same database.

How to implement this:

1. The build Process

As well as packaging up the website into its own Artifact we now package up as another Artifact all the files we need in order to run migrate.exe, so we have 2 extra build tasks as part of our Main branch build:

In the contents section:

line 1 : Copies our dll’s containing our entity framework migrations

line 2: Copies the DeployDatabase.ps1 PowerShell script below

line 3: Copies the migrate.exe provided by Entity Framework in the packages folder.

The DeployDatabase.ps1 PowerShell file contains:

#

# DeployDatabase.ps1

#

[CmdletBinding()]

Param(

[Parameter(Mandatory=$True,Position=1)]

[string]$webAppName,

[Parameter(Mandatory=$False,Position=2)]

[string]$slotName,

[Parameter(Mandatory=$False,Position=3)]

[string]$slotResourceGroup

)

cls

$ErrorActionPreference = "Stop" # Stop as soon as an error occurs

if($slotName -ne $null -and $slotName -ne '') {

$isSlot = $True

}

else {

$isSlot = $False

}

Write-Host "PSScriptRoot : " $PSScriptRoot

Write-host "Web app: " $webAppName

Write-Host "Using slot: " $isSlot " " $slotName

Write-Host "SlotResourceGroup: " $slotResourceGroup

$dll = "SiteDataAccess.Extended.Customer.dll"

Write-Host "Using dll: " $dll

if($isSlot) {

$GetWebSite = Get-AzureRmWebAppSlot -Name $webAppName -Slot $slotName -ResourceGroupName $slotResourceGroup

}

else {

$GetWebSite = Get-AzureRmWebApp -Name $webAppName

}

$Connection = $GetWebSite.SiteConfig.ConnectionStrings | Where {$_.name -eq "ExtendedSiteDBContext"}

$ConnectionString = $Connection.ConnectionString

Write-host "Executing Database migrations and seeding with Migrate.exe"

& "$PSScriptRoot\migrate.exe" $dll /connectionString=$ConnectionString /connectionProviderName="System.Data.SqlClient" /verbose

if ($LastExitCode -ne 0) {

throw 'migrate.exe returned a non-zero exit code...'

}

Write-host "Finished executing Database migrations and seeding with Migrate.exe"

Write-host "Finished"

What that script does when called from an Azure Powershell task in a Release definition is lookup the connectionString from the web app service in Azure and then uses that to call migrate.exe. We did this so we did not have to store any connectionStrings in VSTS. If the sql user in the connectionString used when executing migrate.exe needs different permissions to that of the website you could change this to use VSTS release variables instead.

2. The Release definition

When we release a site our process is:

· Stop the deployment slot ‘stage’

· Deploy the website zip to slot ‘stage’

· Update the database using the PowerShell script

· Start the Stage site

· Swap Stage with production

· Ping production site

· Stop the stage site (to save resources)

Our release definition looks like this:

Release definition	Task Groups

The Update Database task is just an Azure PowerShell task containing:

In Azure the App Service Plan is configured so that the production site and the stage slot are almost identical (baring a few appSetting values). They share the same connectionString and every appSetting and ConnectionString value has the ‘Slot Setting’ checkbox ticked.

Our big gotcha

Was during the swap slots task the website will be warm started automatically by that process hitting localhost under http, or it will try the domain name of the site but again under http. We had this in our filter.config

// Ensure all http connections are redirected to https

filters.Add(new RequireHttpsAttribute());

Which meant the warm start process instantly failed to start the site, the swap process continued and the site then started up from cold in production. We could see that because our ping task that pinged the production site, was taking a minute to respond.

What we had to do was create a custom version of the RequireHttpsattribute, once we did this our ping task responds in 1-3 seconds:

// Ensure all http connections are redirected to https

filters.Add(new CustomRequireHttpsAttribute());

And the code for the attribute is:

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, Inherited = true, AllowMultiple = false)]

public class CustomRequireHttpsAttribute : RequireHttpsAttribute

{

    protected override void HandleNonHttpsRequest(AuthorizationContext filterContext)

{

    string ipAddress = filterContext.RequestContext.HttpContext.Request.UserHostAddress;

    var userAgent = filterContext.RequestContext.HttpContext.Request.UserAgent.ToLower();

    // 0.0.0.0 and 127.0.0.1 and ::1 are used by the Azure App Service Swap Slot process to Warm up the site before swapping the slot to production

    if(ipAddress == "0.0.0.0" || ipAddress == "::1" || ipAddress == "127.0.0.1")

    //if(userAgent.Contains("sitewarmup")) // doesnt work

    {

        return;

    }

    base.HandleNonHttpsRequest(filterContext);

    }

}

Release Management in TFS 2015.2

With the TFS 2015.2 update we now have the ability to use Release Manager in TFS on premise. This is how I have setup our branching structure, gated check-in and release manager to control the deployment (with approvals at some stages) of an MVC5 web site with Entity Framework into multiple Azure Web Application environments.

We have our branching structure set as:

Dev\developer name 1

Dev\developer name 2 etc

Main

Developers take a branch from main into the dev folder under their name and work on changes. When development is complete they check-in to their dev branch and then merge the changes into the Main branch. This allows the developer to pull in other changes from main into their dev branch, and be able to check-in at least daily into their dev branch so their code is backed up overnight.

We are using the new TFS build tasks; and setup against the Main branch is:

• A gated check-in.

• And a number of build tasks that compile the code in Release mode, run some unit tests and finally copy the files/build artifacts we need to deploy the web site into the drop folder (essentially a web deploy zip file is created).

 

The key to creating a web deploy zip file is the MS Build Arguments:

/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true

In the copy and Publish build artifacts task we have:

This automatically builds all changes checked into the Main branch with the final step it produces the web deploy files. Assuming the build passes then Release Manager will take over to deploy the web site into an Azure Web Application.

We have 4 environments in Azure that we need to deploy to ‘UAT Staging’, ‘UAT’, ‘Production Staging’ and ‘Production’. For us these are split into 2 Azure subscriptions, one for UAT one for Production. Within each subscription is 1 web application e.g. UAT and against that 1 slot ‘staging’ and each has its own Azure SQL database. This gives us 2 websites and 2 databases. We make use of sticky slot settings as well so the connection string / app settings stay against its environment because in production we make use of the Azure web application swap slot functionality.

So we configure release manager to deploy the code to UAT staging, UAT and Production staging. But for production release manager just initiates a swap slot PowerShell command.

So in Release manager we will need to setup the deployment process or tasks for each of these environments.

But first Release manager is setup to watch the main branch for a new build artifact which is done by setting the release trigger to Continuous Deployment.

As soon as a new version is checked in setting the UAT Staging environment trigger to ‘Automated after release creation’ will initiate a deployment into that environment automatically (and for us with no approvals required because that is the first server in Azure which our developers can test against).

On the environments tab we define the deployment steps for each environment so for UAT Staging we have:

• Stop the Azure web application
• Deploy the new code/web site
• Start the Azure web application

We are making use of the new TFS Market Place extension ‘Run Inline Azure Powershell’ task which allows us to stop the Azure web application with:

Stop-AzureWebsite -Name $(WebAppName) -Slot stage

And below are the properties of the Azure Web App Deployment task, with the main one being the path to the Web Deploy Package.

Note: Our web applications are always running in Azure we are not creating them on demand.

The UAT and Production Staging environments all have the same 3 tasks. Deployment into an environment takes about 1 minute 20 seconds.

The Production environment has 1 task which swaps Production Staging with Production by executing this powershell script:

Switch-AzureWebsiteSlot -Name $(WebAppName) -Slot1 stage -Slot2 production -Force -Verbose

That allows us to do a quick production deployment (saving us that 1 minute 20).

UAT, Production Staging and Production all have Pre-Deployment approvers setup.

So deployment to the 1^st Azure environment ‘UAT Staging’ happens automatically upon a successful check-in to the Main branch.

The developer has the chance now to manually test the site. When they want to deploy to the next environment ‘UAT’ they would open the release in release manager and start the deployment:

By using the ‘Deploy’ button to request the code is deployed into the UAT environment this will send an email to the approvers who will ‘hopefully’ approve the release and if they do release manager will execute the 3 deployment tasks setup for the UAT environment.

The same process would be followed for the remaining environments, the developer tests then uses the deploy button to move the same web application code to the next environment.

At any point we can check what release is in what environment by looking at the overview tab.

Database changes are done via Entity Framework Code First Migrations which are executed upon web site start up by running this code added to the MVC site startup.cs class:

var efConfiguration = new Configuration();

var dbMigrator = new System.Data.Entity.Migrations.DbMigrator(efConfiguration);

dbMigrator.Update();

This will execute any schema changes and then run the seed data method. The key to keeping the seed data updated is using the extension method ‘AddOrUpdate’.

 

Useful Links

https://msdn.microsoft.com/library/vs/alm/release/overview