One configuration file to rule them all. Automated deployment for multiple .NET websites.

Spring is almost over and my garden is now full of weeds (especially that evil crab grass), apart from a two metre square where I went nuts for an hour.
So naturally, being at one with nature, I was thinking about build and deploy scripts. About the sticky tape and chewing gum shambles of third party libraries and configuration it's been my pleasure to marshall into release packages over the years.

It doesn't have to be that way.  As I watch a line of ants crawl up a weed into the car..

Build and deployment can be challenging to manage. I have settled on a strategy I find flexible enough to cope with complex web deployments and easy enough to reliably maintain.

The Plan

So in short, here's the plan

• Build a package to be deployed that contains all the software and configuration sets for each environment;
• All configuration originates from a single source, with optional defaults and different values for each environment;
• Generate configuration for each website or application for each environment from values in the root configuration file;
• Deployment scripts use the same root configuration file; and
• Specify the environment when running deployment, which selects the appropriate set of configuration.

A Windows Example

You could feasibly use this strategy on any platform.  This example will implement it using MSBuild, Visual Studio Web Deploy Packages, MSDeploy and only 71 lines of PowerShell (and that's counting comments and braces).

MSDeploy solves many of these problems for a single website.  In particular, flexible parameterisation is available using Parameters.xml and SetParameters.xml files.  Some additional work is needed to share configuration items across multiple websites and the deployment process.

Please note for brevity, I have removed any specific error handling from any of these examples.

Build a Package for Each Website

We will use MSBuild to create a Web Deploy Package for each website.

The first thing to do is open the solution in Visual Studio 2012 (or 2013 now) and create a publish profile for each website to be deployed.

1. Right-click on the website and select Publish...
2. Select <New...> from the dropdown and enter a name, eg "Deploy". Make sure this name is the same for each website you wish to deploy.
3. Select Publish method: Web Deploy Package.
4. Then browse for a package location.  Put all the website packages into the same folder at the solution level eg ..\DeployPackages
5. Enter the site URL assuming all sites will be deployed under Default Web Site for the moment. eg Default Web Site/MvcApplication1 (You will change this for different environments later)
6. Click Next
7. Enter any required database connections, if any and click Next
8. Click Publish.

The package will be created with a .deploy.cmd file in the DeployPackages folder.  This command is a wrapper for MSDeploy.exe.  There will also be a .SetParameters.xml file which we will use to provide different sets of configuration during deployment.

To deploy the package to your local IIS, run the .deploy.cmd file with /Y.  To test packaging run it with /T, it will act like it is doing a deployment without actually doing anything.  Make sure you run it as administrator.

Note: The Deploy.pubxml file created for the Deploy profile will have an absolute path for the Package location.  This will likely cause problems for your team members and your build machine.  Edit the file and replace the path with $(ProjectDir) as follows.

    <DesktopBuildPackageLocation>$(ProjectDir)..\DeployPackages\MvcApplication1.zip</DesktopBuildPackageLocation>

Parameterise your Configuration

Now this will just deploy your software with configuration as is.  We now need to parameterise for different environments.

Add a file called Parameters.xml to the root of each website to deploy.

Add a <parameters> element.  Then for each parameter that differs per environment, add <parameter> and <parameterEntry> definitions as follows.  Notice how the parameterEntry match value replaces the value attribute by locating the matching key.
<?xml version="1.0" encoding="utf-8" ?>
<parameters>
  <parameter name="MyValue"
             description="Please provide my configuration value."
             defaultValue="Wilma">
    <parameterEntry kind="XmlFile"
                    scope="Web.config"
                    match="//configuration/appSettings/add[@key='MyConfigurationValue']/@value" />
  </parameter>
</parameters>

When you run publish, this will now create a .SetParameters.xml file in the DeployPackages folder. A copy of the .SetParameters.xml file must be created for each of your environments.

To run the publish step from the command line, use MSBuild from PowerShell (or your favourite shell):
$MSBUILD = Join-Path $Env:SystemRoot "Microsoft.NET\Framework\v4.0.30319\MSBuild.exe"
. $MSBUILD MvcApplication1.sln /p:DeployOnBuild=true /p:PublishProfile=Deploy


Generate .SetParameters.xml Files

These .SetParameters.xml files start to multiply (literally) once you start to deploy a few websites in a few different environments.

The answer to this problem is to put any configuration that changes per environment into a .CSV file with rows for configuration items and columns for each environment.  This .CSV file will then be used to generate the .SetParameters.xml files.

Create a folder called "Templates" in your solution folder and copy each .SetParameters.xml file into it.  Replace configuration values as required like so:
  <setParameter name="MyValue" value="{MyConfigurationItem}" />

In the solution folder, or wherever you put your deployment scripts, create a file Configuration.csv.  The first column should be called "Item", and the remaining columns the names of your environments as follows:
Item,Build,Test,Production
MyConfigurationItem,Barney,Barney,Wilma
MvcApplication1.Server,localhost,TestServer,ProdServer



This script will now perform the package as well as generate a set of configuration for each environment.
# Package.ps1: Packaging script
$ScriptPath = Split-Path $SCRIPT:MyInvocation.MyCommand.Path -Parent
# Build the package
$MSBUILD = Join-Path $Env:SystemRoot "Microsoft.NET\Framework\v4.0.30319\MSBuild.exe"
. $MSBUILD MvcApplication1.sln /p:DeployOnBuild=true /p:PublishProfile=Deploy
# Generate configuration
. .\Configuration.ps1
$ConfigurationFile = Join-Path $ScriptPath "Configuration.csv"
$TemplateFolder = Join-Path $ScriptPath "Templates"
$PackageFolder = Join-Path $ScriptPath "DeployPackages"
$Environments = (Get-Content $ConfigurationFile | Select -First 1).Split(",") | Select -Skip 1
$TemplateFiles = Get-ChildItem $TemplateFolder
foreach($Environment in $Environments) {
 $Configuration = ReadConfiguration $ConfigurationFile $Environment
 $EnvironmentFolder = Join-Path $PackageFolder $Environment
 if (!(Test-Path $EnvironmentFolder)) {
  New-Item -Path $EnvironmentFolder -ItemType Directory | Out-Null
 }
 
 foreach($TemplateFile in $TemplateFiles) {
  $TargetFile = Join-Path $EnvironmentFolder $TemplateFile.Name
  $SourceFile = $TemplateFile.FullName
  GenerateConfiguration $SourceFile $TargetFile $Configuration
 }
}
# Configuration.ps1: Common configuration functions
function ReadConfiguration([String] $ConfigurationFile, [String] $Environment)
{
 $Configuration = @{}
 # Read the Item column and the column matching the name of the environment
 # Rename the environment column to "Value"
 $Config = Import-Csv $ConfigurationFile | Select Item,@{Name="Value"; Expression={$_.$Environment}}
 foreach($Item in $Config) {
 $Configuration.Add($Item.Item, $Item.Value)
 }
 # Return configuration
 $Configuration
}
function GenerateConfiguration([String] $SourceFile, [String] $TargetFile, [Hashtable] $Configuration)
{
 $Text = [IO.File]::ReadAllText($SourceFile)
 foreach($Item in $Configuration.Keys) {
 $ItemTag = "{" + $Item + "}"
 if ($Text.Contains($ItemTag)) {
 $Text = $Text.Replace($ItemTag, $Configuration.$Item)
 }
 }
 Write-Host "Writing $TargetFile"
 [IO.File]::WriteAllText($TargetFile, $Text)
}




Deploy the Packages

The deployment can now use the packages and configuration created under DeployPackages to deploy to any environment.

Make sure the "Web Deployment Agent Service" is running and that you run the script as an administrator.

Pass the environment to this deploy script and it will use the set parameters files generated for that environment.

eg powershell .\Deploy.ps1 Build

And there we have it; a single location for all configuration across multiple websites for both package and deployment processes!

An interesting side note: MSDeploy errors with -1, which PowerShell doesn't catch. Stay tuned for the solution to that nice little problem.
# Deploy.ps1: Deployment script
Param(
[String] $Environment = $(throw "Please specify the environment")
)
$ScriptPath = Split-Path $SCRIPT:MyInvocation.MyCommand.Path -Parent
Write-Host "Deploying to environment $Environment"
$ConfigurationFile = Join-Path $ScriptPath "Configuration.csv"
$PackageFolder = Join-Path $ScriptPath "DeployPackages"
$Packages = Get-ChildItem $PackageFolder *.deploy.cmd
. .\Configuration.ps1
$Configuration = ReadConfiguration $ConfigurationFile $Environment
$ConfigurationFolder = Join-Path $PackageFolder $Environment
foreach($Package in $Packages)
{
 $PackageName = $Package.Name.Replace(".deploy.cmd", "")
 $ServerItem = "$PackageName.Server"
 $Server = $Configuration[$ServerItem]
 $SetParametersFile = "$PackageName.SetParameters.xml"
 Write-Host "Deploying $PackageName to server $Server"
 $Env:_DeploySetParametersFile = Join-Path $ConfigurationFolder "$SetParametersFile"
 . $Package.FullName /Y /M:$Server
}

Now, the ants in the car..