Using python and OctopusApi to handle rollbacks

This week we have migrated a large amount of servers and services to use new clusters of MongoDB, Aerospike and RabbitMQ. Because all this configuration is in Octopus variables (connection strings), we just needed to change those variables to point to the new infrastructure. All good in principle, change the variables and create new releases for each service; but that leaves us exposed to three potential problems:

1. if an emergency release is needed, it will automatically point to the new infrastructure and it could not be ready for the switch over.
2. if an unexpected problem is found after the release and we want to rollback, we would need to modify all the Octopus variables and create a new release to rollback to the old infrastructure (or rollback to the old version of the code).
3. additionally, in both cases, creating a new release after the deployment could pick up variable changes made for future releases like feature flags.

To address those possible problems, I decided to add a rollback role to all the servers that were planned to be deployed (30+) and then add the same rollback role to the existing connection string variables, and then create new variables with the same roles as the originals. This way if a new release needs to be created it will use the rollback variables. Then right before the release we would remove the rollback role so when deploying the new connection strings would be used. And in the case of needing a rollback we would just add back the rollback role and deploy the same release.

The only problem was adding and removing that role to the large amount of servers, this is were I just put together my very first python program to use Octopus Api to add or remove a role. The code is a bit too raw, but other members of the team have already started to used it to setup new deployment targets. I have to say that I'm impressed how natural and quick was to write the script in Python.

import requests

OctopusUrl = 'http://***octopusserver***'

headers = {'X-Octopus-ApiKey' : 'API-***********'}

newRole = 'Rollback'

environmentName = 'Production'

#environment

machinesUrl = 'null'

environments = requests.get(OctopusUrl+'/api/environments', headers=headers).json()

for environment in environments['Items']:

if (environment['Name'] == environmentName):

       machinesUrl = environment['Links']['Machines']

#machines

machines = requests.get(OctopusUrl+machinesUrl, headers=headers).json()

machinesList = []

machineEndPage = False

while not machineEndPage:

for machine in machines['Items']:

    if ('MainRole in machine["Roles"]):

           machinesList.append(machine)

   nextMachinesUrl =machines['Links'].get('Page.Next', 'null')

if (nextMachinesUrl != 'null') :

    machines = requests.get(OctopusUrl+nextMachinesUrl, headers=headers).json()

else:

       machineEndPage = True

for machine in machinesList:

#if (newRole not in machine['Roles']): #Add role

    #machine['Roles'].append(newRole)

if (newRole in machine['Roles']): # remove

    machine['Roles'].remove(newRole)

       machineUrl = OctopusUrl+machine['Links']['Self']

    result = requests.put(machineUrl, json=machine, headers=headers)

       print(machine['Name']+' '+result)

Reverse Proxying Kestrel with URL Rewrite for IIS

As said in previous posts, we have been developing new services using Asp.net Core and self-hosting as Windows services using Kestrel. A new interesting problem arised, as Kestrel can not share the Http/Https port with IIS or Owin self hosted services we had to use a different port to host each service. In he past we had this problem when hosting APIs with f# suave, and had overcome it redirecting the requests in the F5 Big Ip load balancer. That solution works, but sometimes I feel like it crosses in to the domain of infrastructure instead of development. So in order to contain all the deployment and configuration in TeamCity/Octopus, I decided to use IIS to reverse proxy the calls made to these microservices as explained in: https://weblogs.asp.net/owscott/creating-a-reverse-proxy-with-url-rewrite-for-iis
Basically requests coming to http://+/newmicroserviceapi/ will be rewritten to http://localhost:<microserviceport>/
This is the Octopus Deploy step I created to automate all this:

PowerShell Script:

$siteName = $OctopusParameters['ReverseProxy.IISSiteName']
$pathBase = $OctopusParameters['ReverseProxy.PathBase']
$port = [int] $OctopusParameters['ReverseProxy.Port']
$site = "iis:\sites\$siteName"
$filterName="Reverse proxy inbound $pathBase"
$filterPath = "system.webServer/rewrite/rules/rule[@name='$filterName']"
Write-Host 'Adding reverse proxy rule to '$siteName' for '$pathBase'/*'
Clear-WebConfiguration -pspath $site -filter $filterPath
Add-WebConfigurationProperty -pspath $site -filter "system.webServer/rewrite/rules" -name "." -value @{name=$filterName; patternSyntax='Regular Expressions'; stopProcessing='True';}
Set-WebConfigurationProperty -pspath $site -filter "$filterPath/match" -name "url" -value "$pathBase/(.*)"
Set-WebConfigurationProperty -pspath $site -filter "$filterPath/conditions" -name "logicalGrouping" -value "MatchAny"
Set-WebConfigurationProperty -pspath $site -filter "$filterPath/action" -name "type" -value "Rewrite"
Set-WebConfigurationProperty -pspath $site -filter "$filterPath/action" -name "url" -value "http://localhost:$port/{R:1}"

Octopus Step:

Note that the Application Request Routing IIS pluging is needed.