Using python and OctopusApi to handle rollbacks

This week we have migrated a large amount of servers and services to use new clusters of MongoDB, Aerospike and RabbitMQ. Because all this configuration is in Octopus variables (connection strings), we just needed to change those variables to point to the new infrastructure. All good in principle, change the variables and create new releases for each service; but that leaves us exposed to three potential problems:

1. if an emergency release is needed, it will automatically point to the new infrastructure and it could not be ready for the switch over.
2. if an unexpected problem is found after the release and we want to rollback, we would need to modify all the Octopus variables and create a new release to rollback to the old infrastructure (or rollback to the old version of the code).
3. additionally, in both cases, creating a new release after the deployment could pick up variable changes made for future releases like feature flags.

To address those possible problems, I decided to add a rollback role to all the servers that were planned to be deployed (30+) and then add the same rollback role to the existing connection string variables, and then create new variables with the same roles as the originals. This way if a new release needs to be created it will use the rollback variables. Then right before the release we would remove the rollback role so when deploying the new connection strings would be used. And in the case of needing a rollback we would just add back the rollback role and deploy the same release.

The only problem was adding and removing that role to the large amount of servers, this is were I just put together my very first python program to use Octopus Api to add or remove a role. The code is a bit too raw, but other members of the team have already started to used it to setup new deployment targets. I have to say that I'm impressed how natural and quick was to write the script in Python.

import requests

OctopusUrl = 'http://***octopusserver***'

headers = {'X-Octopus-ApiKey' : 'API-***********'}

newRole = 'Rollback'

environmentName = 'Production'

#environment

machinesUrl = 'null'

environments = requests.get(OctopusUrl+'/api/environments', headers=headers).json()

for environment in environments['Items']:

if (environment['Name'] == environmentName):

       machinesUrl = environment['Links']['Machines']

#machines

machines = requests.get(OctopusUrl+machinesUrl, headers=headers).json()

machinesList = []

machineEndPage = False

while not machineEndPage:

for machine in machines['Items']:

    if ('MainRole in machine["Roles"]):

           machinesList.append(machine)

   nextMachinesUrl =machines['Links'].get('Page.Next', 'null')

if (nextMachinesUrl != 'null') :

    machines = requests.get(OctopusUrl+nextMachinesUrl, headers=headers).json()

else:

       machineEndPage = True

for machine in machinesList:

#if (newRole not in machine['Roles']): #Add role

    #machine['Roles'].append(newRole)

if (newRole in machine['Roles']): # remove

    machine['Roles'].remove(newRole)

       machineUrl = OctopusUrl+machine['Links']['Self']

    result = requests.put(machineUrl, json=machine, headers=headers)

       print(machine['Name']+' '+result)

Video: What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney

This morning during breakfast I came across this great video https://youtu.be/kb-m2fasdDY regarding the problems Uber had to overcame when it moved to a microservice architecture. Although in a very smaller scale, it is surprising how the organisation I'm currently part of, had/has the same problems: 

• rest/json contracts need integration tests
• too much logging
• logging not uniform across different technologies
• tracing agreement
• language silos (zealots)
• too many repositories
• hard to coordinate multi team deployments
• incidence ownership
• load testing is hard

Request Linking in asp.net core

In systems where events are processed in several microservices is useful to maintain a request identifier to be able to link logs from each microservice to do some investigation or to measure performance.
So far we have relied in a custom http header and owin middleware to read, add a new segment to this request identifier and then set it in the log4net context so it can be written when logging an event.
For a new asp.net core service I'm working, I have quickly put together new middleware to do the same operation using the documentation, but instead of using log4net I have found that it is easier to use NLog to capture the TraceIdentifier property in the HttpContext.

Middleware

public class TransactionLinkingMiddleware{

private readonly RequestDelegate _next;
public const string ParentTransactionHttpHeader = "Transaction-Link";
public TransactionLinkingMiddleware(RequestDelegate next)
{ 

_next = next;   

}

public Task Invoke(HttpContext context)

{

if (context.Request.Headers.TryGetValue(ParentTransactionHttpHeader, out var headerValues)
                                    && headerValues.Any())

{

context.TraceIdentifier = $"{headerValues.First()}/{context.TraceIdentifier}";
}

// Call the next delegate/middleware in the pipeline

return this._next(context);

}

}

public static class TransactionLinkingMiddlewareExtensions{   

public static IApplicationBuilder UseTransactionLinking(this IApplicationBuilder builder)
                  {        

return builder.UseMiddleware<TransactionLinkingMiddleware>();
} 

}

Startup.cs

public Startup(IConfiguration configuration, IHostingEnvironment env)
{   

Configuration = configuration;    env.ConfigureNLog("NLog.config");
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env,
ILoggerFactory loggerFactory,    IApplicationLifetime applicationLifetime)

{

if (env.IsDevelopment())

{       

app.UseDeveloperExceptionPage();

}

app.UseTransactionLinking();

loggerFactory.AddNLog();

app.UseMvc();

}

NLog.config

<?xml version="1.0" encoding="utf-8" ?><nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"> 

<extensions>

<add assembly="NLog.Web.AspNetCore"/>

</extensions>

<targets>

<target xsi:type="File" name="File"
fileName="${basedir}/logs/log.txt"
layout="TimeStamp=${longdate} Level=${uppercase:${level}} Transaction=${aspnet-TraceIdentifier} Message=${message}" />
</targets> 

<rules>

<logger name="*" minlevel="Debug" writeTo="File" />

</rules>

</nlog> 

NLog.Web and NLog.Web.AspNetCore packages need to be installed https://github.com/NLog/NLog.Web