One year of Akka.net in production

One year ago we went live in production with a rewrite of the core of the platform in Akka.net. Although the initial release date had to be delayed because a bug we found in the pre-production environment in large Akka.net Clusters, I have to say that after fine tuning remoting and clustering parameters, the system has been stable and the improvement of the performance in soft real time processing is noticeable and also more scalable.
The core of the system is formed of two services plus the lighthouse, initially we went live with a single cluster of more than 50 instances, in the latest release this week the count has gone up to 101! service instances distributed in three clusters.
After this experience we have continued using Akka.net in other parts of the platform like dynamic clusters of pub-sub actors used to distribute the load and control the concurrency when processing messages from RabbitMq.
Also have started using Akka.net in conjunction with .net core, and had the opportunity to contribute to the Akka.DI.Ninject repository to convert it to .net standard 2.0 so it could be used in .net core services.

Using python and OctopusApi to handle rollbacks

This week we have migrated a large amount of servers and services to use new clusters of MongoDB, Aerospike and RabbitMQ. Because all this configuration is in Octopus variables (connection strings), we just needed to change those variables to point to the new infrastructure. All good in principle, change the variables and create new releases for each service; but that leaves us exposed to three potential problems:

1. if an emergency release is needed, it will automatically point to the new infrastructure and it could not be ready for the switch over.
2. if an unexpected problem is found after the release and we want to rollback, we would need to modify all the Octopus variables and create a new release to rollback to the old infrastructure (or rollback to the old version of the code).
3. additionally, in both cases, creating a new release after the deployment could pick up variable changes made for future releases like feature flags.

To address those possible problems, I decided to add a rollback role to all the servers that were planned to be deployed (30+) and then add the same rollback role to the existing connection string variables, and then create new variables with the same roles as the originals. This way if a new release needs to be created it will use the rollback variables. Then right before the release we would remove the rollback role so when deploying the new connection strings would be used. And in the case of needing a rollback we would just add back the rollback role and deploy the same release.

The only problem was adding and removing that role to the large amount of servers, this is were I just put together my very first python program to use Octopus Api to add or remove a role. The code is a bit too raw, but other members of the team have already started to used it to setup new deployment targets. I have to say that I'm impressed how natural and quick was to write the script in Python.

import requests

OctopusUrl = 'http://***octopusserver***'

headers = {'X-Octopus-ApiKey' : 'API-***********'}

newRole = 'Rollback'

environmentName = 'Production'

#environment

machinesUrl = 'null'

environments = requests.get(OctopusUrl+'/api/environments', headers=headers).json()

for environment in environments['Items']:

if (environment['Name'] == environmentName):

       machinesUrl = environment['Links']['Machines']

#machines

machines = requests.get(OctopusUrl+machinesUrl, headers=headers).json()

machinesList = []

machineEndPage = False

while not machineEndPage:

for machine in machines['Items']:

    if ('MainRole in machine["Roles"]):

           machinesList.append(machine)

   nextMachinesUrl =machines['Links'].get('Page.Next', 'null')

if (nextMachinesUrl != 'null') :

    machines = requests.get(OctopusUrl+nextMachinesUrl, headers=headers).json()

else:

       machineEndPage = True

for machine in machinesList:

#if (newRole not in machine['Roles']): #Add role

    #machine['Roles'].append(newRole)

if (newRole in machine['Roles']): # remove

    machine['Roles'].remove(newRole)

       machineUrl = OctopusUrl+machine['Links']['Self']

    result = requests.put(machineUrl, json=machine, headers=headers)

       print(machine['Name']+' '+result)

Video: What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney

This morning during breakfast I came across this great video https://youtu.be/kb-m2fasdDY regarding the problems Uber had to overcame when it moved to a microservice architecture. Although in a very smaller scale, it is surprising how the organisation I'm currently part of, had/has the same problems: 

• rest/json contracts need integration tests
• too much logging
• logging not uniform across different technologies
• tracing agreement
• language silos (zealots)
• too many repositories
• hard to coordinate multi team deployments
• incidence ownership
• load testing is hard

Request Linking in asp.net core

In systems where events are processed in several microservices is useful to maintain a request identifier to be able to link logs from each microservice to do some investigation or to measure performance.
So far we have relied in a custom http header and owin middleware to read, add a new segment to this request identifier and then set it in the log4net context so it can be written when logging an event.
For a new asp.net core service I'm working, I have quickly put together new middleware to do the same operation using the documentation, but instead of using log4net I have found that it is easier to use NLog to capture the TraceIdentifier property in the HttpContext.

Middleware

public class TransactionLinkingMiddleware{

private readonly RequestDelegate _next;
public const string ParentTransactionHttpHeader = "Transaction-Link";
public TransactionLinkingMiddleware(RequestDelegate next)
{ 

_next = next;   

}

public Task Invoke(HttpContext context)

{

if (context.Request.Headers.TryGetValue(ParentTransactionHttpHeader, out var headerValues)
                                    && headerValues.Any())

{

context.TraceIdentifier = $"{headerValues.First()}/{context.TraceIdentifier}";
}

// Call the next delegate/middleware in the pipeline

return this._next(context);

}

}

public static class TransactionLinkingMiddlewareExtensions{   

public static IApplicationBuilder UseTransactionLinking(this IApplicationBuilder builder)
                  {        

return builder.UseMiddleware<TransactionLinkingMiddleware>();
} 

}

Startup.cs

public Startup(IConfiguration configuration, IHostingEnvironment env)
{   

Configuration = configuration;    env.ConfigureNLog("NLog.config");
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env,
ILoggerFactory loggerFactory,    IApplicationLifetime applicationLifetime)

{

if (env.IsDevelopment())

{       

app.UseDeveloperExceptionPage();

}

app.UseTransactionLinking();

loggerFactory.AddNLog();

app.UseMvc();

}

NLog.config

<?xml version="1.0" encoding="utf-8" ?><nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"> 

<extensions>

<add assembly="NLog.Web.AspNetCore"/>

</extensions>

<targets>

<target xsi:type="File" name="File"
fileName="${basedir}/logs/log.txt"
layout="TimeStamp=${longdate} Level=${uppercase:${level}} Transaction=${aspnet-TraceIdentifier} Message=${message}" />
</targets> 

<rules>

<logger name="*" minlevel="Debug" writeTo="File" />

</rules>

</nlog> 

NLog.Web and NLog.Web.AspNetCore packages need to be installed https://github.com/NLog/NLog.Web

Using Memory and CPU profilers to identify a production issue

As I explained in my previous post, the previous Friday I managed to fix and deploy a priority one issue that caused a prolonged outage in production, while I had the flu.

The symptoms were evident, randomly, one by one, the instances of one of the most important services in the platform started to use more memory and after few minutes it was unresponsive. Luckily for us, the load of this service is distributed among over twenty instances in the production environment, this allowed the team to actively identify and restart the services that were becoming irresponsive, minimizing that way the downtime of the system.

At first we thought that this issue could be caused by some Aerospike server maintenance we had that week causing Aerospike's .net client to misbehave and leaking memory. To make things worst, we also had a production release the same week with over fifteen services deployed, so while part of the team worked on palliative measures to downtime each team that had released a new version started to review all changes made and think if they could be causing the issue downstream. The situation escalated quickly, and soon we started to turn off feature flags and later to rollback to previous versions of the services.

In parallel to helping other members of the team to create and configure more boxes to distribute the load and to redirectionate traffic, I began to investigate the problem.

After looking at logs and the typical performance indicators like number of threads, handlers, connections and so on; and because one of the symptoms was high memory usage, I waited for one of the instances to start misbehaving to disconnect it from the main cluster and then take a memory snapshot using dotMemory.

Originally I thought that I would find a large number of objects related to one of the client libraries used by the service like Aerospike, MongoDb or RabbitMq, but instead I found a large amount of Akka.net messages. This services form an Akka.net cluster of more than fifty nodes and the actors within the system communicate between them using messages, so looking at the type of the message I was able to identify what part of the flow uses those messages and that way narrow down the problem.

This service is a rewrite of part of the system, but unfortunately, not a full rewrite, so the actor affected by the issue is calling in to legacy code too convoluted to easily identify the issue. With this information, my next step was to take a CPU snapshot using dotTrace; at first, the snapshot was pretty overwhelming, as there were hundreds of threads going on. Here is where I used the information I had gathered previously form the memory snapshot to filter threads and identify the threads that were misbehaving.

At the end I found that the method blocking the processing was three to five years old code used to perform tag replacement on user's templates. It was evident to me that the while loop below was causing the code to enter in an infinite loop and so, taking down a thread (and also an actor). The only thing left was to refactor that code, write unit tests around it to prove that the bug was there, and that my fix  would work. And once the pull request had the green light from all other team members I released it to preprod, and then production.

This loop never ends if the input string is malformed, for example "{tag} with {bad) format", and there were no validations in the UI or back-end for it...

//Remove any unmatched tags
while (name.IndexOf("{", StringComparison.Ordinal) != -1)
{
name = name.Substring(0, name.IndexOf("{", StringComparison.Ordinal))
+ name.Substring(name.IndexOf("}", StringComparison.Ordinal) + 1);
}

Friday afternoon deployment

Just finished finding, fixing and deploying to production a bug (infinite loop!) that had been there for more than three years and was causing a priority one issue. The best part is that I have been only around for around a year and a half and had not even seen that code before and that was supposed to be off because have the flu... what could have gone wrong?

Span the best part of the .net core 2.1 roadmap

The newly announced .net core 2.1 performance enhancements in build time and Http Client are great, but the best addition is the Span<T> and Memory<T> types in C# 7.2 that follow the low memory allocation fashion started with ValueTask<T> in C#7.
The .net environment has became one of the top contenders performance-wise, and although this new features won't probably be extensively used by most .net developers; now, I can only think of all the services where I could have or will use them.