Visualising Lots of Data

In a previous blog entry (Calculating Percentiles with Tableau.) I discussed ways to visualise the distribution of transaction response times and showed how a percentile chart and a scatter plot can provide a very effective and complementary pair of views of the same data.  In particular, I described how to make that viable with large datasets.

I’ve recently been analysing some really, really large data sets with an extremely wide range of data values and experimenting with alternative ways to visualise that.  One particular representation which I describe below has proved very effective.

The data sets in question hold details of events recorded for all threads of all processes on a busy database server around the time of unexplained blockages.  I am trying to find clues about the cause of the blockages.  Each sample has around 100 million events, spanning anywhere from 20 seconds to 2 minutes, depending on system activity levels at the time.  The peak event rate was around 10 million events per second.

With that much data, it’s useful to start by getting a high level understanding of the data distribution in as many dimensions as possible, before getting too buried in the detail.  One measure which has proved interesting is the delay between events for each thread.  The actual values of the delays don’t initially give much insight in themselves, but changes in the pattern of delays can provide clues about changes in system behaviour, and hence where to look next.

The delays ranged from around 100 nanoseconds to 100 seconds, with the vast majority down at the microsecond level.  The challenge was therefore to find a way to represent the distribution of such a large set of values spanning such a wide range, in a way that would highlight changes in patterns of behaviour.

How about a percentile plot?

The technique I discussed in the previous blog posting proved to be quite capable of calculating the percentiles over a data set of this size – in fact this data was one of the triggers which prompted me to figure out that technique.

The first thing that became clear when trying that on this data distribution was that the very wide range of the scale of the values (100 nano-seconds to 100 seconds) required a logarithmic scale for the display, otherwise all of the detail is obscured by the scale required to see the outliers.

With the benefit of the log scale, the period of the blockage (from about 50 seconds to 118 seconds into the sample) becomes very obvious.

But even with a log scale, displaying one percentile at a time (such as the line chart shown here of the 99th percentile) really doesn’t convey the whole story at a glance.  Sure, you can explore the distribution by looking at different percentiles – but I want a single visualisation which gives an “aerial view” of everything.

Or a scatter chart?

The next thing I tried was a scatter chart, bucketing both the offset time and the delay values, to make the number of marks manageable, using the approach I described in the previous blog posting.

Once again, I had to use a log scale to handle the range of values (i.e. bucketing the log of the delay values), but even then, I was left with a dense mass of marks and it was initially impossible to tell much about the data distribution.

One approach which can help with that is to use a heat-map – with the colour of each mark representing the number of events in that bucket.

Once again, the scale of numbers involved meant that initially that wasn’t very effective.  The number of events per bucket ranged from 1 to over 100,000, with the vast majority of marks down at the low end of that range.  So it was hard to arrange the colour scale to convey any useful information.  So I ended up using a second log scale – this time colouring the marks by the log of the number of events (below).

That was more effective and definitely started to convey something of the distribution – with the blockage times (from about 52 to 118 seconds) again clearly visible, since there were far fewer events during the blockage, resulting in much fewer red marks at those times.

Especially when viewed at higher resolution, a wealth of detail can be seen during the period of the blockages, as can be seen at the left.

I have no idea what the “gothic arches” mean, but as this is the time when the blockage was happening, they may well be significant.

[Edit: I had a quick poke around to see what causes the arches.  The reason for the shape is simple.  These are really just straight lines, distorted by the log scale.  It turns out that they are generated by the processor's idle loop.  I still don't understand why the delays between those events are following that pattern, but I'm inclined to think that this is a consequence of the blockage, not the cause.]

Whilst this view is quite effective, it is still very difficult to quantify anything from looking at it.

What was the rate of events?

What was the distribution of response times?

What about a log-scale percentile rank heat-map?

A what?

I decided to try a cross between the two approaches and use a heat map representation of percentiles.

The problem with the first attempt with percentiles was that it only shows one percentile at once.  So why not display all the percentiles – by calculating the percentile rank of all points and using a continuous colour scale to show the distribution.

That approach is shown below.  The colour scale shows the percentile rank, ranging from 0 (minimum) in blue to 100 (maximum) in red.  The 50th percentile (median) is in the yellow to orange range.

The coloured marks show the distribution of response times in every time bucket very effectively, but on their own, they don’t give any indication of the rate of events.  The dark line overlaying the chart is therefore needed to show the event rate.

Note that the “gothic arches” from the scatter chart can still just be seen, even with the much reduced level of detail.

This visualisation gives the sort of "aerial view" I was after.  Drilling down on this overview by various dimensions can then be done by producing a series of mini-charts.  For example the view can be broken down by process or type of event.

Applying the technique to simpler data sets

Having come up with this approach, I’ve found that it actually proves to be a really effective way to visualise less demanding data sets.

One of the most common types of data that we need to present in Performance Engineering is system response time, often presented over the course of a test run or a period of monitoring of a production system.  The information we want to convey is typically the distribution of response times as a function of workload.

The chart below shows all percentile ranks for transaction response times, in one minute buckets, over the course of a 2 hour “ramp-up” test.  The workload, in transactions per second, is shown by the overlaid line. There are about 2 million rows in the underlying data.

I think that chart provides a really effective summary of the performance characteristics for the test.

Notice how the response times improve over the course of the test, with clear steps down in response time visible as the workload steps up, particularly at the start of the test.  Those response time steps would be barely detectable in a scatter plot, but the discontinuity in the colour gradient highlights them really clearly.

Hang on.  Isn’t that the wrong way round?  Don’t systems get slower as the load increases?

This is actually a beautiful illustration of the effectiveness of Nagle's_algorithm, a clever optimisation which helps avoid network congestion (and one of the reasons the Internet works).  Nagle’s algorithm is the TCP/IP equivalent of getting an airport shuttle-bus rather than a taxi.  At off-peak times, the driver will wait a few minutes for more passengers rather than head off nearly empty.  At peak times the shuttle fills up quickly, so you head off sooner.  A few shuttle buses rather than a fleet of taxis means the roads are less congested.  Nagle’s algorithm says wait up to 100 ms before sending that half-empty packet, thereby reducing network congestion.

The transaction response times measured here involve several network round-trips.  So at low transaction rates there will be several delays of up to 100 ms.  At high transaction rates the packets fill quickly and the delays virtually disappear.

The insight this visualisation gives means it's going to be a very regularly used tool in my kitbag from now on.