The Contemporary Doomsday Book is out!

"Reading is equivalent to thinking with someone else's head instead of with one's own" - Arthur Schopenhauer 

So the long awaited library consultation report to the neighbourhood scrutiny committee was published on Thursday. It is a mammoth 164 page document of 9MB in size, which is why I've been somewhat quiet. Unsurprisingly, I've been doing what all good library users do. I've been engrossed reading it before the weekend. For what it is worth, I am shocked at the poor quality of the analysis and together with the rest of the campaigns around Manchester, I am shocked at the rhetoric.

For those who've got to know me over this campaign period, they will immediately know I am a numbers guy. Whilst a lot of people tend to bracket "lies, damn lies and statistics" together, the truth is that statistics do not actually do the lying. It's the interpreters of those statistics that do. This document is an example of just such a thing. Of course, being a guy who's not afraid of an intellectual 'punch up', I am going to give my reasons bit by starting with the below, but suffice to say that even the work submitted by the Save Burnage Library campaign is more solid than the MCC analysis conducted here, even without MCC giving the campaign all the information it needs.

Viability and Needs Analysis (or as I like to call it "This analysis isn't Viable, we Need better analysis!")

Appendix 7 of the document is probably one of the worst examples of a document that's full of statistical biasing I have ever come across. Here are some reasons as to why (the list of things wrong with this report and the corroborating data is so large, I'd never get it out in one go, hence the splitting of it into chunks).

The report consistently refers to taking account of the Charteris enquiry. Note, these variates (the columns across the top) are not actually specified within Charteris at all. 6.17, 6.26, 6.27 show the sorts of evidence that Charteris would have expected to occur within an effective assessment of local need. Manchester City Council's assessment (Appendix 7) is shown below (click the image for a larger version)

fig 1 - MLIS Viability and Needs Analysis (click to zoom)

This analysis was the major evaluation cited at the beginning of the report, which selected the six libraries set for closure. Aside from the fact that the equality impact assessment is not cited here, yet this analysis was stated as the main reason for choosing the 6 libraries, the things that also struck me immediately were the fact that they were using a combination of:

• Ranking system (notoriously inaccurate, since there may be a massive disparity between actual values, but they will not necessarily rank that far apart. Hence, some will 'over perform' overall)
• High/Low ranking - Where the values were ranked differently across the board. In itself this isn't a problem until you consider...
• The combined score is simply a sum of the ranks and a selection of the lowest ranking performers by this combined value. Given the 'flipping' between ranking high/low based upon this can be unsound.

For those that have a solid analytical background, this should immediately ring alarm bells and send you looking for more. So I got my spade and started digging.

The ranking for Total library visits, Total active user rank and PC usage were all based on the aggregate results per library. What I mean by that is in a scenarios where two Districts A and B, which have a catchment area of 30,000 and 10,000 people respectively, have total library visits of say, 2,000 and 1,000 respectively, District A will rank higher than B, despite the fact that library usage is 6.7% and 10% respectively. This means the council's statements about considering library usage in a catchment, are simply false. 

I constructed a correlation matrix between the factors on the columns at the top of their analysis. In simple terms, a correlation matrix is a grid where each cell effectively tells you how similar the factors along the columns are, relative to the factors along the rows. High positive or negative numbers mean that the factors are very definitely related to each other. High positive numbers mean that as a variable (in this case rank) increases so the other variable increases. A negative number means if the ranking increases, the other value decreases. This sort of matrix can tell you a tonne of stuff, including how similar columns actually are to to each other The assumption that MCC put to us in this report is that it was a fair assessment and as such, 'Total library visits', '...user ranking' and PC usage (in hours) were not correlated to the combined score (which was the important factor). The image below shows my correlation matrix snipped from Excel. Again, click the image to expand it.

fig 2 - Correlation matrix (click to zoom)

Correlations and anti-correlations? Who are they? 

OK, a lesson in these 'correlations'. The darker the red, the greater the positive correlation. The darker the blue, the greater the negative correlation (aka anti-correlation). Any column containing any of the darkest reds can pretty much be considered to be identical. To illustrate that, look along the diagonal from top left to bottom right. The first element is examining 'population of catchment area' long the top against itself along the left. You'd expect that to be the same and sure enough, it has a value of 1 (i.e. 100% which is as high as you can go in correlations and that means they are 100% the same). Anything over 50% is considered very strongly to hint at the existence of a 'dependent factor' and anything over 80% is pretty much the the same variable, just in a different form such as a dependent variable. 

For a really basic example of what a dependent variable is, the distance a car travels in an hour is a 'dependent variable' based on the independent variables of its speed and time. So in a constant two hours, a selection of cars travelling at 10, 20 30 and 50mph will travel around 20, 40, 60 and 100 miles. As the independent variable of speed increases, so does the dependent variable of distance. 

You can rearrange most equations to change the dependent variable. For example, if you have been given distances travelled in 2 hours, you can find the average speed by dividing the now dependent variable of speed by the independent variable of distance. In our analysis, if the matrix is what is known as 'symmetrical' (i.e. if you 'flip' the matrix along the top left to bottom right diagonal, does it look the same?), then you can rearrange the independent and dependent variables this way. 

Basically, I was looking for those tell-tale signs and in this case, the dark reds give me that.

Well, what do you know[?]

The important things on the above correlation matrix are as follows:

1. Population of the library catchment area is strongly correlated with the total number of library visitors, Total active user ranking, PC usage and building performance.
2. Population of the library catchment area, Total number of library visitors, Total active user ranking and PC usage are basically the same variable! This is corroborated because of the clump of red right in the middle, which shows the variables are also almost identical to each other, never mind just the catchment area population. For example, you will have a high active user rank if you have a high number of users that's basically common sense. If it looks like a duck, walks like a duck and quacks like a duck...  
3. The impact of this same variable on the results effectively weights the analysis massively in favour of the larger library catchment areas and wholly against the neighbourhood libraries. 
4. Neighbourhood libraries then get a further kick in the groin because they don't have access to venues to hold sessions or events. This causes a selection bias in the sample, since you are automatically excluding libraries from the top rankings which can't host events due to H&S legislation.or otherwise having no room. Note in the link in this point it states "If the selection bias is not taken into account then certain conclusions drawn may be wrong."
5. This is corroborated when you look at the correlation between the combined score and the population of the library catchment area. This is the kick in the teeth, an 88.77% correlation! You pretty much cant get any closer than that in this sample size.

It gets worse than that. The first principles of the method are fundamentally flawed, so consequentially the resulting analysis is fundamentally flawed. In data processing, this causes us to go GIGO, or "Garbage-In, Garbage-Out". In this case, nothing says that more than the building performance correlation. To quote the MCC document at the bottom of that column:

"rank 1 = low score means building has had high level of refurbishment / new build so higher viability as less work required"

Well, for those who work in the building trade or have some understanding of civil/structural engineering, they know this simply isn't true. The way this ranks libraries means that if a library is undergoing ongoing refurbishment work, then it will have a higher ranking due to more recent work. Low maintenance buildings will have a lower ranking. As a result, it is possible for 'money pits' (buildings which have had a whole host of maintenance work conducted) to rank higher than solid, low maintenance buildings, which if we look at MCC's premises cost figures (Appendix 6), show us that the top 10 lowest maintenance libraries have been:

1. Barlow Moor
2. Northenden
3. Brooklands
4. Burnage
5. Hulme
6. Levenshulme
7. Moss Side Powerhouse
8. New Moston
9. Newton Heath
10. Withington

So even MCC's own report has contradicted itself (as these rankings are completely different to the Viability/Needs analysis rankings in Appendix 7). However, you could argue that the ranking should take into account the size of the building, which is technically correct. So shouldn't the 'Green Quadrant' in the middle of Appendix 7's viability and needs assessment also take into account catchment area size? You know MCC, like maintenance cost per square foot like total library active use per head maybe? Does MCC see the parallel?

This is yet another example to me of MCC conducting a poor analysis. It brings back into question statements made previously by cllr Sue Murphy about how they planned for closing lunchtime opening hours and were surprised by the drop in service uptake. From this analysis alone, I am not convinced they have the statistical know-how to make that sort of decision in the first place and come to that conclusion. 

So what should the table look like?

MCC need to do a lot more work, especially around the assessment criteria. However, even if we allow the rest of the variables to exist ranked as is, we need to adjust by the 'Green Quadrant' population to get a fair comparison for smaller catchment area. If we do this, 13 libraries change rank. Two of the libraries slated for closure come out of the relegation zone altogether to be replaced by two more. This adjusted analysis is shown below and the blue bars illustrate MCC's previous relegation zone (since Barlow Moor Library's funding has been withdrawn and Hulme Library will move to the leisure centre). Basically, from this alone, MCC's method is unsound.

fig 3 - Quadrant adjustment for population catchment area size (click to zoom)

*** EDIT: 11/05/2013 ***
In theory, when you see a variable have such a massive influence, this needs to be adjusted across the board.  The Save Burnage Library campaign reports never claimed a requirement for blanket coverage across the city (as stated on page 21, point 8.5). The adjustments were conducted to be able to compare like-with-like based on catchment population size. Indeed, I would go so far as to say column 1 in Appendix 7 the MCC report should have no direct bearing on the final result, since the information it brings is implicitly included in the analysis as part of the adjusted columns mentioned above. Following on from the fig 3 above, to turn this into a table:

fig 4 - The change in ranking when direct effects of catchment area size are removed.

In total, some libraries have moved 11 places up the rankings! Miles Platting (11), Crumpsall (6), Brooklands (5), Northenden (5) and Fallowfield (4)  in particular have moved significant steps up the ranks. In this arena, of the two libraries in "Newton Heath and Miles Platting" ward, they are closing the one which is better used per head of population.

*** END OF EDIT ***

Summary

As a numbers guy, I am shocked! Truly shocked! Reading along the bottom row (or rightmost column) to determine what factors MCC really used and to put it in political rhetoric that MCC should understand, the low correlation of the combined score with any other factors aside from the positively correlated ones, are truly saying to the people of Manchester, when it comes to libraries:

• MCC don't care about our young people
• MCC don't care about local deprivation
• MCC they don't care about libraries existing within 1.5 miles (that was just a decoy) and 
• MCC don't care about how much each library visit costs

t's all about the catchment area. I am inclined to believe others who say we'll be subsidising the more affluent areas of the city. I have never ever come across a document from an allegedly professional organisation which has produced a statistical model to fit the conclusion they want. If ever there was one, it is this. I am absolutely disgusted with it!

I call upon the neighbourhood scrutiny committee to return the consultation report for further work on the grounds that from this alone, MCC is not setting out on it's own success criteria. There are a number of other areas which I will address over the next few days.

Take note MCC, I have a fine toothed comb and I work quickly. 

EA