Botox

TL;DR

You may be selling snake oil.

What’s this?

Most of my working life has been spent in large organisations, public and private, trying to get something useful done, and I have known for a long while that in the grand scheme of things I have had essentially no success. I want to tell you why I think that is.

I don’t mean that I never had any good ideas or that things didn’t get built and deployed successfully, but simply that this made absolutely no overall difference to the real problems that plague these organisations. In fact, I think I have often added to the net total of problems. And this despite being part of projects and (the always more impressive) programmes, with transformation and digital and agile and product-centric and leadership (sometimes even thought leadership) and other important words at their hearts. And the feeling persists despite the presence of burndown charts and velocities, colourful slide decks, hearing “what good looks like” at least once a day, and dutifully attending retrospectives (some of which even included the selection of cat memes).

Something has gone wrong.

Maybe it’s me. It is perfectly possible (likely, some may say) that I am simply not good at what I do, but plenty of others are, and things really are quite excellent. There is certainly no shortage of victories being celebrated, congratulations issued for jobs well done, and employee-of-the-month-style awards handed out. But I’m not buying it. Most of the victories feel very, very, very small, the allegedly great jobs sound more like normal days in the office than anything extraordinary, and we seem to celebrate success even when we’re months or years late delivering a lot less than we promised at the start. And worst of all, I have a strong suspicion that a few years from now, most of the excellent achievements won’t be so excellent anymore, and the organisations in which they happened will have seen little improvement. But if it is me, then what follows is just sour grapes.

Size matters. Age matters.

The main criteria for being at all able to make a meaningful organisation-wide difference are the size and age of the organisation.

One way that this kind of success is more likely to come your way is if you re-define “the organisation” to mean “the specific area of the organisation to which I belong” or “my project” or “the technology I’m working on”. That’s cheating. You may well have built a platform, or some microservices, or whatever, and they may actually be great (and you should celebrate), but if people outside that bubble still think and do what they thought and did before you came along, your successes are limited to the bubble. There’s nothing wrong with that, but it’s just not that transformational.

Another path to success is to work in a small-enough organisation, say, a government department with a very limited brief, or a medium-size company that only does one kind of thing. Similarly, you can choose an organisation that is young-enough and hasn’t had time to build up serious sediment. The extreme version of this is start-up work, where there is no baggage and you can feel like a genius for a long time, almost by definition.

There are many other ways of gaming this, of course, but the kind of organisation I have in mind here is something like a large investment bank, a large government department, or a large retailer. (Keyword = Large.) Something which could have its own internal departments, pyramids within the pyramids within the pyramid, decades’ worth of legacy systems, maybe it existed when your grandparents were young, etc. These organisations are orders of magnitude more difficult than smaller ones, and that’s that scope of what I’m talking about here. This is where all the hard problems live.

Why is it hard?

If our current solutions are eventually no better than the solutions of the past, why is this? I consider three reasons to be most important.

Shut up and code

The first reason is simply a lack of knowledge (a.k.a. good explanations). Richard Feynman once said “I think I can safely say that nobody understands quantum mechanics”. I don’t think he meant that no one knows the theory or is capable of solving the equations (lots of people do and can), but rather that the theory doesn’t contain an explanation of what’s really going on behind the scenes. In quantum physics, an attitude took hold decades ago often referred to as shut up and calculate!, meaning that you’re supposed to learn the rules and apply them, but not wonder why quantum mechanics is as mysterious as it is. This has significantly held back progress in that field.

Our world is similar. Many people can design and create software, and of course we all use it constantly, but software is still a very mysterious thing. It’s not physical, but it runs on physical hardware. It doesn’t wear out. It contains abstractions. It can be copied accurately. It is (or can be) both data and procedure. It’s unlike anything else in the world and hasn’t been with us for that long, so it’s not surprising that we don’t have good instincts about it, or that we struggle making it work for us. But we spend very little energy on these mysteries, and most of it on finding patterns to apply without much thinking. And although we have a much longer experience of humans than software, we clearly don’t understand them very well either. Psychology is full of internal contradictions about human nature, and for every self-help book that claims something, there is another one that claims the exact opposite. This is not an area with reliable knowledge.

So then, in our business, something we don’t understand (software) is being made by, and for the benefit of, beings we don’t understand either (humans). No wonder it’s hard, and no wonder the outcomes are not good yet. We should, of course, be seeking good explanations about these things, but on most software projects the concerns are considerably more parochial and we get stuck fiddling with lower-level things that don’t matter as much as we’re told.

We can’t help it

Secondly, contrary to received wisdom, the generation of new knowledge does not start with observations, it starts with problems, and to solve new problems, you need creativity. Creativity is the ability to be able to think of something new (an explanatory hypothesis or theory) that has the potential to solve a problem. You then test your theory with observations. (This, by the way, is why data-driven approaches have limited returns - they start on the wrong end. You first need a problem to solve, then a creative theory, and then data might be useful. This is also why AI cannot ever be creative - it’s trained on old, often incorrect, ideas and at best rearranges them to give the illusion of creativity.)

The opposite of creativity is bureaucracy. Bureaucracy is the inability, or reluctance, to think of something new. A bureaucrat is someone who likes rules and templates, and dislikes uncertainty. When a problem arises, the bureaucrat reaches for a known solution - a process, a technology, a pattern, a methodology, stock phrases, or whatever - and applies that to the problem (and tells us that this is “what good looks like”). If he’s lucky, the problem happens to be a solved one and the existing solution actually works for that particular problem, and here bureaucracy is very useful indeed. But if the problem isn’t a solved one, or the known solutions aren’t good enough, it’s a recipe for failure and disappointment.

In an ideal world, we would choose the relative amount of creativity and bureaucracy as appropriate for the situation, but this is very difficult due to The First Law of Humans:

The bureaucrats always take over.

We have all seen this in action countless times. A small number of people come up with a creative solution to some problem (a company that does things differently, a new way to deliver software, whatever). As long as the creatives dominate, things go well and progress is rapid (errors are allowed and they’re corrected), but as the ideas spread and more people get involved, the bureaucrat:creative ratio increases exponentially, until the field is dominated by templates and unthinking rituals, and progress stops or things start going backwards.

The example closest to us is Agile. A handful of creative nerds decided to try to solve problems such as not enough work getting done and poor quality of the work. They hypothesised some solutions, many of them pretty good (and others not), and the initial iterations (error correction) resulted in refinement and new ideas. While this was limited to a small group who mostly understood the ideas, this seemed to be a good way to do things in general. But this did not (and wasn’t meant to) solve the general problem of humans, particularly the First Law, so the inevitable happened. It didn’t take long for the ideas to be taken over by uncreative people looking for a quick fix, and now nearly all the potential benefits have been industrialised away, to the extent that anyone promoting this should be viewed with suspicion. All of this happened within a smallish part of my working life. It’s a very potent force.

Unless we can generate enough knowledge to overcome the First Law, we will not make fundamental progress. Until then, we must assume that whatever we make will be taken over by bureaucrats (if it hasn’t already), so we should design it accordingly, and not design it for a world that doesn’t exist.

That’s not my problem

Thirdly, we often misdiagnose the problem. We are in the software business, so we naturally see software solutions to every problem (hammers and nails, and all that). Frequently, this mistake has been made already by the time we enter the scene, where the client is specifically asking for some technical change, e.g. a new platform or a new system to replace an old one. But in the large/old organisations I’m concerned with here, the most acute problem is nearly never the technology, and almost always something about the organisation itself. A dysfunctional organisation will have dysfunctional software, so making software improvements is at best a temporary reprieve - you can not make a good organisation out of good software alone. We sometimes sell the idea that we can make the necessary cultural changes along with the software changes, but as I mentioned earlier, this does not work with large or complex organisations.

But surely the improvements are still worth it, even if they don’t extend outside the bubble? Not necessarily. In any system, there is only so much energy available for useful work. The more of this energy we spend on technology changes, the less is available for other things, and if those other things are more important, then we are preventing necessary improvements in favour of what’s within our skill set. It’s not much talked about, but this can actually make things worse overall. When you introduce a new thing into a dysfunctional organisation on top of the things already there, i.e. your thing didn’t displace anything (or not enough), you now have a more complex organisation (all the dysfunctions from before plus your new thing). If you never experience life outside your thing, you may be unaware of this, but reality for everyone else may be worse.

Of course, in the real world, we can’t be too idealistic. We need work, and if the work demands we solve the wrong problems, we’re unlikely to say no. But maybe we should tune down our expectations and the rhetoric.

Lived experience

Without naming any names, here are some examples from my own experience.

A government department

They have as good a setup as can be expected; a recent cloud platform with small autonomous teams, microservices, self-service everything, emoji-driven development, and all the buzzwords. You would think that this is the recipe for optimal low-waste and velocity, but the feeling I got was that we were going incredibly slowly and wasting a lot of time and money on bullshit - a minority opinion. And then the pandemic hit and suddenly there was actual pressure to get things out the door, and lo and behold, we managed to speed things up, not just a little, but a lot - something like delivering in a month something that normally took a year or two. The resulting software was no worse than the usual stuff. The obvious lessons to learn are that all the Agile dogma was slowing us down, many jobs don’t need doing, we can be much faster without compromising anything, and that we knew how to do it all along. The lesson the department learned, sadly supported by most of my colleagues, was that this was hard work and that they didn’t want to ever do that again. This is clearly absurd. Not only should work be hard, but ignoring the literal proof that you can go significantly faster is unforgivable. But happily the bureaucrats were swiftly back in charge, and the organisation safely back in the taxpayer-funded slow lane. Ah, you say, the temporary acceleration was only possible because of the digital platform and microservices and all that. If this were truly the case, then the analogue people looking after the underlying legacy estate with relational databases and dreaded Waterfall would have failed. But they managed to go much, much quicker too. They did it, like we, by bypassing pointless bureaucratic activities.

Conclusion: Modern approaches do not solve the problem of slow delivery.

The cloud platform and its microservices replaced most of a monolith portal. If you believe the monolith bad - microservices good mantra, then this should be a good thing, and to start with it looked like it was. But over time the number of teams increased and the number of microservices increased even faster, in other words the number of dependencies grew exponentially. This, plus the lack of centralisation, then resulted in a lack of coherency of the services operating as a whole, i.e. no one was designing the overall. Later, we needed to make changes to concepts that affected most of the estate (most teams and most microservices), and this turned out to be a near-impossible job. There were just too many teams and services to shepherd, and they shared far more assumptions than they thought.

Conclusion: Modern approaches can increase dependencies, leave design gaps and make changes harder.

A retailer

Hundreds of ancient systems in lots of different technologies, managed by many suppliers. A general lack of knowledge and maintenance. Some in-house datacentres. A typical situation. Decision-makers concluded that modernising the technology was the most important thing, which of course means digital cloud platforms and all the buzzwords. Quite apart from the obvious fact that the sheer number of systems and technical dependencies makes this impossible in anything much less than 20 years, the main problem here is not that the technology is not modern enough, it is in the general culture of the organisation (too much outsourcing, not enough in-house knowledge, SOWs needed for everything, etc.). Modern software will not fix any of these. And so it’s turned out - modernising has been agonisingly slow and no cultural changes have been made outside of tiny bubbles. Most of the energy this programme has consumed could have been spent doing much more useful non-technical things.

Conclusion: Attempting to solve the wrong problem is harmful.

What to do?

I am as clueless about software and humans as anyone else, and I certainly don’t have a template solution to the fundamental problems, but I do believe there are things we can do to operate more successfully in this ocean of ignorance. The essential thing is to develop the capacity and habit of thinking about problems in as many different ways as possible. The main obstacles to this are consensus and dogma, i.e. institutionalised bureaucracy. It does not matter if the dogma is a currently-fashionable one or something now-discredited, dogma always means that people have stopped thinking and creativity has died - it’s pure bureaucracy. Agile is not better than Waterfall when they’re both in the form of dogma. Nor is any other method or philosophy. The real problem with Waterfall wasn’t so much the underlying ideas (although they made incorrect assumptions about software and people), but that it ended up being implemented and enforced by Waterfall bureaucrats. I have no doubt that creative people can make it work successfully. And now we have exactly the same problems with Agile, which also makes incorrect assumptions about software and people, and Agile projects are now largely run by Agile bureaucrats, blindly going through rituals, forever being thwarted by techies refusing to estimate anything because it’s “not agile”. Layered on top is an increasing number of add-on religions: user-driven, customer-centric, data-driven, value chains, etc. etc., each with its own set of scriptures that must be adhered to. And along with that we have all kinds of job titles that are, frankly, not necessary most of the time (too many chickens in the old Agile story).

To progress, you need to forget all this, it’s not very useful. Pick a process if you must, but don’t expect it to make or break the project. Pick a technology, but don’t expect it to ultimately be much better than the old one. If you do this, you may find some space in your head to think about things in a different way, to prioritise the truly important things, and maybe come up with ways of making more meaningful progress. So what should you be thinking about, if not the things that everyone agrees on? In general, don’t limit your thinking to technology or software, even if that is your work domain. More specifically, find some abstraction or a lens that you can apply to software and other things like people and processes.

My favourite lens is dependencies. You can use them to explain why microservices can be worse than monoliths, why we always under-estimate, why concepts are just as important as code, why there are only 4 ways to deal with broken dependencies, why most changes just shift problems from one form or place to another, and many other things besides. But this is too long already, so I’ll cover them in a separate post.