Servant Leader, Helicopter Parent

Why is developing software so slow now?

Provocative statement, so let's unpick. Maybe I'm getting more impatient as I get older, but the pace of modern software development feels glacial. Honestly, why do simple app features or new API endpoints swallow half a team for three or four weeks at a time? And the important part, now. Half the team for a month used to be what I'd quote for building an entire new thing from the ground up, back when we used to quote development times out of nowhere in the misguided belief they'd be remotely accurate.

I think some of it is that we're struggling to separate communication and support from effective communication and support. Not so much servant leadership as helicopter parenting.

I like to present these things as long and seemingly pointless rambles that eventually arrive at some sort of destination, slightly light on any obvious reason as to how they got there, so let's do that.

2006: Buffer underrun error in requirements

Let me start with a counter-example from my early days in software development. Back then I would mostly get my tasks from a Senior Project Manager or similar who'd tell me, "this is a critical client project at the highest levels of visibility, it is due by this unrealistically early date (subtle implication: this is somehow your fault)" on a Monday morning and swan off to client meetings to promise more unrealistic deadlines. I'd send a mandated status update e-mail twice a day. If things went badly I was told to sort them out myself. Then a couple of days before release the Senior Project Manager would come and tell me I'd done it all wrong and there would be a mad late-night scramble to correct all the problems before the deadline.

This was pretty much the first six years of my professional career. At one point I got to work for a better manager, whose kickoff messages would be more along the lines of, "this is a load of old rubbish and most of the urgency is self-inflicted but we've still got to do it by this date" and wasn't really bothered about status updates but did seem to care about fixing things that had gone wrong. There were far fewer last-minute corrections under his stewardship.

Despite working largely by myself, I delivered an enormous amount of software in this environment. Also despite also doing a bunch of data processing and GIS work in between. And despite several clients having a direct hotline to my desk. Even despite me only spending about 90 minutes a day working and the rest on Pistonheads because I am at heart very, very lazy and prone to distraction. Well - not just that. Because our development process involved a lot of expensed mileage and not much team involvement, there simply weren't enough requirements to fill more than 30 minutes of my day. Buffer underrun error.

At the time I was getting frustrated by this, because it turns out you can only waste so much time on web forums before you get bored. Also I had idealistic notions about being a programmer and felt that maybe if I worked for a company with a better process I'd have more than 90 minutes of programming a day.

2013: The high point

After a few diversions I ended up part of that slightly over-zealous movement that brought small-a agile development to wider consciousness. The wild frontier days when it was all lived experience and first-hand knowledge of the hills you actually needed to make a stand on for it all to work. For those whose experience is the binderware of modern capital-A Agile and its charts featuring the Circle Of Customer Centricity and the Mysterious Diamonds Of Design Thinking, I apologise. We didn't think it would turn out like that.

I digress. The point I'm slowly getting to in an article about why everything is slow: the teams we had in this era were fast. We were seeing somewhere between 4-6 hours of productive coding time per day and making a lot of things happen. At one point with merely a couple of dozen developers we had a mobile energy comparison app, an in-store sales platform, a white-label version of our software and a rewrite of our platform all happening at the same time, with stuff flying out the door at high speed and high quality.

The one challenge we did have was that nebulous area called "stakeholder engagement". Or in plain English, getting people who wanted The Things to come along to meetings in which current progress on The Things was discussed. When this didn't happen, we reverted to my earlier experience where two days before release it would be decided that everything was all wrong and needed to be rewritten. However, this wasn't a case of redoing maybe 12 hours of actual development while putting the Pistonheads habit aside for a bit. The volume of work was such we couldn't remake it from scratch in a single late night.

After a few too many fractious discussions and last-minute rectification sprints (which used to make scrum masters very angry indeed back before the folder-brandishing consultants turned up) we compromised on someone turning up to every demo so at least they were only kicking apart the last two weeks of work. And kick they did.

These fortnightly kicking sessions were interesting, not least because I'd moved to architecture by this point so would often be playing stakeholder. Sometimes they would be packed with volley after volley of fundamental questions about how the most recent progress operated and was built. Sometimes they'd be 30 minutes on the spelling of a word that was only part of placeholder text, or the anti-aliasing of a work in progress UI element. I swear, "he doesn't like the pixels" was a meme for two solid years of my life, and it only stopped being a meme because I moved to other pastures. For all I know back in West London it's still being used as the stock phrase to describe a meeting in which all the feedback is about inconsequentially minor distractions.

Hold this picture. I'm going to come back to the pixels not being sufficiently liked.

Now: Inexplicable slowness

This was the point at which I made my move to off-the-chart enterprise insanity in the hope I could play a part in fixing it. And suddenly everything was slow. Initially I blamed this on the development process: a Project Manager would give me an unrealistic deadline, and ask me at least 12 times per day how much this was costing, followed by a similar question of how much this would be costing if we could pretend half the team was on the "TRAINING _ SELF DIRECTED" project code and the other half claimed to be working on a project for those people over there, who have a big budget, don't check their internal invoices too closely and probably smell a bit of TCP anyway so don't feel too guilty about this small degree of internal fraud.

But we fixed the process. I've written quite extensively about the pain and frustration incurred while fixing the process. And even after all of that it was still frustratingly slow in a lot of areas. Team XP was a high point but as mentioned in that article we struggled to scale the approach elsewhere. More frustratingly, the more we tried to drive communication and stakeholder engagement the worse things seemed to get.

Besides, I exited at about the point the SAFe charts started going up on the wall and now have one of those wonderful trust-based, collaborative and experimental environments I'm always going on about. I'm still annoyed how slow things are. It's just I have nobody up the chain to blame for not fixing it or making it worse now.

This isn't just a Kimber Dissatisfaction Cycle. We are doing all the right things and yet in many places I feel I was getting more done with less technical debt back when I was spending most of the day browsing Pistonheads. This is not a boast about my skill level either. If I do actual coding in this environment, I'm just as frustratingly slow. Slower, even, because I'm so far out of it I may as well be introducing my team to this great new band I've just heard of called Hawkwind Zoo.

What's happening?

Powered by the idea that communication is good, stakeholder engagement is good, and leadership should be always available... we've gone too far. Not so much servant leaders as helicopter parents: ever-present and suffocating in our influence.

Space to Create

In that 2006 scenario, people were so uninterested in the low-level details that some of the less good managers treated not knowing anything about the mechanics of building software as a badge of pride, the signifier that they were a cut above the mere peons who did the work. But what it meant for me as an individual contributor was space.

I don't think I even used the word "spike" in a software context until about 2014 - if I needed to figure out whether a framework made sense for me or try a new approach I just put, "spent the day researching whether we can build accordion controls in our existing tech stack, currently still on track" in my status report and that was it. Nobody gave me a lecture about velocity or the importance of making sure all work is feature-focused. I never had a design pattern "helpfully" imposed on me.

Not that 2006 was perfect: if we trod on the toes of the core development team or tried to use something new and exciting like jQuery then there would be an angry missive from head office in the inbox and I'd get a bonus 30 minute meeting about how I'd made a grave error of judgement and should have known better and everyone was very disappointed in me. But once I'd built up a decent idea of what was safe to do, I existed in a world where I was given problems and it was up to me how I solved them within the constraints given. As long as those magic words "on track" were in that twice-daily status report, nothing else mattered.

This is why I reckon we peaked some time around 2013, at least on my journey. The combination of committing to an amount of stuff the team could get done in a couple of weeks and then being left alone to do it before submitting it for feedback was incredibly powerful. We also helped ourselves a lot at SDWS by treating sprint backlogs as immutable - this flies in the face of what a lot of agile coaches will tell you, but we found it acted as a heavy disincentive to start working on anything which wasn't still going to be important after two weeks had elapsed. (A topic for another article, but the "today's fire" school of scheduling is immensely harmful to a team's ability to concentrate and get things done.)

The key part is that even in the technical leadership we allowed people space. (I had to learn how to do this, and there were a few incidents where I went off the handle 2006-style upon someone suggesting a technology or design pattern I didn't like. I got better.) We held our teams to account on improving flow and throughput over the course of a project, and would step in if a sprint got drastically off track, but even the internal team decision to use a completely untried open source content management system was upheld - despite us ending up having to rewrite a significant chunk of it ourselves.

By contrast, the helicopter-servant leader would have been constantly interfering, and no doubt said team would have spent more time on "why don't you just use Sitecore?" discussions than it took them to make their chosen CMS usable. Yes, back in the day there were times that someone didn't like the pixels. The important bit is they only expressed that pixel dislike every other Wednesday, rather than micromanaging a designer on how Photoshop should be operated to make the pixels likeable. When faster feedback cycles were necessary, we trusted the team to ask for them.

Not only that, but the mechanics of having a half-hour feedback meeting once a fortnight provided a powerful forcing function. You know I mentioned some meetings being a festival of pixel-induced disappointment, while others asked serious and fundamental questions about what we were building? That's because a short and somewhat infrequent meeting forces you to prioritise. You don't have the luxury of labouring every small point it's possible to labour, from the architect's dislike of the mapper pattern to the CEO's annoyance at a bit of missed punctuation - especially if the review shows pages taking 25 seconds to load. It's not just that the team get feedback, they get the most important feedback. The stuff that really needs to get done, not the small things which swallow time and yet don't improve the software by much if at all.

But it feels this is being lost. Whether it's overbearing cost control and standardisation in the enterprise or an overwhelming keenness to be helpful in smaller organisations, teams today suffer from constant interference in what they're doing. And... it's not helping. Our leadership needs to get back to a world in which we give people problems and priorities, explain the constraints around them, then leave the team alone to figure it out. In particular, trusting teams to let us know when they need feedback or guidance and making ourselves available to give it, rather than foisting it upon them adhoc. In other words, support. And not helicopter parenting.