How strict governance causes rather than prevents shadow IT

Shadow IT is when something's happening in your organisation that ops ought to have control over, but don't. Common examples include personal laptops connected to the network, software frameworks covertly installed on servers, corporate mail sitting on personal phones, internal code published to Github, and services running on cloud subscriptions that only a handful of people know about.

The funny thing is, the more policies a company has put in place to explicitly prevent shadow IT, the more of it that it has. I'm not talking in the sense of, "if there are no laws then nothing can be illegal"; I've worked places that were extraordinary laissez-faire and yet their operations team had pretty complete pictures of everything that was going on.

Why this happens is simple. People aren't employed purely to follow rules. They're employed to solve problems. If you block the obvious and optimal way to solve a problem, they won't give up; they'll find a less obvious and less optimal way. Overbearing rules just mean they won't tell you about it when they do. Too much governance ends up giving you the illusion of total control when the reality is a total lack of it.

Imagine, if you will, two J Random Programmers. They work at different companies but both are facing the same problem; their company's official Linux image is Debian Squeeze, and it's missing the recent kernel features and up-to-date packages they need to do their job.

Permissive Governance: Jane

Jane Random Programmer goes to her ops team and asks if she can have Ubuntu 15.04 on her production servers instead. Her ops team ask what she needs it for; she complains that the packages and kernel of the company's official image are hopelessly out of date. Ops say that they'd much prefer to use Debian as that's where their experience is - but would an upgrade from Squeeze to the most recent Jessie work? They've been waiting for an excuse to get some servers updated. Jane agrees; anything with a recent kernel is fine. Problem solved, everyone leaves happy.

Strict Governance: Joe

Joe Random Programmer goes to his ops team and asks if he can have Ubuntu 15.04 on his production servers instead. His ops team tell him there's a company policy that all servers must run Debian Squeeze, although there's a long-outstanding plan to upgrade to still-obsolete Wheezy, which might happen before the heat death of the universe if they ever stop having to deal with constant firefighting.

Joe goes away disappointed, but then he remembers his team have an account on AWS that got set up with someone's company credit card. He checks with the cardholder and spins up an EC2 instance with his preferred version of Ubuntu, and gets coding. The service he's working on gets deployed.

A sprint later, Joe's working on the web application which will consume the service running in AWS. When he deploys, he can't connect: firewall issues. Joe trudges down to ops to ask if he can have some ports opened up.

"It's company policy not to open non-standard ports outbound to the public Internet. You can't have those rules."

Joe ponders bodging something together to abuse a port his ops team consider standard, but when he mentions this at the morning stand-up the team decide there's no reason why the web application can't go in AWS as well. They just need to buy their own domain so they don't have to fight operations over yet more rules.

Now that the solution is wholly cloud, they even start up a build server there, to be able to do continuous deployment without the network getting in the way. That then causes trouble connecting to the company's source control server... but that's okay, one of the developers has a private GitHub repository the team can use. All problems solved. Sorta.

The Conclusion

Jane's company had loose policies, which they were prepared to compromise on. What's the result of this lax governance? Their ops team has a box running a slightly newer version of their standard operating system, which they know about and have documented.

Joe's company had strict policies. They have a complete infrastructure running with no visibility to the ops team, and no centralised billing. Will that domain get renewed on time? Who knows? Concerns like information security, monitoring, backups and disaster recovery are down to whether the developers had enough server-side experience to remember those things. Even the source code is off-site. It's a bunch of disasters waiting to happen; the very thing all that governance was supposed to prevent!

This may sound contrived, but it's a real picture and it's happening every day in organisations across the world. Let me point this out: the above are subtly anonymised but otherwise genuine examples from my experience.

Developers are trying to get their day job done. Managers have company credit cards. Executives will defend almost any supposed gross misconduct if it resulted in getting some (apparently) working software on time and to budget. The days when operations teams could act as sole gatekeepers to infrastructure are over, and that has big implications. Your internal infrastructure and datacentres are not monopolies, they're competing in an open market. That means you need to collaborate, compromise, and work to help people understand your needs and difficulties rather than create inflexible edicts.