DevOps: A Definition

I've begun a number of posts since my first entry ten days ago, and what I realized in each of those posts is that I had failed. Failed, that is, to define my terms.

So, I shall now define to single most important term to this blog: DevOps.

DevOps (a clipped compound of "development" and "operations") is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, releasing to deployment and infrastructure management. DevOps aims at shorter development cycles, increased deployment frequency, more dependable releases, in close alignment with business objectives - Wikipedia

Thanks Wikipedia. But what exactly does that mean?

DevOps, as defined by our dear friends at Wikipedia asks you to believe that Developers, and Operations can be combined, should be combined. Note, they don't even mention testers, project managers, release managers, program managers, content writers and a whole host of other titles you'll see in nearly every software company. Oh dear...

But, I promised you a definition and a definition you shall have.

DevOps, to me is: "The belief that Developers should own the operational collateral around building, testing, deploying, and servicing their software."

Bullshit.

Modern DevOps is simply bullshit. Not that it isn't a good idea, it's a fantastic idea, it's in my mind the only way to ship quality software, yet it falls flat time and time again. Why?

Because Developers want to write code. A good developer doesn't want to think about writing tests, or about managing servers, or about adding in telemetry to monitor live-site performance. When a software system gets to the point where it actually needs a DevOps culture it's often too bloated, too poorly structured, and too poorly documented for a DevOps culture to be sufficient. So, companies kludge other titles, and bodies, onto the DevOps process to compensate for the simple human failing that people don't think ahead.

But, but, but, you called your blog DevOps!!!

That feeling that you're probably experiencing is called cognitive dissonance.

DevOps, is a lot like "Agile" in that a lot of teams and companies "do Agile" or "do DevOps" without actually doing anything meaningful. That is, nothing on the ground actually changes.

I want to be clear. You cannot, in my experience successfully implement a DevOps culture onto an existing product. I've never seen it done successfully, please don't try. You can implement something approaching a DevOps culture onto a new, or very young, product. How you ask?

1) Build. test, and deliver often: Build with every commit, test with every commit, ship every commit. From day one the team needs to embrace this. You are not doing DevOps, or Agile, unless you are shipping code as often as your systems allow. If you're not comfortable with this figure out why, and fix that. Tests unreliable? Fix them. Builds slow? Fix it. Everything you do as a team MUST move you closer to reliable, continuous builds and deployments. Anything less and I have to ask "Why are you bothering?"

2) Let people do what they are good at: This seems like a laughable point, but it's one that's so often missed. If you have a team member who loves to write tests, and really enjoys writing tests, and mocks, let them. Don't shoe-horn them into only writing test code, as you want the entire team to at least understand the test infrastructure but if you have someone who wants to do the work, let them.

3) Communicate, often: You're not communicating enough. If at any point anyone on the team doesn't know what the rest of the team is doing, or doesn't know how to find out what the rest of the team is doing you've got problems. Leverage information radiators, and dashboards to quickly disseminate information. Nothing should be tribal knowledge, nothing should be hidden away. If any member of your team leaves there should be no need for a transition plan, as nothing should be locked in any team members head.

4) Measure Everything: From the start, figure out how you're going to collect live-site data, even if it's simply piping well-formed JSON into a tool like Splunk, figure it out now, and set the expectations that all product code will log meaningful metrics. It'll save you a massive headache later.

5) Design for failure: Netflix has open-sourced the Simian Army if at all possible configure Chaos Monkey day one. Learn to plan for failures, and outages, design your systems to be robust from the start.

6) Own responsibility: Have an on-call rotation that begins as soon as you have the first build. Even if you have no consumers, no customers, get your team in the mindset of being on call, and having the authority as the on-call to respond to problems. Tied with Chaos Monkey (#4) by the time you go live you'll have gotten the kinks out of your process, and the team will be used to owning their systems.

DevOps is a mindset, and it's an incredibly hard mindset to implement. A lot of people with cushy, well-paying jobs are not needed in an Agile, DevOps centric world. Those people will fight this change, either consciously, or unconsciously. The times are changing, and we must change with them.

From Shoes to Shiny Doughnuts

The term "DevOps" gets bandied about a lot these days, I'm guilty of it myself if I'm honest. However, the ideas behind DevOps, building and deploying rational, cost conscious software aren't exactly new. It's simply the idea that your developers should be thinking about things that were traditionally the realm of the Operations team that's "novel."

I wasn't in Ops. I was an SDET, a software tester at one of the "big four" in Seattle. A few years back they showed me, and a bunch of folks like me the door. They were moving towards a "DevOps" model, and away from silo'd Developer, Test, and Operations teams.

The results of that decision are well documented, and I won't go into them here except to say I wasn't sad to leave.

I landed soft, as most in the tech industry do, at Nordstrom. From software, to shoes, it was quite the transition. I was hired as an SDET, with the stated goal of writing the tests, and collateral, to make sure we shipped a bug-free product. Within a matter of months I was bored. Painfully bored.

Our project consisted of a suite of micro-services, fancy data-transform services, acting as a go-between Nordstrom and a third-party vendor. These were not complicated, and it was trivial to cover the majority of the major, minor, and esoteric scenarios. My biggest bottleneck was my developers, specifically how they were "deploying" code. Locally built binaries copied by hand to a suite of VMs running in AWS. Not exactly scalable.

So, I turned my coat, and moved into Ops. Specifically so that I could have more than a test pass a day.

I spent the next year and a half building out the AWS infrastructure, and Operational collateral to deploy the Nordstrom Rewards program. If you shop at Nordstrom, and you get a Nordstrom Note, you have my team, and pipeline, to thank. To this day, nearly two years after I left the team they're using nearly the exact pipeline, and deployment strategy that I built. To say that feels good, is an understatement.

Yet leave I did, and I spent the next year and a half working in a pharmaceutical data company, before jumping ship back to the "big four" and with that, this blog, with the simply goal of showing how my little slice does DevOps.

Ciao.