The Rosser Controversy

When I joined Rackspace in 2013, I joined a group called DevOps Automation. It was a premium managed-service tier where Rackers ran client environments using Chef. We sat in a Slack shared with the client, we managed their infrastructure, and the whole pitch was that we’d make their day-to-day easier, including deployments.

By the time I got there I was pretty deep in the Chef community. I’d written some popular OSS cookbooks and a handful of libraries and tools around them. So I came in with strong opinions about how Chef was supposed to be used, and I came in genuinely excited about the role.

The Stacks

Here was the setup. There was a product team whose job was to build baseline cookbooks for popular tech stacks at the time. They called these stacks. A Ruby on Rails stack, for example, or a Magento stack. The idea was that the support team, which is the one I’d joined, would take a stack the last mile to fit a specific client’s application. The product team built the prescription, support delivered it.

It didn’t work. The prescriptions were too narrow. A stack would commit you to Apache when the client was running Nginx, or commit you to a specific deployment pattern that the client’s app didn’t fit. And because the stacks were built as cookbooks rather than as something more composable like custom resources or providers, working around them was almost always worse than starting from scratch. You’d end up with a client cookbook that pulled in a few recipes from the stack, then a wall of inline attribute overrides to undo the assumptions you didn’t agree with, then your own logic stapled on top. The result was a mess that was hard to read, hard to maintain, and hard to hand to anyone else.

The deeper problem wasn’t the cookbooks. It was the philosophy. Instead of teaching the support team to be proficient in Chef and acknowledging that every client environment was different, the org pushed an “easy” prescriptive solution on the people closest to the work, and then shrugged when the prescription didn’t fit.

Raising It The Normal Way First

I tried the obvious path first. I gave the product team direct feedback. I made the case that the prescriptive-stack approach was working against the support team more than it was helping us, and that we’d get better outcomes if the building blocks were more composable and the team was trusted to use Chef properly. It didn’t move.

So I started working around it. I began teaching some of the more senior techs on support how to use community cookbooks directly, how to think about real customizations, how to build something maintainable instead of layering overrides on top of a stack that didn’t fit. That kept growing. Other senior people on support started doing the same.

At some point we realized we needed somewhere to put the public community cookbooks we were maintaining for client work. So we made a GitHub organization for it. We were frustrated enough at that point that we named it The Galley and used a skull-and-crossbones for the avatar. It was a joke, but it wasn’t really a joke. The flag said exactly what it looked like it said.

We ran like this for several months. The stacks became less and less relevant to what we actually did day to day. Our direct managers had a general sense of what was happening. They didn’t actively bless it, but they didn’t stop it either, which in that culture was its own kind of endorsement.

If you’re going to push hard, push through the front door first.

The Meeting

Eventually, myself and a few other senior support engineers got pulled into a meeting by a team adjacent to the stacks product team. They were working on some new IaC ideas for provisioning and managing cloud customers (something that, in retrospect, looked a lot like modern-day CloudFormation) and they wanted our feedback.

Somewhere in the conversation, it came up that the support side had almost completely diverged from the stacks model. That was news to the room. The product team’s understanding of how their own product was being used was several months out of date.

My manager later thanked me for the “week of meetings” he had directly after that one. Those meetings ended with the model getting overhauled. They brought me and the other senior support engineers from across the teams into what was left of the stacks product team. They replaced the leader of that group with Brint O’Hearn, who I’m grateful to as someone I learned a lot from. And they reshaped the offering into something much closer to what we’d been doing on the side.

What I Actually Took From It

This whole thing became known as the Rosser Controversy, coined by our GM at the time.

What it actually was: a product team built the wrong solution, the people closest to the work routed around it, and eventually the org noticed and corrected course. The interesting part of the story isn’t that we built The Galley. The interesting part is what the org did once it found out.

There are a few things I take from it that have held up.

Treat your internal users like customers

The product team probably could have caught this much earlier just by paying attention to the volume of questions and concerns coming from the support side. They almost certainly read the eventual quiet as things getting better, when in reality it was the sound of people giving up on the product and going their own way. If they’d verified instead of assumed, they would have found out months earlier.

The deeper issue underneath that is they never really treated us like customers. We were the people who were supposed to implement what they built. That framing locked them out of the most useful information they had access to. A real customer relationship has feedback loops in it. Account check-ins, success metrics, an honest accounting of where the product is and isn’t working. None of that existed between the product team and support, because internally we weren’t customers, we were implementers. When your only model for the people downstream of you is “people who execute what we ship,” you stop being curious about whether the thing you shipped is working for them.

If you’re building internal tooling or platforms, the question “are the people I built this for actually using it, and how is it going for them” is one of the few that doesn’t lie to you. But you only get an honest answer if you’ve set up the relationship to give you one.

Raise it normally first

I gave the product team direct feedback before I started routing around them. That mattered later, when the dust settled. Nobody could honestly say I hadn’t tried the normal channels. If you’re going to push hard, push through the front door first. Not because the front door always works, but because when it doesn’t, you want the record to be clear about what you did before you stopped using it.

Leadership treated it as information, not insubordination

The reason this story ended well is that at every layer, leadership treated what we were doing as information about the work, not as a breach of process. That happened at two layers.

Our direct managers knew enough about what we were doing to stop us if they’d wanted to. They didn’t. They let us run, and when the bigger meeting happened, they didn’t throw us under it. The version of this story where direct leadership panics the moment it gets visible looks completely different and ends with people getting reprimanded for “going around process,” not promoted into the team they were going around.

The second layer was the leadership above the product team. When it became clear the support team had built a parallel system, they treated that as information rather than as insubordination. They pulled the people who’d been doing the work into the room and rebuilt the offering around them. The version of this story where the org closes ranks around the original product team and disciplines support for going off-script is one I’ve seen play out in plenty of other places. It’s a much worse story.

The throughline is the same at both layers. When leadership saw practitioners routing around an internal product, the response was curiosity about what they knew that wasn’t being captured, not enforcement of the existing structure. That instinct is what made the difference.

None of this would have been necessary if the product team had been in a real relationship with us the whole time. The Galley existed because there was no honest channel between the people building the product and the people using it. The reorg fixed that, eventually, but a healthier setup would have meant the product never drifted that far from the work in the first place.

If you’re building something for another team inside your own company, they are your customers. Not your implementers. Not your distribution channel. Customers. Treat the relationship like one.