In praise of simple tools
It’s been six months since I left AWS, and I’m now feeling detached enough to write about some of the things I liked (and eventually maybe, didn’t like) about working there.
In his now infamous screed about Google, Steve Yegge pointed out that Amazon’s tooling sucks. That was in 2011 and I’m sure it’s a lot better a decade later. I used it for almost 3 years and thought it was mostly “OK, not great.” But there is benefit to that. Amazon’s tooling and organization very much reflects what Amazon is and also what it’s not.
Over the years I’ve realized that tools, work structures and organizations need to complement each other. This is something of an addition to Conway’s Law, that talks about org structure and product structure, but it’s not much of a leap to see that the capabilities of tools selected will also tend to match the org structure. A decentralized organization does not need centralized tools or an organization devoted to centralized management. The very existence of any such tools would be subtractive. An organization with high profit margins and a need for centralization of complex initiatives (like, maybe, ones that have to deal with DoD purchasing requirements and sell toilet seats for $600, or… I dunno… Microsoft) may require complex management tools and people to run them. An online retailer with sub-1% margins (like, you know, Amazon pre-AWS) can’t afford these and needs to find another way.
A few years ago my friend Corey Quinn gave a talk that addressed some of these issues. It’s worth watching. The big takeaway is that my environment isn’t your environment. Constraints are different. Taking the tooling and process that works at a high-margin company like Microsoft, Google or Facebook and applying it to a low-margin business is a recipe for disaster. (Corey refers to this as “cargo culting” the idea.) No matter how much the idea makes sense in one environment, it may be unaffordable or otherwise destructive elsewhere.
While Yegge is right about Amazon’s tooling, it’s one of the things I’m missing as I’m currently on hiatus and exploring other possibilities. There is an almost evil genius to the idea of providing people with tools that are adequate, but don’t allow them to be distracted by extraneous features and capabilities.
I like distractionless tools. I write using Emacs and LaTeX. It’s an ancient but still completely adequate and completely universal mechanism for putting together docs. I’ll grudgingly copy and paste to other formats because the organization demands it, but I do better work with the simpler tool and I think that’s generally true. There is little that a good document needs that can’t be addressed with the the basic formatting available in LaTeX. Some people of similar-mind prefer Vim over Emacs. They are Wrong, but less wrong than people who use Google Docs, Quip, or (angels and ministers of grace defend us!) Microsoft Word.
The reason I find Emacs/LaTeX to be so liberating is that it doesn’t have all the “cool” features of those other tools, so I’m never tempted* to use them. I’m able to be more productive, because there’s nothing to distract me from the core purpose of putting meaningful words on the screen. Anybody reading a doc produced that way is going to waste less time as well, with fewer distracting boxes, shaded areas, fonts, highlighting, and especially no pie charts. Using these simple tools forces me to write well because I can’t rely on any other elements in the document to drive home the point.
Pie charts
— Edward Tufte (@EdwardTufte) February 11, 2021
John Tukey “There is no data that can be displayed in a pie chart, that cannot be displayed BETTER in some other type of chart”
Jacques Bertin: “Pie charts are completely useless”
ET: “Pie charts subtract from the world’s knowledge”
One of the reasons I could use simple writing tools is that with the exception of a monthly formal report for each area, we communicated in very straightforward documents. There were no slides. Bezos banned those years ago, for good reason. In three years, the only time I opened up Powerpoint was for an in-house training session I put together about my organization’s product that was presented to a large audience. Even then, there was an accompanying doc.
Jira doesn’t exist at AWS, and working even briefly where it is not only used, but used for a lot of management and operational tasks that go well beyond what I’ve seen in previous engagements, has been a major shift. At AWS, we used a clunky in-house thing called SIM to handle ticketing and work management. Like Emacs/LaTeX, SIM is crappy in a lot of ways, but it is fit for a specific purpose at AWS. I know there were possible use cases beyond the way most of us used it, but I didn’t know anybody who put much effort into using them because it was… painful. The limitations of the tool meant you weren’t tempted* to think about “what else could I do with this thing?” just as the limitations of LaTeX discourage me from spending hours on pointless formatting.
On the flip side, you could experiment with any technology or programming language you wanted to so long as it didn’t violate security rules. That was rightly considered a valuable use of your time.
What is your organization?
What tools and roles do you need to run it?
AWS is a loosely-coupled, service oriented organization, producing loosely-coupled service-oriented platforms, for you to use to build your loosely-coupled service-oriented applications. Approved AWS-wide initiatives were tracked in a spreadsheet managed by one person. There were necessarily few of them and you have to really fight to get your initiative onto that list.
One common complaint about AWS is that it’s difficult for customers to manage things centrally. That’s a feature not a bug. Read through AWS’s Well Architected Framework in detail and you’ll see “loosely coupled” come up a lot. If you want tightly-coupled and highly integrated systems, AWS will more or less tell you that you’re wrong, but feel free to rent servers and storage by the hour/MB and install whatever centralized platforms you want to build on your own. Many companies that require greater centralization of resources do exactly that.
[Microsoft, from what I’ve seen, is very different. Their entire history is of tight integration: Windows with the various sub-programs, services, the browser, the media player, Office, Xbox — that one’s a non-deletable item on the Windows menu even though I don’t play video games — and hundreds of other things that become part of the overall “Windows Experience.” Flexible, customer-definable, service-oriented Windows is called Linux, and last I checked it isn’t especially successful for the personal desktop. I use it, but I’m weird. That means Microsoft can’t be like Amazon and has to put a lot of effort into coordination of all those pieces to make them appear seamless. They also have the margins to afford that kind of coordination overhead, which pre-AWS Amazon did not, and most companies never will.]
In retrospect, one of the most remarkable things about AWS is how well aligned the organization, the technological environment, the end-products and the internal tooling are. When Jeff Bezos wrote his now-infamous API mandate, he didn’t do it just to introduce a new technological paradigm at Amazon, he did it because of the frustration of coordinating an ever-growing organization. His solution to the coordination problem was “let’s not.” As Steve Yegge’s rant points out, that decision had implications for the organization and for every product it produced. It also has had implications for the tools that were chosen or excluded.
AWS didn’t have a whole lot of program managers (technical or otherwise). Programs are understood to be cross-team initiatives, and in a service-oriented company, many organizations don’t need that role. There is no standalone program management organization as there are at some companies. Technical (and other) program managers report into senior managers or directors, not to a single team or a dedicated organization. They are responsible for managing the organization’s cross-team requirements and commitments where they absolutely must exist. One of the reasons I decided to take a hiatus and explore options is that it’s really hard to move beyond my level as a TPM at AWS. I did a quick search a few months before I left and I found something like 14 above my level. If you wanted to move beyond “senior,” the typical path was to become an engineering manager where the opportunities to progress were greater. (The retail side of Amazon is different, and had quite a few more of us at higher levels.)
The flip side is that if you are a TPM at AWS, it means there is something pretty damn important for you to do, because TPMs don’t get hired to manage routine activity. I miss that and it’s one of the things that makes me still question whether I made the right choice.
Alignment
“Alignment” is an overused buzzword that MBAs like myself throw around too much. AWS has incredibly strong alignment between the products it offers (service oriented), the way it operates (loosely-coupled, low-trust, high team freedom, structured and limited status reporting), the tools it uses (mostly lightweight, often kind of crappy, but designed to help teams and individuals manage themselves internally), and the roles it selects to have and not to have (limited program management). But as Corey noted in his talk, different organizations face different constraints.
The problem I have seen over my career is that this alignment often does not exist. As a consultant prior to working at AWS, much of the work — whether stated explicitly or not — was in getting alignment between how companies worked, what they produced, and what tools they used. I declined a consulting gig with a company where the CTO was simultaneously talking about service orientation and the need to implement complex central program/project management tools. I heard later that it did not end well. The obsession with managing everything centrally destroyed the SOA strategy upon which the company was depending to deliver innovative products quickly.
I mistakenly took a gig with a company where the responsible individual, like the Banker in Corey’s talk, thought a highly-regulated healthcare environment was compatible with the kind of freedom and flexibility engineers at AWS or Netflix enjoy, and happily paid me to remove tooling and associated process that slowed things down, but which turned out to have been necessary. I was new to my role then. Lessons were learned, but sadly not by the client who ejected to a better job elsewhere when things went sideways.
One of the saddest cases I can think of involved a successful company that sold products to a market niche in the US. Financing was a key piece of the sales puzzle. The successful salespeople knew how to tie product value, cost, and financing together into packages that were somewhat individualized to every customer.
A new CFO who was an expert in financial modeling decided it could be better. He came up with a commission schedule that was defined by a complex graph, with the goal of providing more perfect incentives. Everything was taken into account: the margin on the sales price, the amount of time it took to close the sale, the reliability of payments over the life of the financing, the number of times a customer needed to be called if they failed to make a payment. It was a “better” tool, and far more “perfect.” It also destroyed the company.
Salespeople whose income was commission-based could no longer predict or understand their commission payout. (Even an MBA-holding engineer like myself had a tough time following the calculations.) Most of them only had a high school education. The “better incentives” didn’t work because they couldn’t understand them. To better match commission with realized revenue, the commissions spread over months or years depending on the financing scheme. Salespeople stopped going after profitable customers if they needed longer-term financing even though financing was a major source of profit. Informal and always incorrect spreadsheets began circulating to allow salespeople to enter the details of their deals and predict what the next paycheck would be.
The new tool didn’t better inform the important decisions: who the good salespeople were, who should get a bonus, and who should be fired. The much simpler pre-existing tools provided sufficient details to understand that. But it was a “perfect” tool, gave the CFO “correct” visibility, and “perfectly aligned incentives.” Too bad the people whose incentives were being “aligned” weren’t computers. The good salespeople left. Sales declined. The company lost its previous resilience and when the financial crisis hit, it collapsed.
Last I heard the CFO went back to academia where he continues to build ever more perfect models with no real-world application. I’m sure the PE vultures on whose watch this happened did fine as well. Too bad about the hundreds of people who lost their jobs to prove a theory.
Simplify
One of my favorite Amazon Leadership Principles is “Invent and Simplify.” The idea being that you should always be looking for a simpler way of accomplishing the same goal, even to the point of questioning whether the goal was worth pursing at all. It meant that anything adding layers of work, reporting, or organizational complexity was immediately suspect. It gave me cover to push back when I was asked to do the same thing more than once, or with small variations for different audiences. I can’t say I always did this well, but I tried to see the common threads and eliminate distractions that got in the way of that other leadership principle “Deliver Results.” That drove me to regularly ask why I was using the tools I was and whether I could dispense of some of them entirely. It’s why I had Emacs on my desktop.
* The argument is often made that just because a tool has certain capabilities, doesn’t mean you have to use them. My experience is that people will experiment up to the point where the experimentation becomes painful to them. Advanced tools with easy user interfaces make it easy to experiment, so the features get used, processes get implemented, they go organization wide, and 3 years later every manager in the organization is spending several hours a week fighting the formatting of a status report template that a long-departed VP declared as the standard, and then duplicating the information into a program-tracking system that was implemented by another well-intended person, when the actual value-added work of writing up what your team did this week and circulating it should take 20 minutes. If all you have is a hammer, every problem looks like a nail, but if you have a full-blown woodshop at your disposal, there’s a tendency to turn every small project into a major undertaking. When you start demanding that of everybody, you end up doing nothing else.