A piece I read recently about how “Systems Design Explains The World” reminded me of a story. Well, a bunch of stories. And since I was recently asked for a story by a very special person and didn’t have one ready, here it goes.
Really, you should read that blog post and the ones it links to, but if you don’t want to, here is the critical excerpt that smacked me in the face. Fair credit: it’s only a tiny piece of what he discusses and you should really read the whole thing.
There were two groups of misfits:
- People who maxed out as a senior engineer (building things) but didn’t seem to want to, or be able to, make it to staff engineer (translating business problems).
- People who were ranked at junior levels, but were much better at translating business problems than at fixing bugs.
Group #1 was already formally accounted for: the official word was that most employees should never expect to get past Senior Engineer. That was the whole point of calling that level Senior.
People in group #2 weren’t supposed to exist. They were doing some hard jobs – translating business problems into designs – with great expertise, but these accomplishments weren’t interesting to the junior-level promotion committees, who had been trained to look for “exactly one level up” attributes like deep technical knowledge in one or two specific areas, a history of rapid and numerous bug fixes, small independent launches, and so on.
As has often been the case in my life, I was in a category of people who were not supposed to exist. I was that “Type II misfit.” How I got there is the story.
When I joined the tech business it was a much smaller place, and it was one into which it was not possible to hire enough people with solid technical backgrounds. So we had things called “training programs” that most large companies ran to get people without the right tech background up to speed. My company made a point of hiring people with limited tech backgrounds. The approach was based on the recognition that most of our programming problems were not very hard, but that the larger problems we were solving could be very challenging. They hired for generic problem solving not tech skill, and they put you through 6 months of hell to come up to speed on enough tech to get started.
I had the tech background, but everybody went through the same (more or less) training. I realized later that this was also a great filter for the company. When it was over they knew who was good at what. Most of us went into software development roles, but some went into data center ops, some worked on internal customer service teams, and others went on to specialties like database management or systems programming. You had six months to demonstrate what role you’d be most ideal for.
Training was a full-time job. You spent 8 hours a day paying your dues doing grunt work in the data center, then 4 hours a day of training. Depending which shift you were on that week, you worked 8-4 and trained 4-8, trained 12-4 then worked 4-midnight, or trained 8-midnight, then worked through to 8am. Unless you had a weekend shift in which case you worked two 12 hour shifts on the weekend, normal shifts 2 other days, and did one day of solid training.
We still went out for beers after work, even after getting out at 8am. It was NYC, there were lots of people working a variety of night shifts, and the bars open early. The bar that we went to is still there, and from what I’m told still does a solid early-morning business for Wall Streeters who are up all night. [There are plenty of bars in NYC that will lock the doors and stop serving at 4am as the law requires, but allow you to remain and “finish your drink” until they re-open at 6. If you’re in such a place, please be polite and pick up your feet when asked so they can mop the floor.]
I graduated from the training program two months early. They had just increased the class size and didn’t need that many people in the data center. I was one of the few with a CS background and I could accelerate through the training quickly once they put me on it full time since all I needed to pick up was some language and environment-specific details. Then I got moved uptown into a development and ops support job.
It was different then. We always deployed on Fridays, because as a financial institution we were closed weekends, and that meant we would always have a full weekend to recover if things really screwed up. That was preferable to having to be shut down for a day during the week. Code was monolithic, running a single version on a single platform. Ops was relatively straightforward. I had to wear a tie. The programming we did was not incredibly technical and we worked in straightforward languages. Programming for financial operations is mostly reads, sorts, selects, and occasionally updates. We used standard libraries for everything complex. Very little of my CS background was critical. I’ve never in my life actually implemented a sort or search algorithm. Has anybody?
I spent six months doing small maintenance tasks, feature improvements, some reporting, and ops support. I got to support our Tokyo operation’s first major audit by the Japanese Ministry of Finance. They were surprised at how quickly we were able to collect and present the data they wanted. It was the last time I had a role with a significant ops support responsibility.
I worked independently. The guy who had previously supported our Tokyo office ops had quit and I moved into the role. My boss brought me up to speed and turned me loose. Colleagues working on our “primary” (US markets) systems helped me figure out how to manage what was effectively a limited-feature fork of their larger environment. There wasn’t anybody else managing our daily “Tokyo flow” and time differences meant that the “overnight” processing that was the bulk of my work happened during our early morning. I had a fairly independent schedule and role.
Part of my job was being the one who was pulled in to support major projects being done by the New York systems teams, to ensure that whatever changes were being made would be properly reflected in our Tokyo environment and not break anything. Unlike most of my classmates who were assigned to teams with a narrow focus, my function was more as a generalist than a specialist. I had to have decent knowledge of all the systems used by a small office rather than highly-detailed knowledge of a single system used in a large one. I learned the breadth of all our major systems, rather than one of them in depth.
Along the way I really impressed a manager who was managing a very disruptive change to the way we processed and recorded end of day balances and positions. Much of the work was done in advance, with all systems updated so they could handle the change to the underlying database structure when it occurred. To put the change into effect, the database had to be re-written and the standard routines used to update it changed. It was a lot to do all at once, but a lot better than having to re-write everything that read the database. It was a pretty ingenious change that reduced our database utilization by over 90%, with minimal impact on real-world use cases. I learned a lot just from watching.
The change to the databases and the systems that updated them had to be done as a “big bang” in one weekend. Everything would run in parallel with the old system: we would do a full night’s processing run, then change over everything and run it again. Everything in every system had to match the second time around, or the whole thing would get rolled back. No other changes were permitted anywhere. Everything was locked down (or so we thought).
Time zones meant that Tokyo went first. I was the guinea pig. It worked. By mid-afternoon on Friday I was able to advise everybody on the team that the full systems run was complete and everything matched up. The London stream went next and also ran well. We were in great shape.
Well, except the part where a “star programmer” on the NY trading system decided that he had to get a change in, that he was sure it wouldn’t have impact, and oh, he happened to have a senior person’s override code to approve it despite the lockdown. This caused there to be discrepancies in some of the back end reporting for New York (Tokyo and London didn’t use that particular functionality so it didn’t show up for us), and almost caused the whole thing to be backed out.
Fortunately somebody looked more critically and asked “hey, who the hell approved this?” A lot of us were really, really pissed that that “star” kept his job. The fact that he did taught me a lot of lessons about office politics.
Not long after that, a new team was assembled to build a Tokyo-native development and ops team. It was long overdue given the rapid growth we had experienced. I trained them and got them ready to take on my role along with other tasks, then had little to do.
The manager who I had impressed had moved on to a new role and asked me to join her team. It was also internationally focused, but mostly on smaller geographies where we didn’t have a physical presence and instead worked through intermediaries. I was to help figure out how to automate procedures that previously had been managed with faxes and odd-hour phone calls. Two weeks later, I was in London.
Much of the work I needed to do leveraged systems that had been developed by our London team. The idea was that I would learn from them while building new pieces of software. In retrospect, I used a lot less of their work than anybody thought I would. I could have gotten all I needed from the trip in about a week, but I was there for six. I wrote some of my best code in that job during that period, and also learned about the things I should never do, like build user interfaces.
[For whatever reason, user interfaces that make sense to me are useless to anybody else. Most “professional” user interfaces are painful for me to use. Things designers tell me make for a great “user experience” I find hateful. The internet has only made this worse which is why I own so many broken keyboards.]
I spent my 1-year work anniversary drinking beers in London with a bunch of Lebanese guys I had met in a pub near my hotel rather than with my classmates who had gathered at our old “8am after the night shift” bar. It was a fun summer and while I could have done a much shorter trip, I also learned a lot about working cross-border and cross-culture by spending as much time there as I did. Shortly after my 23rd birthday, I flew back to New York.
I spent a couple of months finishing what I had started, testing it and working through the bugs. The tech world was smaller then. When I was experiencing problems between my software, the Hayes modem that we expected our counterparts overseas to use, and Microsoft drivers, I could just pick up the phone, call Hayes, call Microsoft, ask for “whoever is working on X,” find the right people and figure it out with them. I worked with an outside programmer whose office was in WTC Tower 2, the only time I had ever been in one of those lost buildings.
It was a lot of coordination, understanding the larger problems, and working around them based on the realities of what we needed to achieve and the limitations of the technologies and communication infrastructure. There was no map and no process for getting from where we were to where we needed to go, so I made up my own. By October it was done, and we were ready to deploy. Italy was first. I was off to Milan.
Milan set the tone for the next 6-12 months of deployments. I arrived on an early-morning flight at the start of fashion week after spending my first night in Europe at the Zurich airport because of a lack of hotel rooms in Milan. This would also become a recurring theme. Almost every deployment we did was at the worst possible time for the place we were doing it. I learned a lot from that too.
I was the sole technical member of a team of business development and market operations people. We gave presentations, we did training, we helped our counterparts understand the docs which were all in English and translate them as needed. I troubleshot their PC and network/modem connection. We worked with our software, showing them how to connect to our systems remotely, download information, process it locally and upload results reliably. By the end of the week, faxes were a thing of the past.
My role was to hold together all the pieces, to be the “glue” on the technical side that ensured that the software I wrote, the software the guy at WTC 2 wrote, systems software on a mainframe in New York, a leased network, and all the other pieces, worked together. It was about figuring out how to solve problems as they came up and address them in the most flexible and agile manner possible. There was no real model and no real metrics to use as a guide. I wasn’t in a position to do much real development work when I was at a partner site, so the key was to know who to call and to be able to accurately specify what was needed.
We were working with a bank that had emerged from the bank at the center of the Vatican Bank Scandal, after its former chairman inexplicably jumped off a bridge in London with a rope around his neck and bricks in his pockets, then the pope died mysteriously after a month on the job. On the one hand, they were eager for the business and an opportunity to be part of something forward-looking. On the other hand, it was… weird. Our primary contact was a real hustler we nicknamed “Lucky” because he always seemed to be focused on other opportunities and possibilities. He spent most of his time monitoring the prices of gold and South African shares. Several months later, he disappeared. I decided not to ask questions.
[Years later, he resurfaced in Switzerland. Today, LinkedIn shows him running his own asset management firm, so my worst guesses at the time were apparently wrong.]
The same repeated elsewhere. I held things together dealing with the kinds of messy problems that aren’t solved through the application of algorithms and technical knowledge. Not that those hurt, but they just weren’t the key to success. Over time, the “on-site” team was reduced to just me, and occasionally one business development person who would come along for the first day of introductions, wining and dining.
I have great memories, but none of them are about programming: the time that I — by far the most junior person on the team — ended up in the most expensive hotel in Paris (Intercontinental) while everybody else was in a cheaper place down the street. (I was added to the team last, and by then the cheap rooms were gone, so travel put me in the more expensive place.) They got me back by drinking in the bar every night and charging it to my room. My boss was not amused at the $2000 bar bill. I learned a lot about dealing with colleagues.
My boss was also not amused when I called from Melbourne on my first day there to explain that they had sent me during Melbourne Cup week, that the entire country would be drunk for the rest of the week starting the middle of the next day, and by the way I’m going to have to extend my stay another week to get anything done. As usual, I was there at the worst possible time. I figured out how to make it work.
But through this I built a lot of trust with both my internal and external partners, and we did something that a year prior nobody thought was possible.
Years later, a manager I worked with said my greatest strength was my ability to sit at the back of the room, take everything in, figure out the direction things were going, and only say something when the team needed to be put back on course, or when they hit a stumbling block that — he thought — I had probably anticipated before the meeting even started and had prepared for. (He exaggerated, I’m not that good.) That’s the skill I learned in those few early years.
During all this I had been promoted, ahead of the normal schedule. There’s a downside to that. I had not done the more detailed coding and ops work that everybody else who came up with me and had been slotted into more typical developer roles had done. When it was over, I moved into a related role with another startup division that was using variants on the systems I had built. In that role I learned one of the most important lessons I’ve ever had about where and when engineering and technology matter.
The manager of that division was a youngish British banker who had come over to New York for the opportunity. He was really smart about how we could use our technology to disrupt the staid world he had come out of. But there was a very cold realism to his approach. After one particularly successful customer launch, he reminded us: “Our customers don’t come to us because we have the best technology, they come to us because we do a better job and provide a better value. They’re happy to see us using great technology and are impressed with our mastery of it, but in the end, if we can provide the same level of service by putting a million monkeys in front of a million typewriters, they’d be just as interested in the value proposition.”
[Years later, I realized that his idea may be the first known example of a serverless architecture.]
I have never forgotten the “million monkeys” analogy. I credit him for my tendency to always be skeptical of technology even as I work to master it. I’ve saved my clients and employers millions of dollars by warning them off of expensive “investments” that would only generate value to the vendor, and resume-padding material for the people working on it.
I’ve pissed off hundreds of software and services sales people by keeping my clients focused on the question of “what problem does this solve at what cost?” rather than “is this cool tech?” or even “could this work?” To this day, I use less tech in my home than a lot of people who aren’t in the tech business, because, really, what problem is it solving?
I also learned something from him about pricing. We were delivering a better and more accurate service, and were able to do it cheaper than the competition. Some thought we could charge less and just take over the market. But we charged more. When you’re doing a better job, you charge more. If you don’t, the smart customers will wonder why. If you’re able to do it at lower internal cost, that just means more profit to you.
In that business, competitors eventually caught up and prices/profitability declined, but I learned not to under-price, and the importance of using higher prices as a signal of value. This has held true for me working independently. I was never more in demand than when I raised my rates.
My further success in that job was short-lived. It’s tough to come back to the “standard progression” when you’ve gone off in a completely different direction for 2-3 years. I had spent my time working on very messy problems of very high importance, but where rigorous engineering formulas failed or just weren’t applicable. I loved every minute of it, but it came back to haunt me when financial markets declined, budgets were cut, and the focus moved to incremental efficiencies in existing products and refactoring code. That is to say, work that required the skills I had mostly not developed while doing other things.
I became the Type II misfit cited at the start. It’s not because I couldn’t be a great developer/engineer. Whenever I had to dive into hard engineering again I did well. But once you jump off that train and start dealing with the incredibly messy problems of inserting systems into different organizations in different countries with different expectations of how they would and should work, it’s hard to go back. I had been doing systems design, deployments, project management and customer relationships since I was a year out of school and I was good at it, but I moved there well ahead of the normal progression one would expect. Like other “Glue” types in the Type II Misfit category, I was in an odd position: a early/mid-level employee with senior or better skills in many critical but less technical areas, and far less rigorous engineering. After two other relatively brief jobs in New York, I took a few months off to ski, then on the advice of one of my first managers I applied to business school and never looked back.
[OK, business school may have been a mistake for other reasons, but the decision to refocus was not.]
I do still wonder what might have been had I been slotted into a different job early, or had I decided to take a step or two back to return to hardcore programming. I was good at it and might have done better for myself, but it’s impossible to know. Of the members of my original training class, I’m the only one who was remained in a tech-related field over the long haul.
As a technical program manager I’m still very much in the “glue” category, and I still wonder where the future will take me. I’m long past the point of going back to engineering, but it’s not clear to me that there is a strong career path from my current role to… something else.
Maybe that’s OK.