Follow-up post: here
It’s probably not a secret that I dislike the “Agile” fad that has infested programming. One of the worst varieties of it, Scrum, is a nightmare that I’ve seen actually kill companies. By “kill” I don’t mean “the culture wasn’t as good afterward”; I mean a drop in the stock’s value of more than 85 percent. This shit is toxic and it needs to die yesterday. For those unfamiliar, let’s first define our terms. Then I’ll get into why this stuff is terrible and often detrimental to actual agility. Then I’ll discuss a single, temporary use case under which “Agile” development actually is a good idea, and from there explain why it is so harmful as a permanent arrangement.
So what is Agile?
The “Agile” fad grew up in web consulting, where it had a certain amount of value: when dealing with finicky clients who don’t know what they want, one typically has to choose between one of two options. The first is to manage the client: get expectations set, charge appropriately for rework, and maintain a relationship of equality rather than submission. The second is to accept client misbehavior (as, say, many graphic designers must) and orient one’s work flow around client-side dysfunction. Programmers tend not to be good at the first option– of managing the client– because it demands too much in the way of social acumen, and the second is appealing to a decision-maker who’s recently been promoted to management and won’t have to do any of the actual work.
There’s a large spectrum of work under the name of “consulting”. There are great consulting firms and there are body shops that taken on the lowest kind of work. Companies tend to give two types of work to consultancies: the highest-end stuff that they might not have the right people for, and the low-end dreck work that would be a morale-killer if allocated to people they’d actually like to retain for a year or few. Scrum is for the body shops, the ones that expect programmers to suffer when client relationships are mismanaged and that will take on a lot of low-end, career-incoherent work that no one wants to do.
So what are Scrum and “Agile”? I could get into the different kinds of meetings (“retrospective” and “backlog grooming” and “planning”) or the theory, but the fundamental unifying trait is violent transparency, often one-sided. Programmers are, in many cases, expected to provide humiliating visibility into their time and work, meaning that they must play a side game of appearing productive in addition to their actual job duties. Instead of working on actual, long-term projects that a person could get excited about, they’re relegated to working on atomized, feature-level “user stories” and often disallowed to work on improvements that can’t be related to short-term, immediate business needs (often delivered from on-high). Agile eliminates the concept of ownership and treats programmers as interchangeable, commoditized components.
In addition to being infantilizing and repellent, Scrum induces needless anxiety about microfluctuations in one’s own productivity. The violent transparency means that, in theory, each person’s hour-by-hour fluctuations are globally visible– and for no good reason, because there’s absolutely no evidence that any of this snake oil actually makes things get done quicker or better in the long run. For people with anxiety or mood disorders, who generally perform well when measured on average long-term productivity, but who tend to be most sensitive to invasions of privacy, this is outright discriminatory.
Specific flaws of “Agile” and Scrum
1. Business-driven engineering.
“Agile” is often sold in comparison to an equally horrible straw man approach to software design called “Waterfall”. What Waterfall and Agile share (and a common source of their dysfunction) is that they’re business-driven development. In Waterfall, projects are defined first by business executives, design is done by middle managers and architects, and then implementation and operations and testing are carried out by multiple tiers of grunts, with each of these functions happening in a stage that must be completed before the next may begin. Waterfall is notoriously dysfunctional and no Agile opponent would argue to the contrary. Under Waterfall, engineers are relegated to work on designs and build systems after the important decisions have all been made and cannot be unmade, and no one talented is motivated to take that kind of project.
Waterfall replicates the social model of a dysfunctional organization with a defined hierarchy. The most interesting work is done first and declared complete, and the grungy details are passed on to the level below. It’s called “Waterfall” because communication goes only one way. If the design is bad, it must be implemented anyway. (The original designers have probably moved to another project.) Agile, then, replicates the social model of a dysfunctional organization without a well-defined hierarchy. It has engineers still quite clearly below everyone else: the “product owners” and “scrum masters” outrank “team members”, who are the lowest of the low. Its effect is to disentitle the more senior, capable engineers by requiring them to adhere to a reporting process (work only on your assigned tickets, spend 5-10 hours per week in status meetings) designed for juniors. Like a failed communist state that equalizes by spreading poverty, Scrum in its purest form puts all of engineering at the same low level: not a clearly spelled-out one, but clearly below all the business people who are given full authority to decide what gets worked on.
Agile increases the feedback frequency while giving engineers no real power. That’s a losing bargain, because it means that they’re more likely to jerked around or punished when things take longer than they “seem” they should take. These decisions are invariably made by business people who will call shots based on emotion rather than deep insight into the technical challenges or the nature of the development.
Silicon Valley has gotten a lot wrong, especially in the past five years, but one of the things that it got right is the concept of the engineer-driven company. It’s not always the best for engineers to drive the entire company, but when engineers run engineering and set priorities, everyone wins: engineers are happier with the work they’re assigned (or, better yet, self-assigning) and the business is getting a much higher quality of engineering.
2. Terminal juniority
“Agile” is a culture of terminal juniority, lending support to the (extremely misguided) conception of programming as a “young man’s game”, even though most of the best engineers are not young and quite a few are not men. Agile has no exit strategy. There’s no “We won’t have to do this once we achieve ” clause. It’s designed to be there forever: the “user stories” and business-driven engineering and endless status meetings will never go away. Architecture and R&D and product development aren’t part of the programmer’s job, because those things don’t fit into atomized “user stories” or two-week sprints. So, the sorts of projects that programmers want to take on, once they master the basics of the field, are often ignored, because it’s either impossible to atomize them or it’s far more difficult to do so than just to do the work.
There’s no role for an actual senior engineer on a Scrum team, and that’s a problem, because many companies that adopt Scrum impose it on the whole organization. Aside from a move into management, there is the option of becoming a “Scrum master” responsible for imposing this stuff on the young’uns: a bullshit pseudo-management role without power. The only way to get off a Scrum team and away from living under toxic micromanagement is to burrow further into the beast and impose the toxic micromanagement on other people. What “Agile” and Scrum say to me is that older, senior programmers are viewed as so inessential that they can be ignored, as if programming is a childish thing to be put away before age 35. I don’t agree with that mentality. In fact, I think it’s harmful; I’m in my early 30s and I feel like I’m just starting to be good at programming. Chasing out our elders, just because they’re seasoned enough to know that this “Agile”/Scrum garbage has nothing to do with computer science and that it has no value, is a horrible idea.
3. It’s stupidly, dangerously short-term.
Agile is designed for and by consulting firms that are marginal. That is, it’s for firms that don’t have the credibility that would enable them to negotiate with clients as equals, and that are facing tight deadlines while each client project is an existential risk. It’s for “scrappy” underdogs. Now, here’s the problem: Scrum is often deployed in large companies and funded startups, but people join those (leaving financial upside on the table, for the employer to collect) because they don’t want to be underdogs. No one wants to play from behind unless there’s considerable personal upside in doing so. “Agile” in a corporate job means pain and risk without reward.
When each client project represents existential or severe reputational risk, Agile might be the way to go, because a focus on short-term iterations is useful when the company is under threat and there might not be a long term. Aggressive project management makes sense in an emergency. It doesn’t make sense as a permanent arrangement; at least, not for high-talent programmers who have less stressful and more enjoyable options.
Under Agile, technical debt piles up and is not addressed because the business people calling the shots will not see a problem until it’s far too late or, at least, too expensive to fix it. Moreover, individual engineers are rewarded or punished solely based on the completion, or not, of the current two-week “sprint”, meaning that no one looks out five “sprints” ahead. Agile is just one mindless, near-sighted “sprint” after another: no progress, no improvement, just ticket after ticket.
4. It has no regard for career coherency.
Atomized user stories aren’t good for engineers’ careers. By age 30, you’re expected to be able to show that you can work at the whole-project level, and that you’re at least ready to go beyond such a level into infrastructure, architecture, research, or leadership. While Agile/Scrum experience makes it somewhat easier to get junior positions, it eradicates even the possibility of work that’s acceptable for a mid-career or senior engineer.
In an emergency, whether it’s a consultancy striving to appease an important client or a corporate “war room”, career coherency can wait. Few people will refuse to do a couple weeks of unpleasant or career-incoherent work if it’s genuinely important to the company where they work. If nothing else, the importance of that work confers a career benefit. When there’s not an emergency, however, programmers expect their career growth to be taken seriously and will leave. Using “fish frying” as a term-of-art for grunt work that no one enjoys, and that has no intrinsic career value to any one, there’s enough career value (internal and external to the organization) in emergency or high-profile fish frying that people don’t mind doing it. You can say, “I was in the War Room and had 20 minutes per day with the CEO” and that excuses fish frying. It means you were valued and important. Saying, “I was on a Scrum team” says, “Kick me”. Frying fish because you were assigned “user stories” shows that you were seen as a loser.
5. Its purpose is to identify low performers, but it has an unacceptably false positive rate.
Scrum is sold as a process for “removing impediments”, which is a nice way of saying “spotting slackers”. The problem with it is that it creates more underperformers than it roots out. It’s a surveillance state that requires individual engineers to provide fine-grained visibility into their work and rate of productivity. This is defended using the “nothing to hide” argument, but the fact is that, even for pillar-of-the-community high performers, a surveillance state is an anxiety state. The fact of being observed changes the way people work– and, in creative fields, for the worse.
The first topic coming to mind here is status sensitivity. Programmers love to make-believe that they’ve transcended a few million years of primate evolution related to social status, but the fact is: social status matters, and you’re not “political” if you acknowledge the fact. Older people, women, racial minorities, and people with disabilities tend to be status sensitive because it’s a matter of survival for them. Constant surveillance into one’s work indicates a lack of trust and low social status, and the most status-sensitive people (even if they’re the best workers) are the first ones to decline.
Scrum and “Agile” are designed, on the other hand, for the most status-insensitive people: young, privileged males who haven’t been tested, challenged, or burned yet at work. It’s for people who think that HR and management are a waste of time and that people should just “suck it up” when demeaned or insulted.
Often, it’s the best employees who fall the hardest when Agile/Scrum is introduced, because R&D is effectively eliminated, and the obsession with short-term “iterations” or sprints means that there’s no room to try something that might actually fail.
The truth about underperformers is that you don’t need “Agile” to find out who they are. People know who they are. The reason some teams get loaded down with disengaged, incompetent, or toxic people is that no one does anything about them. That’s a people-level management problem, not a workflow-level process problem.
6. The Whisky Googles Effect
There seems to be some evidence that Agile and Scrum can nudge the marginally incompetent into being marginally employable. I call this the Whisky Goggles Effect: it turns the 3s and 4s into 5s, but it makes you so sloppy that the 7s and 9s want nothing to do with you. Unable to get their creative juices flowing under aggressive micromanagement, the best programmers leave.
From the point of view of a manager unaware of how software works, this might seem like an acceptable trade: a few “prima donna” 7+ leave under the Brave New Scrum, while the 3s and 4s become just-acceptable 5s. The problem is that the difference between a “7” programmer and a “5” programmer is substantially larger than that between a “5” and a “3”. If you lose your best people and your leaders (who may not be in leadership roles on the org-chart) then the slight upgrade of the incompetents for whom Scrum is designed does no good.
Scrum and Agile play into what I call the Status Profit Bias. Essentially, many people in business judge their success or failure not in objective terms, but based on the status differential achieved. Let’s say that the market value of a “3” level programmer is $50,000 per year and, for a “5” programmer, it’s $80,000. (In reality, programmer salaries are all over the map: I know 3’s making over $200,000 and I know 7’s under $70,000, but let’s ignore that.) Convincing a “5” programmer to take a “3”-level salary (in exchange for startup equity!) is marked, psychologically, not as a mere $30,000 in profit but as a 2-point profit.
Agile/Scrum and the age discrimination culture in general are about getting the most impressive status profits, rather than actual economic profits. The people who are least informed about what social status they “should” have are the young. You’ll find a 22-year-old 6 who thinks that he’s a 3 and who will submit to Scrum, but the 50-year-old 9 is likely to know that she’s a 9 and might begrudgingly take 8.5-level conditions but is not about to drop to a 6. Seeking status profits is, however, extremely short-sighted. There may be a whole industry in bringing in 5-level engineers and treating (and paying) them like 4’s, but under current market conditions, it’s far more profitable to hire an 8 and treat him like an 8.
7. It’s dishonestly sold.
To cover this point, I have to acknowledge that the uber-Agile process known as “Scrum” works under a specific set of circumstances; the dishonest salesmanship is in the indication that this stuff works everywhere, and as a permanent arrangement.
What Scrum is good for
Before the Agile fad, “Scrum” was a term sometimes used for what corporations might also call a “Code Red” or a “War Room emergency”. This is when a cross-cutting team must be built quickly to deal with an unexpected and, often, rapidly-evolving problem. It has no clear manager, but a leader (like a “Scrum Master”) must be elected and given authority and it’s usually best for that person not to be an official “people manager” (since he needs to be as impartial as possible). Since the crisis is short-term, individuals’ career goals can be put on hold. It’s considered a “sprint” because people are expected to work as fast as they can to solve the problem, and because they’ll be allowed to go back to their regular work once it’s over.
There are two scenarios that should come to mind. The first is a corporate “war room”, and if specific individuals (excluding high executives) are spending more than about 6 weeks per year in war-room mode, than something is wrong with the company because emergencies ought to be rare. The second is that of a consultancy struggling to establish itself, or one that is bad at managing clients and establishing an independent reputation, and must therefore operate in a state of permanent emergency.
Two issues
Scrum and Agile represent acknowledgement of the idea that emergency powers must sometimes be given to “take-charge” leaders who’ll do whatever they consider necessary to get a job done, leaving debate to happen later. A time-honored problem with emergency powers is that they often don’t go away. In many circumstances, those empowered by an emergency see fit to prolong it. This is, most certainly, a problem with management. Dysfunction and emergency require more managerial effort than a well-run company in peace time.
It is also more impressive for an aspiring demagogue (a “scrum master”?) to be a visible “dragonslayer” than to avoid attracting dragons to the village in the first place. The problem with Scrum’s aggressive insistence on business-driven engineering is that it makes a virtue (“customer collaboration”) out of attracting, and slaying, dragons (known as “requirements”) when it might be more prudent not to lure them out of their caves in the first place.
“Agile” and Scrum glorify emergency. That’s the first problem with them. They’re a reinvention of what the video game industry calls “crunch time”. It’s not sustainable. An aggressive focus on individual performance, a lack of concern for people’s career objectives in work allocation, and a mandate that people work only on the stated top priority, all have value in a short-term emergency but become toxic in the long run. People will tolerate those changes if there’s a clear point ahead when they’ll get their autonomy back.
The second issue is that these practices are sold dishonestly. There’s a whole cottage industry that has grown up around teaching companies to be “Agile” in their software development. The problem is that most of the core ideas aren’t new. The terminology is fresh, but the concepts are mostly outdated, failed “scientific management” (which was far from scientific, and didn’t work).
If the downsides and upsides of “Agile” and Scrum were addressed, then I wouldn’t have such a strong problem with the concept. If a company has a team of only junior developers and needs to deliver features fast, it should consider using these techniques for a short phase, with the plan to remove them as its people grow and timeframes become more forgiving. However, if Scrum were sold for what it is– a set of emergency measures that can’t be used to permanently improve productivity– then there would be far fewer buyers for it, and the “Agile” consulting industry wouldn’t exist. Only through a dishonest representation of these techniques (glorified “crunch time”, packaged as permanent fixes) are they made salable.
Looking forward
It’s time for most of “Agile” and especially Scrum to die. These aren’t just bad ideas. They’re more dangerous than that, because there’s a generation of software engineers who are absorbing them without knowing any better. There are far too many young programmers being doomed to mediocrity by the idea that business-driven engineering and “user stories” are how things have always been done. This ought to be prevented; the future integrity of our industry may rely on it. “Agile” is a bucket of nonsense that has nothing to do with programming and certainly nothing to do with computer science, and it ought to be tossed back into the muck from which it came.
