This is a work of fiction. Any resemblance to existing software companies is entirely coincidental.
Johnny is a programmer. He likes to solve difficult problems: eliminating race conditions, improving performance, and fixing issues everyone else gave up on. In this way, he expresses loyalty to his company. Colleagues respect Johnny, as does his boss, Jessica. That makes Johnny a happy programmer.
One day, Jessica calls him into her office.
“Listen, Johnny,” she says. “I think you care about quality, more than anyone else in our department. Is that right?”
“Um… I definitely care, but I’m not sure how I compare to others,” answers Johnny.
“Just believe me,” Jessica replies, smiling. “But… you know what the issue is?”
“Yes. Any guesses?”
“Do I spend too much time fixing bugs?”
“No! Absolutely not! The issue, Johnny, is that your care does not scale. And we need more of it as we grow. We are going to hire hundreds of new developers, you know? You’ll burn-out cleaning up garbage they’ll produce. Any thoughts on how we can be proactive about this?
“Umm… maybe test automation?”
“Exactly! Imagine our product without regressions, with a test cycle of a couple of hours! Imagine that we can deliver every week! What impact would that make?”
What Jessica says touches Johnny. He tries to write a simple test but quickly finds that everything is so interconnected that he needs to include nearly 80 percent of the production code in the test library.
Johnny knows that the test should be fast and stable, with all the dependency mocked, but it seems too distant and inapplicable in this situation. He instead decides to build testing capabilities directly into the executable. That way, his first test can be written in a high-level modern language, such a C# or Java, which most people in his organization use.
He writes a testing framework and a first test and shows it to Jessica. What she sees is clean, comprehensible and has a great potential.
“Listen, Johnny,” says Jessica. “What you’ve made is amazing! Finish your framework, and I want you to write more tests and start running them on the continuous integration server. I heard people like TeamCity. Could you try it out?”
“Sounds great, but I’ve never done anything like this before. Besides, I have other work to do.”
“Listen, Johnny. I promise you, once you TeamCity is setup, you can get back to your primary job. It’ll be all automated. For now, think of the impact this could have on the company. Who knows, maybe it’ll be a good way to advance in your career.”
One week later, TeamCity is up and running. Johnny creates more tests to prove that the system is working. Jessica is happy: She grants Johnny a raise and sends out a company-wide email announcing that the company is stepping into the Automation Era thanks to Johnny.
Johnny gets a lot of positive feedback. Never before in his career has he felt so proud of and happy about his work. He doesn’t realize he’s been knocking on the gates of automation hell.
As people write more and more tests, Johnny starts to notice something strange. Some tests pass and fail on the same revision. Everyone blames TeamCity, and Johnny seconds them, but his intuition tells him that it isn’t TeamCity’s fault. Whatever it is, Johnny has other work to do. It’s not a major issue, so he decides to look into it later.
After a year, there are thousands of tests. It is mandatory to run all the tests before the pull request goes to a production branch. However, TeamCity does not have enough resources and cannot handle the increased load anymore. Jessica calls Johnny in to discuss possible solutions. Johnny tries to persuade Jessica that they should buy more hardware. Jessica, however, does not like this idea: She wants to avoid to contact the hardware department.
Eventually, they decide to split the tests into two sets: mandatory and optional. Mandatory tests are something that must be run on each pull request. However, it is up to the developers to decide whether to run the optional tests.
Johnny makes some guesses and split tests into two sets. Jessica sends out an email explaining the new the test-running policy. It brings a sense of relief.
After a week, Johnny starts to discover that more and more optional tests are failing in the production branch.
“Optional tests are rotting,” Johnny thinks. “I’ll just ask the area owners to investigate it.
Hopefully, they’ll be able to fix the problem, and it will not appear in the future.”
The area owners, however, are busy and don’t want to spend their time fixing broken tests.
Johnny describes the situation to Jessica, and she promises to hire a student worker responsible for keeping the optional tests green.
The student worker, Jimmy, cannot figure out all the root causes, so he disables the failing tests and assigns the defects to the area owners. The defects never get fixed until one day, a couple of regressions sneak into a public release. Disabled tests become the number one priority to fix. In the meantime, the optional tests continue to rot.
After some googling, Johnny finds an article outlining a solution to the rotting tests problem. The idea is to group pull requests into a batch. Each pull request in a batch must have mandatory tests green. Then, the optional tests are run on top of the batch. If any test fails, the system bisects the failure and removes the offending pull request from the batch. The “good” part is merged into the production branch.
He comes to Jessica and tries to persuade her that this is the way to address the problem. Jessica agrees to hire one more person to implement this.
It works well, and again, it brings relief. Automation gains the employees’ trust back. However, as the number of tests grows, another issue arises on the automation stage: flaky tests.
Johnny starts to hear conversations like this:
“Tests suck! The testing framework is unstable. Please just let my pull request go through! The customers are waiting for it.”
“No, man, your code sucks! It looks like executable is crashing from time to time and crash might be related to changes you’ve made.”
“What? Can you point at what’s crashing?”
“Just look at the system logs!”
“I have other work to do!”
Johnny carefully investigates each case, categorizing them and trying to address the root causes of their instability, which often are: unmocked external dependencies, timing issues, race conditions, and incorrect assumptions about floating-point rounding.
Another meta-pattern Johnny discovers is that the higher a test’s level is, the more vulnerable it is to these issues. His testing framework was high-level from day one, so Johnny cannot blame anyone but himself.
Johnny fixes the testing framework, infrastructure issues and improves the diagnostics, which helps a lot. However, there is still one issue against which he is powerless: People keep adding more and more unstable tests.
He writes an educational email explaining how to avoid writing flaky tests. But it’s too late: Her infernal majesty Instability shows too much pain and despair. People are craving blood. So Johnny receives enraged replies.
Poor Johnny. He forgets his happy programming days. He combats instabilities all day long. His slack is full of desperate requests for help, and the public channels are full of complaints. The TeamCity load is high because people rerun unstable tests to make them green. A release is delayed. Everything is bad.
He still loves his job, mostly because of the memories of his earlier days in it. He remembers when he could afford to debug challenging issues all day long. Now his attention is spread everywhere. No more deep thinking: only a state of constant alert.
Jessica calls him into her office.
“I have bad news, Johnny… after a discussion with the board of directors, we decided that it would be best for you and the company if you leave. Today is your last working day. I’m very sorry, Johnny.”
Johnny is shocked and doesn’t know what to say. With tears in his eyes, he leaves Jessica’s office.
On the way to his desk, he realizes what’ve just happened.
“Just a fucking scapegoat” Johnny whispers.
“Pardon?” one his colleagues ask
“Nevermind. Just thinking out loud.”