Dive 77: The Costs of Irreversible Risks

Avoid the risk of a permanent game over.

Hey, it’s Alvin!

On June 26, 2017, Pedro Ruiz wanted to “launch his YouTube channel with a bang.” His girlfriend pulled the trigger on a gun. Now, Ruiz is dead. And his girlfriend was sentenced to 180 days in prison.

Today, I want to share with you an important lesson this story shares with my field of software development. We’ll also learn about risk. Because the only way to prevent these tragedies is to learn from them.

But first, we need to dive into what happened.

On the stunt.

In 2017, 22-year-old Pedro Ruiz arranged a stunt to launch a YouTube channel. The stunt would have his girlfriend firing a gun at him.

This was no ordinary gun. It’s a .50-calibre Desert Eagle—one of the most powerful handguns in the world. It’s not like Ruiz had a death wish. He was going to stand one foot away from the gun and he would hold a book in front of his chest to block the bullet.

His pregnant girlfriend was far from eager to go along with the stunt. But Ruiz insisted on it in part because he already test fired the gun on at least one book. There is evidence of a book that stopped the bullet.

The book he held that fateful day did not.

On test environments.

Former New York police officer and former Bronx County Assistant District Attorney, Nate Broughty, summarized this case in a YouTube video. One of the most important lessons from Broughty’s video was that “sometimes, it’s just not worth it.” Pedro thought he knew what was going on but failed to take all the precautions.

One of those precautions is using the same book in the test for the actual stunt. That’s what reminded me of how we test code in the world of software development.

For those who don’t know, we don’t change code directly on a live service that’s used by our users and customers. Because otherwise we’d introduce bugs that disrupt the service constantly.

Instead, we have development environments where we can test our code thoroughly before we push an entire update to the live “production” environments that you see as a user. In fact, even before pushing to a development environment, we’ll have a “local” environment set up on our laptops or desktops where we write code and unit tests.

But local and development (or “dev”) environments are usually different from production (or “prod”) environments. Because prod infrastructure is designed for heavy internet traffic and is secured against malicious actors. It has expensive features and configurations that aren’t really needed for day-to-day development work. It would not be economical to give each developer their own replica prod environment. So, dev environments are simpler and cheaper. But there’s a problem with this arrangement…

Code that runs well in one environment might break in another one that’s set up differently. So, we want to make sure our local and dev environments are as much like the prod environment as possible.

Ideally, if my production app connects to a PostgreSQL database, I’ll want my local code to connect to a PostgreSQL DB, too. That way, we can minimize surprises that will only surface when we push our updates to production. Because it’s harder and more time-consuming to investigate and fix bugs in “higher-level” environments like prod.

This isn’t unique to software. If you have a big, complex project that requires rehearsals, you’ll want to make sure your test runs are as similar to the live environment as possible.

That way, you’ll face fewer surprises when you “go live.”

A common business example would be a job interview. Many companies struggle with turnover. And hiring the wrong worker is expensive. But a common complaint among interviewees is that job interviews rarely represent actual working conditions accurately. I know I’ve been in interviews where I felt like I was being interrogated by enemy soldiers. But we’re supposed to be collaborators in the actual job. How can an interviewer expect a job candidate to perform as well as they would on the job if the interview environment differs so much from the actual job environment? How can a company expect to find people who fit into their work environment if it’s completely different from the interview environment?

Of course, there are plenty of other ways the YouTube stunt could’ve gone wrong; hence, Nate’s message that “sometimes, it’s just not worth it.”

On irreversible risks.

Software development is mostly about solving problems with software. And every big, complex problem can be solved many ways. Sometimes, we’ll have two solutions that take almost the same amount of effort to implement. The pros and cons between the two options might weigh about the same. They might both seem equally decent. Even though it might be unclear how either solution will meet the customer’s long-term needs. So, which option should we go with?

One tiebreaker to consider is reversibility.

If there’s one solution that’s easier to reverse, then that might be a better option. That way we incur less cost if we decide to swap it out if a better solution emerges later. At least we can change course later.

There are many ways to launch a YouTube channel. There are many stunts one can perform. Firing a gun with a live round at another person is far from the best option. No matter the precautions. Because the consequence of a mistake is irreversible.

On life lessons.

Learning from a tragedy like this ensures that:

1. Lives are not lost in vain.

2. We can help one another prevent similar tragedies in the future.

If you have a big, complex project to deliver, you might prepare with test runs or rehearsals. It’s vital that at least one test run be as similar to the “live” situation as possible to minimize the nastiest surprises when it’s time to go live.

And if you face a tough choice between two or more options, consider the reversibility of each one. Go with the option that’s more reversible. That way, switching to a better option is easier (and at least possible) if one appears further down the road.

Avoid painting yourself into a corner.

Reply to belowthesurfacetop@gmail.com or click “Message Alvin” below if you have questions or comments. I’d love the hear from you.

Thank you for reading. Avoid irreversible risks. And I’ll see you in the next one.