Tackling the Billion Dollar Tech Debt Problem

Data Unfiltered

Tech Debt is a Silent Plague

Tech debt is a billion-dollar problem that affects everyone regardless of organizational level.

As tech debt piles up, it becomes difficult to unravel and sometimes impossible to eliminate in organizations.
 
There's a simple reason for that: it is easier to ignore tech debt and leave it unresolved.

Not dealing with tech debt:

• Takes less time.

• It is easier to implement solutions.

• Gets issues and problems resolved quickly.

• Introduces a small piece of tech debt over time.

The alternative path to resolving it:

• Needs buy-in from all parties.

• Has longer cycle times for development.

• Requires more learning by everyone involved.

• Develop a more sound, future-proof solution.

While it’s easy to understand why people opt for the simpler, short-term fixes, as that tech debt accumulates, it creates expensive long-term problems.

• Some calculate that teams spend 23-42% of their development time tackling tech debt.

• Another McKinsey study found that some companies could generate $200-$300 million in benefits by addressing their tech debt.

• People may quit and leave their teams to deal with their old tech debt with no documentation.

• Several cycles of 6- to 7- 7-figure consulting engagements are needed to unravel tech debt across the organization.

• Product consistently breaks with teams becoming overwhelmed with fixes instead of focusing on new initiatives.

• New product implementations require migration of legacy systems that are impossible or have longer than expected cycle times.

Yet, even though tech debt can result in millions, possibly billions of dollars, from operational costs, revenue loss, resource attrition, customer churn, and product failures—many companies, teams, and individuals rarely address it.

How Tech Debt Takes Form (and How to Resolve It)

Tech debt isn’t something that comes up out of nowhere. Instead, it tends to be a symptom of a “leaky faucet” that everyone ignores, eventually flooding the entire house.

This “leaky faucet” comes about in a variety of ways:

1. Strong egos are anchored to built products, making change difficult.

2. Leaving the team unprotected to excessive ad-hoc requests.

3. Being reactive instead of proactive on tech debt.

4. Zero investment in reducing tech debt.

Strong egos anchoring to build products, making change difficult.

Humans are walking paradoxes when it comes to technology.

Most of us will upgrade our phones, computers, and TVs when new ones come out. Yet, that 10-year-old code base that was developed when we first launched the company — yes, that’s the bedrock of our data infrastructure, so we don’t need to upgrade or change anything.

The attitude towards “protecting” previously developed work (or the person who created it) is the number one item that plagues most organizations.

The person who developed it is typically protected by leadership because:

1. They created an innovative solution that helped the company get to where they are today.

2. They manage or maintain the data infrastructure and must approve any changes.

3. They are leadership.

The problem with the notion that this code is infallible is it was usually developed as an ad-hoc request because it was “what worked” at the time. Then that evolved with patch fixes over time, but no one has taken a holistic view to see that it is an accident waiting to happen.

If you must upgrade your phone firmware after 2 weeks because of a minor patch fix that resolves a flashlight issue, you should probably upgrade your 10+ year-old code that supports a billion-dollar enterprise.

Leaving the team unprotected to excessive ad-hoc requests.

The most dangerous note a person can receive on the team is, “It’ll just be a quick fix.”

1. It is never a “quick fix.”

2. The “quick fix” now displaces the planned work.

3. Most “quick fixes” result in the creation of tech debt.

4. Those “quick fixes” are a slippery slope into an endless stream of ad-hoc requests.

Leaders must protect their teams and create the proper intake process for these ad-hoc requests to prevent this.

I break down requests in three ways:

1. Bugs - these are problems with current solutions that need to be addressed immediately because they are causing issues with production reports, pipelines, and models.

2. Additions - these are iterative improvements for existing solutions and should be planned for depending on the effort needed and impact (ROI) on the business.

3. New development - these initiatives do not have an existing solution and should similarly be planned for depending on the effort needed and impact on the business.

The first request can displace planned work, but the last two should not.

If you let the last two requests interrupt work consistently, you are enabling ad-hoc requests, turning off the development of long-term solutions, and ultimately creating the perfect environment for unresolved tech debt to run wild.

Being reactive instead of proactive on tech debt.

Remediating previously developed tech debt helps resolve your current state problems, but it does not allow you to prevent it from occurring in the future.

It’s the equivalent of cleaning your house only when guests come over and then letting the same poor processes get the house dirty again as soon as those guests leave.

You are treating the problem instead of developing a solution to stop the problem from happening or worsening over time.

For example, if you used v2, v_final, vfinal_v2 instead of standardizing the naming conventions:

• Being reactive would mean you removing the previous versions and maintaining the final version — there problem solved. But then someone creates v3_final the next day, and you must do the same exercise again.

• Being proactive would mean that you leverage a version control tool (git, SharePoint) to maintain the version history without the need for an endless amount of suffixes. Now, no one has to identify the most recent version and can revert to a prior build if the file is corrupted or the pipeline is broken. Now, the problem is solved, and you’ve set yourself up for future success.

Zero investment in reducing tech debt.

Whether you are an executive, a people leader, or an individual contributor, everyone should focus on reducing and preventing tech debt.

Unfortunately, this is not the case at most organizations because everyone is focused on “what’s next,” and “what happened” isn’t relevant unless it breaks down and impedes “what’s next.”

Each role can take a few proactive steps to ensure that tech debt is being addressed and prevented from occurring in the future.

For executives:

1. Provide teams with the resources, time, and space to resolve their tech debt.

2. Prioritize process vs. innovation appropriately for each initiative.

3. Create and enforce enterprise standards.

4. Educate on the fundamentals. 

For people leaders:

1. Analyze the team's data products regularly to identify tech debt.

2. Make time to resolve tech debt before jumping into new initiatives.

3. Prioritize intelligent design over quick cycles.

4. Emphasize the fundamentals.

For individual contributors:

1. Remove the ego from your work and focus on developing the best product.

2. Identify how to help debug others' work.

3. Consistently revisit and refine your work.

4. Learn the fundamentals.

You won’t prevent all tech debt from occurring, but by taking these steps, you can likely reduce the 23-42% of development time spent on tech debt and focus on building for the future rather than always remediating the past.

Data in the World

CES Highlights that Not Everything Needs to be AI-Driven

The Consumer Electronics Show is the first conference each year to highlight the current trends in tech and innovations on the horizon.

Given the hype cycle we’re in, it is no surprise that AI is everywhere—even where it shouldn’t be.

Layoffs, Layoffs, and More Layoffs

2024 is picking up right where 2023 left off, with layoffs hitting many companies—especially Big Tech.

• Amazon has now cut positions across Audible, Prime Video, and Twitch.

• Alphabet (Google) laid off 1K+ to gear up for AI hiring.

• Citigroup is cutting 20K roles after hitting lows in fixed-income trading revenue.

Although many layoffs are occurring, and people are documenting it, I expect many will be reshuffled to other companies rather than be out of work for extended periods.

The great reshuffle continues.

Celebrity Influencers Teaching Mathematics

What if your favorite celebrity influencer taught you calculus instead of a dry professor?

Or if Einstein taught you physics?

Kim K and Taylor Swift take their shot at teaching mathematics, which reimagines the way we may learn in the future.

Data Career Tips

In the Interview Process, Analyze Where You’re At

Interviewing and going through the process can sometimes feel like a full-time job.

You’re sending out constant emails, crafting the perfect resume and cover letter, spending 30 minutes to an hour on calls, practicing negotiating, and more.

It is easy to get consumed by all these steps and not evaluate your performance, where you’re stuck, and how to improve that specific portion.

Traditional steps in the process are:

1. Submitting an application.

2. Interviewing with a recruiter.

3. Interviewing with the hiring manager.

4. Interviewing with the divisional lead or executive.

5. Receiving an offer and negotiating/accepting/rejecting the offer.

If you aren’t getting recruiter calls, it likely means:

• Your resume or cover letter isn’t good enough.

• You are applying to jobs that aren’t a fit.

• You aren’t standing out as an applicant.

If you aren’t getting to the hiring manager interview, it likely means:

• You aren’t selling yourself and how it ties to the role well.

• Your experience isn’t resonating with the recruiter.

• You haven’t practiced interviewing enough.

If you aren’t getting to the divisional lead or executive interview, it likely means:

• You aren’t connecting the dots between the role and your experience.

• You don’t come across as a good team player or seem coachable.

• You aren’t specific enough about examples from your past.

If you aren’t getting offers:

• You fail to see the big picture items and how your role helps.

• You don’t seem like a culture fit.

Each step in the process allows you to analyze your performance and evaluate what is needed to proceed to the next step.

If you’re stuck on a particular phase, figure out why and adjust accordingly.

Datinuumber of the Week: 65

Nearly 65 years ago, Dr. Martin Luther King Jr. was shot in the Lorraine Motel in Memphis, Tennessee.

18 years later, on January 20th, 1986, President Reagan signed a bill making today, MLK Day, a federal holiday.

Thank You

Thank you for subscribing and reading this week’s newsletter.

If you enjoyed the newsletter, the best way to help is by sharing it with colleagues and friends.

If you prefer to listen to the newsletter, the Datinuum Podletter will be released weekly on Apple, Spotify, or your favorite podcast player.

Feedback is a gift. Please reach out to [email protected] with any feedback or questions.