Stop The Bleeding: 4 Strategies To Troubleshoot, Triage Data Anomalies

Quickly identify, isolate and fix malfunctioning data pipelines for quality data, happier stakeholders and a stress-free workday.

Zach Quinn

— 6 min read
Share
Someone wrapping a hand with gauze.
Put a bandaid on your broken pipelines. Photo by Slashio Photography on Unsplash.

The Usual Suspects Cause Anomalies In Your Data

Only in data science does a mistake make the company look better.

Spikes in user activity, millions more rows of mineable data and inflated revenue values.

On the surface, these all sound like good things. In more volatile fields like finance it’s rare but still plausible for an investment banker to approach a manager, say an investment’s returns increased by 5x overnight, and for the manager to not think anything of it.

If you say that to a data-minded executive, in their mind, your voice will fade and the only sound they’ll hear is alarm bells.

This kind of strange, out-of-nowhere variance in data has a name: Anomaly.

Looker time series graph.
This engagement ratio should be normalized to be out of 1.0. The fact that values spike to 1.5 is concerning and would be considered anomalous. Data: My own. Screenshot by the author.

Companies with solid data infrastructure incorporate upstream and downstream checks for anomalies to ensure that the data that is delivered is clean, timely and, above all, accurate.

But such detection systems aren’t intelligent enough to just “know” how to spot an anomaly. These systems become more reliable as their underlying models are trained over time and on increasingly vast sources of data.

So if your model is unqualified for the job, who identifies, investigates and troubleshoots anomalies in newer data sources?

You.

For a newer data engineer, getting a message like “I don’t know what’s going on with this data” can be a bit intimidating, even if it’s something you partially or entirely built.

Luckily, as you gain experience investigating data anomalies, you start repeatedly encountering the same “usual suspects.”

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

  • The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
  • Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
  • Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
  • The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.

Read more