Overcoming The Final Hurdle of Data Automation With Fewer Failures

Learn the components of data pipeline production to take your ETL build from code to cloud with automated, actionable results.

Zach Quinn

— 7 min read
Share
Sprinter jumping over hurdles.
Overcome those development hurdles. Image courtesy of Unsplash.

The Development Practice You Take For Granted

I’m the embodiment of the meme in which a developer spends hours automating a relatively simple task. In other words, while much of the world is increasingly apprehensive of replacing processes with AI, I’m still pro-automation.

Drake in a meme with accompanying text.
Image courtesy of starecat.com.

And while I’ve developed some pipelines outside of work to serve my own needs or to help out a friend, I still struggled with one very important aspect of each ETL build.

If you’re reading this, I imagine you might struggle with the same issue.

Deployment.

To be clear, at work in nearly two years I’ve written thousands of lines of code, created probably 50-ish pipelines and written CI/CD processes ranging from cloud function deployment to Docker image updates.

When I wanted to replicate some of these processes with my own builds, I experienced a lot of failure.

This was due to two main reasons that I would guess most new/new-ish data professionals also overlook in personal projects:

  • I didn’t take the time to set up a proper environment in GitHub
  • I certainly didn’t write proper CI/CD pipelines.
Errors in Google Cloud Platform console.
Errors in GCP during a recent personal project deployment. Image courtesy of the author.

After struggling to replicate some of the best practices I do, reflexively, at work, I finally took the time to really understand what it means to “production-ize” something.

As a result, I now have two cloud functions running in my personal GCP project.

But, with a surplus of documentation and lack of tutorials that explain the “why” of deployment, getting to that point was frustrating.

So this is an effort to help you and, to an extent, myself, better understand the anatomy of deployment.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

  • The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
  • Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
  • Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
  • The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.

Read more