Create Your ETL Pipeline In 90 Min. (A Best Case Scenario)

Creating a data engineering pipeline using Python, SQL and Google Cloud in less than 2 hours.

Zach Quinn

— 5 min read
Share
Speedometer
Build with speed. Photo by Chris Liverani on Unsplash.

The first ETL pipeline I turned in at work was a buzzer beater. Not only was my release date on a Friday, it was the Friday before my wedding. With the blessing of my at-the-time senior engineer, we broke a cardinal rule of data engineering and merged on a Friday.

Joining my current team in its infancy (I was the second full-time hire for the new, official data engineering team) means that, inevitably, I would build a lot of pipelines. Basically connections hitting lower-priority vendor APIs that the seniors didn’t have the bandwidth to work on. But, as I gained experience, I was able to take on higher-priority projects and, soon, I had the very unofficial and very specific distinction of building the most ETL pipelines in a fiscal year.

With volume and “reps” inevitably comes speed, or at least less roadblocks.

Which is how I was able to build out an end-to-end connection to my target API, ConvertKit, in under two hours.

To be clear, what follows is the best case scenario for any data engineering development process. I had my own requirements, possessed incredibly clear API documentation and, most significantly, I didn’t have to wait for any reviews or QA.

I hope seeing my thought process is helpful and can maybe demystify the intimidating idea of building an ETL pipeline.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

  • The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
  • Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
  • Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
  • The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.

Read more