Why Is Authenticating Google Cloud Platform Applications So Hard?

Distinguishing between Google Cloud Platform and a typical API’s authentication process emphasizes the need for secure credential storage.

Zach Quinn

— 6 min read
Share
401 numbers above thermostat
Photo by Ashwini Chaudhary(Monty) on Unsplash

Moving from a pre-configured integrated development environment (IDE) to deployment can be as jarring as being jolted awake by a herd of pets or children in the morning (as a pet parent, I have firsthand experience with the former).

One of the biggest hurdles I worked to overcome in my first year of data engineering was understanding how to authenticate both to APIs and to my cloud provider, Google Cloud Platform (GCP). After building 50+ production pipelines, even now, the step I consistently get stuck on is authentication. API Authentication errors are particularly difficult to troubleshoot because error text is limited and status codes go under-documented. Typically an API request that is unauthenticated will return a status code of 401 and, simply, “Unauthorized.” Or, worse, simply: “Bad request.”

# JSON response example. 
 
{"message": "Unauthorized", "status_code": "401"}

Things become even more confusing when a pipeline or application must utilize both an external vendor API and GCP. That’s two separate authentication steps for two separate APIs that must be executed prior to deployment.

Unfortunately, storing credentials in this way makes code vulnerable to manipulation and outright attack.

We’ll ultimately review a script that needs to authenticate to both GCP and a vendor API to make requests as part of an extract-load (EL) pipeline.

While reviewing this code I’ll exhibit the “wrong” way to store credentials to emphasize the need to follow a multi-step authentication process, especially when attempting two-pronged authentication.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

  • The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
  • Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
  • Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
  • The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.

Read more