How Data Engineers Can Use Python to Schedule BigQuery Queries

BigQuery provides guidance for using Python to schedule queries from a service account but does not emphasize why this is an important, if not overlooked step of automating and sustaining a data pipeline.

Service accounts are preferable to personal accounts because service accounts can be accessed by anyone on the team with the corresponding IAM role, meaning that even if someone in the organization leaves, their work can still be accessed, edited and scheduled with ease.

Below, I’ll provide guidance on using Python for scheduling queries and how to handle common pitfalls I’ve experienced in both Python and SQL.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

• The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
• Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
• Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
• The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.