How to Use Python & SQL to Append New Rows to a BigQuery Table Without Overwriting Your Data

Although Google’s BigQuery is intended to be an append-only database, there are use cases in which it would be necessary for data engineers to update only a single row based on new data.

However, getting to this point requires some Python and SQL hacking since BigQuery only supports truncate and append methods separately.

While you could ingest results into one table and load them into another, there is a way to append and truncate simultaneously using the BigQuery API with Python.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

• The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
• Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
• Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
• The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.