3 Lines Of Python To Create Timezone-Aware Pipelines
Building time-sensitive data pipelines is a challenge; luckily, there is a way to build localized pipelines with 3 lines of Python.
Zach Quinn
Authenticate Google Cloud In Docker Without Getting Your Creds Stolen
Avoid GCP 401 errors — and security concerns — by passing project credentials into your Docker image the right way.
Execute A Multi-CSV Backfill From Google Cloud Storage In 50 Seconds
Leverage the Python Google Cloud Storage and BigQuery APIs to bulk download, transform and upload CSV files in < 1 minute.
The 1 Question New Data Engineers Never Ask (But Absolutely Must)
Either afraid or stuck in old habits, new engineers fail to ask important probing questions; how devs can think critically.
Parse 12 Months Of Credit Card Statements In 3 Minutes
How to use Python to read multi-page PDFs, transform unstructured data and SQL to format the final result in BigQuery.
Set Up A Virtual Environment In A Compute Engine VM In 5 Min.
Set up a virtual environment, install Python & pip and run Python scripts in a Google Cloud Compute Engine virtual machine.
Why Is Authenticating Google Cloud Platform Applications So Hard?
Distinguishing between Google Cloud Platform and a typical API’s authentication process emphasizes the need for secure credential storage.
Beyond Timestamps: Using BigQuery's APPENDS() for Forensic Table Validation
Leverage BigQuery SQL table metadata to deduplicate, partition and delete data — all using only one word.
Deploy Your Google Cloud Functions The Right Way (Step-by-Step Guide)
Covering GitHub versioning, CI/CD pipeline development and scheduling jobs within Google Cloud Platform.
Go From Staging Table To Production Data With 1 Stupid Simple SQL Line
A risk-averse approach to “flipping the switch” from test tables to production tables featuring a subtle BigQuery SQL function.
Why Do My Data Engineering Requests Take Forever?
How data engineers can set realistic development expectations and respond to impatient stakeholders.
Using PyPDF For PDF →CSV Conversion To Find Missing Groceries
Convert PDFs to CSVs in an unusual–but practical–use case of real-life data engineering problem-solving.