Tmux Is Tmagic
Leverage a subtle bash framework to execute data pipelines in the background, saving time for mission-critical tasks.
In non-data news, I recently became a father of a small but very vocal newborn. In case you’re unfamiliar, human children require (both logistically and legally) near-constant supervision.
Sure you can lie a baby in a bassinet, but there is no guarantee they’ll fall asleep, leaving you with any amount of free time. This means, typically, I need to anticipate needs and execute tasks (aka chores) prior to starting my “dad shift.”
But there is one increasingly popular option to supervise your newborn and increase your mobility range from the 3 feet between you, a bassinet and a phone charger — a baby carrier.
It turns out strapping a small human to your chest enables you to complete one task like doing the dishes while another, child rearing, runs in the background.
And the command line equivalent of a baby carrier is tmux.
Ever since a teammate introduced me to the tmux framework, I’ve saved several hours of time and compute-consuming “background” tasks ranging from iterative API requests to multi-terabyte backfills.
Tmux has prevented me from having to work on lengthy weekend backfills and enabled me to free up bandwidth to work on mission-critical tasks.
As a senior engineer, this doesn’t just mean more time to code. I can dedicate specific mental energy to architectural designs, vendor communications and planning meetings. These are the areas of my job where I’m able to make the most significant and visible impacts to my team and org.
Consequently, when paired with a Python-configured VM tmux is one of the most subtly powerful tools for passively running a process that interacts with your data infrastructure.
Build Your Pipeline To A Data Engineering Career
You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.
When you join PipelineToDE, you get:
- The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
- Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
- Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
- The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.
