Packaging And Remote Installation With Pip, Python & Artifact Registry

Neatly package and securely store Python packages in a Python repo with Google Cloud’s Artifact Registry for your cloud builds

Save 1000 Lines of Code With Artifact Registry

Tesla, X and SpaceX CEO Elon Musk, when determining which engineers to jettison from the platform formerly known as Twitter, chose a strange metric. He didn’t request aggregate data of the previous year’s performance reports or hours of sleep sacrificed for the company’s advancement.

Instead, he looked at one data point engineers often ignore: Lines of code.

While the markets will determine the success of X as an organization and platform, choosing this metric was an early misstep.

If you’re new to software engineering and data roles let me be clear: There is rarely, if ever, any correlation between more lines of code and increased functionality.

In fact, I often spend part of my development time trying to reduce the lines of code I write through the process of refactoring.

I would have been fired on Elon Musk’s first day.

Recently, I found myself in a scenario in which I was dependent upon a Python package stored in GitHub but not available for download on Python’s Package Index, PyPi.

To make this code, which acted as an SDK to interact with a vendor’s API, function, I had to run my script in a cloned repo. Once I read the docs and made my tweaks it worked flawlessly in my local development environment.

But this wouldn’t be a sustainable way to store and leverage the code in production. If I were to deploy the code in the same manner I described above I would be committing nearly 1000+ lines of excess code. Some of which, like the license and README, isn’t required for my code to function. Plus, this would really annoy my coworkers, come pull request (PR) time.

At a coworker’s suggestion, I looked into a way to package, or repackage, this code, store it remotely, and install it upon my build’s scheduled run.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

• The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
• Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
• Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
• The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.