How I’m Using Data Engineering To Help A Wall Street Banker Pick Better Stocks (Part I)

“Random Question — Can You Get Earnings Data On Every Publicly Traded U.S. Company?”

“Hey man, is there a way to automate this?”

I look up from this weekend’s bed (AKA my friend’s couch) to see my childhood friend, an investment banking analyst, copying and pasting a company’s earnings data into an Excel document on his dual-screen desktop.

As a data engineer, I cringe.

There is so much wrong with his setup (aside from the screens, that is).

The Excel spreadsheet, the manual copying/pasting of numbers, the fact that source data is contained in a PDF and not a reliable stream like an API.

After a bit of back and forth, it becomes clear that my suggested automation approach wouldn’t mesh with his work set up.

My intervention would clearly violate several company policies.

The phrase “security risk” gets tossed around.

Then, a new request.

This time for a side project.

“Would you be able to do some tech stuff to get me data on stocks?”

“Yeah. That’s pretty doable. What do you need?”

“To be honest? — Every publicly traded US company.”

I sit up. Suddenly the back pain from the couch and travel fades.

I fire up my laptop.

It’s time for some fun.

Build Your Pipeline To A Data Engineering Career

You’ve reached the limit of the public preview. The full version of this post includes the implementation details: The code, the edge cases, and the "why" behind the architecture.

When you join PipelineToDE, you get:

• The DA → DE Pathway Course: A structured roadmap to bridge the gap between analysis and engineering.
• Weekly Senior Deep Dives: Fresh, tactical insights on Python, Cloud (GCP/AWS), and modern orchestration delivered every week.
• Production-Ready Blueprints: Access to 80+ protected stories and code repos from my time in the trenches as a Senior DE
• The DE Job Board (Coming Soon): Exclusive access to a curated board of high-agency Data Engineering roles.