I’m sure there are lots of personal stories about career growth and achievements, but just like a roundtable, there’s always room for one more. In this story, I want to tell you more about my journey on the path of the Data Engineer.
In the past two years I’ve been rather busy working on various projects - exploring cloud architectures, working with seemingly alien data-processing and data-management tools, paired with loads of fun with Python. My journey led me through the corridors of Microsoft Azure, Amazon Web Services and Google Cloud. I’ve had to untangle the mysteries of hundred-line SQL queries, fought my way through the peculiarities of the ‘null’ value handling of Python and last but not least - created ETL pipelines out of CSV files. It sounds scary to data engineers, I know, but bare with me. The path of the data engineer is a long and winding one. It does not get as much marketing and glamour as the one of the glorious data scientist, who strikes his analytical sword through the layers of data and uncovers stellar predictions. But don’t be disheartened, I’ve found it to be a great journey for the right candidate, which will leave you fulfilled and with potential anxiety stemming from all the code you have in production, battling with all the forces of nature.
My adventure into data engineering started by a stroke of luck, paired with a passion for problem solving. I moved to Amsterdam three years ago to study Computer Science at the Vrije University of Amsterdam for the promise of riches and great hacking skills, which seemed like a win-win in this evermore computerised world. Up until this point I had about 8 years of experience in various jobs and industries, but never felt truly fulfilled with my work. I was not creating things or solving meaningful problems and my understanding of the world wasn’t improving as much as I had hoped. My interests have always been in the natural sciences, especially physics, and I am unbearably curious about everything that makes the world tick. So, naively, I believed Computer Science would be a road that leads me down a good career path, but away from my interests. Well, it’s hard to overstate how wrong I was… Turns out, that code and algorithms are extremely powerful at explaining the world, as our universe is all about information. Go figure.
Thus, I ended up training with the most powerful chisel to dig into any mystery I want to solve and it quickly dawned on me that everything arounds us is just systems within systems, bound by logic and explained through information. Slowly, but surely, my skill in working with these systems became better and soon enough I realised that I had the tools to get a crack at real word problems, given enough time and resources. After the first year into my education, I felt hungry to get my hands dirty with real world use cases and help find answers or build systems that are needed. This is when my luck struck again - I joined a student’s “open door day”, featuring a boat trip at Xomnia.
The self-driving boat of Xomnia made me pretty excited about the company - there’s innovation, excitement, and, frankly, it’s hard not to want to be a part of it. Still, I did not believe that Xomnia would need someone as early into their career, as me. But they did. Xomnia was looking for a part-time data engineer for development and maintenance work and I thought that fitted my situation and skills perfectly. Of course, it wasn’t immediately apparent to me what data engineering fully encompassed, but I had the hunger for problem solving, so it was natural to be brave. I started working in a scrum team, ready to learn and help my senior colleagues build a recommender engine. It was intense, but it felt fulfilling. I was happy to see that my interests, skills and education turned out to be a great fit and I quickly started working with things like Docker, Kubernetes and the cloud. Since data engineering is all about building systems that, hopefully, end up working as well as a swiss watch, tinkering and problem solving is the core of this profession and an everyday occurrence. You have to be hungry to solve problems - they pop up by the hour. As you might have guessed, that was exactly my cup of tea. Paired with a great team of colleagues working on the project, it’s hard to ask for more. Now, two years later and with a recommender engine, trading portal, tender optimizer, AWS data lake and an odd but fun BI reporting assignment under my belt, I feel eager for the next project. It’s always exciting to build the next system, solve the next business problem and get to know the world a little better.
If this strikes a chord with you, then do take it as your call-to-arms. The need for this profession increases rapidly. Grab your machete and join me through the jungle of systems, on the path of the data engineer. It will be tough, weary, sometimes strange, but it’s worth it. The future is waiting for us to build it.
“We do not follow maps to buried treasure, and X never, ever marks the spot.” - Indiana Jones
