Embracing Agile: Lessons from a Data Engineering Bootcamp

I anticipated that the Data Engineering Bootcamp at Northcoders would be fast-paced and challenging. What I didn’t expect was the intense and rewarding social experience of working together with other trainee data engineers to learn, solve problems, and deliver a complex and polished product in the final three weeks.

I had heard of pair programming, agile, and scrum, and I’d even read up on them and tried to incorporate some of the principles of agile into my work as an architect. But there’s nothing that compares to rolling up your sleeves and getting stuck in with a fellow coder, sharing challenges and celebrating breakthroughs. This is all old news for anyone already working in the tech industry, but I wanted to record my enjoyment of the process at the start of my journey in a new industry.

In completing our final ETL pipeline project, here are a few of the key elements that I think helped us succeed:

• Questioning and clarifying the brief at the outset. As an architect, collaboratively devloping, refining, and crystalising a brief comes naturally, and I love the proccess of helping a client or stakeholder really question their own needs and motivations to demand the best product.
• Clarity in system architecture and interfaces early on Each element of the system was described in great detail before writing any code. Their inputs, outputs, possible errors, data types and formats, payloads. Individual functions were scoped and docstrings written at the outset, so when a pair picked up a ticket, they could focus on the task at hand, with confidence that if they could meet the spec, the gears would mesh and the system would work.
• Flexibility in teams and reactivity to blockers Regular communication and collaboration were crucial in overcoming obstacles. When significant blockers, bugs, or technical hitches arose, the entire team could pitch in, ensuring that development progress was never stalled. This agility and responsiveness were instrumental in maintaining momentum and delivering the project on time.

As I transition into the tech industry, I am excited to dive deeper into this collaborative and agile way of working. The Data Engineering Bootcamp has not just imparted technical skills but has also instilled in me a profound appreciation for the power of teamwork, communication, and adaptability in software development.