The City of Amsterdam joined hands with Xomnia and other consulting partners to embark on a journey to reshape its data architecture and advance its data platform. This partnership marked a significant milestone in The City's pursuit of technological advancement, and sets a compelling example for cities worldwide seeking to harness the full potential of their data resources.
Our collaborative efforts reshaped the data platform’s tooling and architecture at the client, spearheaded by Data Architect Ozan Dogu Tuna and Data Engineer Roelof Roessingh. The outcome was translating Data Mesh concepts into an architecture that The City can deploy on a wider basis. This forms a stepping stone in the eventual transition of The City to operate in a cloud-based infrastructure.
"Ozan brought a lot of experience by setting up an architecture on Data Mesh principles. He was able to translate the high level intentional architecture to concrete features for the teams to work on." The City of Amsterdam
Challenge
Amsterdam's city-wide data platform operated within an on-premise system. This stood as a challenge in the face of The City’s ambitions to expand and achieve its full data-driven potential. To overcome this, The City wanted to shift to a cloud-based paradigm, migrate to Azure, revamp its data platform's tooling and architecture, and enable self-service analytics. However, this vision posed intricate challenges.
The City’s initial steps towards migrating to the cloud heavily relied on Infrastructure-as-a-Service (IaaS) components. Attempts to bootstrap new solutions atop this framework strained platform teams, causing delays, service quality issues, and maintenance complexities. The data teams, entrusted with making data accessible to the public and enabling self-service analytics, grappled with these impediments.
Overcoming this infrastructural puzzle required a holistic reimagining of The City's data landscape. It entailed a shift in strategy, mission and vision, as well as necessitating a profound embrace of Data Mesh principles. These principles focus on domain-centralized ownership, treating data as a product and implementing data governance.
Xomnia played a pivotal role in The City’s journey to achieve this. This came following Xomnia's help with lying down the blueprint for a cloud-based architecture at The City from 2015 to 2020. In 2022, the City sought Xomnia's architectural and platform expertise to translate plans into action.
Solution
Xomnia assigned Data Architect Ozan to work alongside The City's team in orchestrating a seamless transition to the cloud. His year-long assignment played a crucial role within the Security Architecture Services (SAS) team. Data Engineer Roelof assisted the platform team that built the infrastructural components of the new design.
Starting the project demanded a comprehensive enterprise architecture overhaul, beginning with setting the mission and vision for the new cloud platform. This top-down approach traversed enterprise and contextual architecture, setting the stage for transformative changes, and transitioning to working with the Data Mesh concepts.
One of the pivotal challenges was to deliver the new platform efficiently while preserving the legacy system's functionality. This necessitated the introduction of new architectural patterns that emphasized a cloud-first approach. This was no small feat considering the vast domain size of The City of Amsterdam.
Ozan's expertise was instrumental in navigating these complex transitions. He helped in bridging gaps between the teams and 50 domain experts who used the minimum viable product versions of the platforms.
He contributed towards ensuring a deep understanding of the data teams' needs while helping to guarantee that the delivered data platforms met these requirements, were future-proof, and adhered to a well-defined architecture. His contributions encompassed:
- Clarifying the division of responsibility and designing architectural patterns to enforce domain-centralized ownership.
- Elevating self-service quality and availability by transitioning from Infrastructure-as-a-Service to Software-as-a-Service.
- Managing data ingestion and making data accessible for analysis.
- Establishing robust data governance, treating data as a product, and ensuring data's security, privacy, sharing standards, and interoperability.
- Successfully executing the MVP stage.
“The adoption of the 'safe methodology' facilitated the development of each platform through smaller bi-weekly iterations, with quarterly milestones setting the agenda for the next quarter's goals,” says Ozan. “Architecture played a crucial role in these planning phases, and we dedicated a quarter to designing and reaching consensus regarding the upcoming steps."
In the platform team, Roelof helped design the code-base for rolling out a standard solution to multiple teams. This meant working closely with the cloud infrastructure team to set up networking, designing the modular Terraform code-base and writing IaC pipelines from scratch. All pipelines had to be designed in such a way that they are automatically triggered by a pull request but can also be run manually for ad-hoc changes.
Impact
Xomnia's contributions brought The City of Amsterdam a step closer to the establishment of a more resilient data infrastructure and data ecosystem, both from an architectural and organizational perspective. By May 2024, more than 20 data teams were already onboarded on the new platform.
Ozan and the team of architects successfully executed the first MVP stages across three distinct platforms. These MVP products were positioned as viable enough to replace on-premise workloads, marking a significant step towards the cloud-based platform's full adoption. The ultimate objective remains to onboard the entirety of the client's domain in the years to come.
Solution Architect Richard Kooijman joined The City starting Q3, 2023. His role focuses on supporting the platform teams and constructing use cases atop the data infrastructure. Stay tuned for further updates as the client's data transformation journey continues.