Over years, various individuals and teams within the asset management department of a.s.r. had independently developed a range of tools, data pipelines, models, and dashboards locally on their work computers. Upon completion, these components were typically deployed to on-premises servers. However, this process often lacked standardized guidelines for distinguishing between development and production environments.
Recognizing the need for a more structured approach, the asset management BI department aspired to transition from their on-premises data warehouse to a centralized data platform on Azure Cloud. Their goal was to establish a DTAP (Development, Testing, Acceptance, and Production) setup, enhancing the organization of both formal data streams (for regulatory bodies like DNB) and sandbox environments (for business and data science purposes).
a.s.r.’s asset management department sought the expertise of Xomnia to address the challenge of scattered data and to create a unified repository for building, running, accessing, and managing all data and models. This initiative aimed to streamline their data management and leverage the advanced capabilities of cloud-based solutions.
The client’s initial requirement was to draft a design of the architecture needed to create such a platform, from high-level requirement gathering to a low-level design tested in a PoC environment.
The partnership between a.s.r.'s asset management department and Xomnia commenced with an analytics translator from Xomnia leading the initial exploratory phase. This first step was dedicated to understanding and addressing the unique challenges that the client faces.
The analytics translator played a role in enlightening the a.s.r. team about the potential of cloud systems. This guidance included addressing critical questions about optimal solutions for hosting applications in the cloud, identifying the most suitable Azure building blocks, and advising on the best profiles for recruitment to support these initiatives.
The result of this initial phase was the decision to create a data processing platform characterized by a clear separation of environments. This was to be achieved through the adoption of key DevOps principles, such as continuous integration and continuous deployment (CI/CD).
During the first phase, we started with a PoC to validate certain assumptions in the architecture and design of the platform. Following this, Xomnia’s consultant focused on developing CI/CD pipelines. These pipelines were designed to ensure a seamless flow of work from the development phase to the actual production environment. They were specifically tailored for Azure Data Factory, Azure Synapse and AzureML.
Once the architecture was established and the appropriate DevOps pipelines were in place, we moved on to deploying various projects within the cloud environment. Although these projects had different components, they also shared many similarities. To manage the data requirements of these projects, ETL (Extract, Transform, Load) pipelines were set up. This allowed the necessary data to be readily available within the Azure Data Lake.
Furthermore, the models developed in these projects were exported to AzureML and Synapse and integrated into the ETL pipelines. We also connected reporting tools to the new data resources in the data lake, enabling easier access to and analysis of the information.
Adhering to DevOps principles and effectively structuring the MLOps environment facilitated the process of building stable and scalable data pipelines. This forms a good foundation for developing and productionalizing data flows in the future, which the asset management team can use. The data platform is ready for more projects/use cases to be implemented and deployed on the platform.
The anticipated impact of the project is fostering a more data-driven approach for a.s.r. 's asset management team, and enabling them to fully harness the potential of their data. The integration of DevOps principles in Azure will facilitate improved team synchronization, real-time data updates, and robust version control. Ultimately, this will lead to a simpler, more efficient, and transparent work process for both a.s.r.’s employees and their clients.
