One of the main benefits of using cloud over on-premise data servers is that clouds increase the speed of development, since they provision the necessary infrastructure and services within minutes. This feature is of growing importance, as the amount of data that companies handle continues to grow exponentially. Without ever expanding storage and computational capacities, companies will struggle to maintain their efficiency, speed and competitive edge. On the contrary, clouds can easily or even automatically accommodate extra storage and/or computational power, unlike on-premise architecture where expanding scale requires more (manual) effort.

In the past, companies hosted most of their data and tools in ‘on-premise’ data servers or warehouses, which they created themselves. Storing and working on data and tools on-premise, however, requires many in-house engineers to constantly maintain the IT infrastructure. These engineers also need to worry about purchasing server equipment themselves, and constantly maintaining and updating the on-premise data storage facilities.  

Each of the cloud vendors offer similar services to work with data, such as:

• Databases / Data Warehouses

• Data storage /  Data Lakes

• Services to run data pipelines and do data transformation

• Services to schedule data pipelines

• Services to perform data analyses and machine learning

• Services to support data governance

There are a number of common strategies for cloud migration. Usually, for different components of the existing platform, different strategies are chosen, leading to an overall hybrid approach of the following strategies:

1. “Improve and Move”: This is the preferred way of migrating to the cloud. In this migration strategy, you modernize the existing component, taking advantage of cloud-native products. It is an intensive approach, but it also has the greatest benefit - better performance, cost, features and maintainability. But, we want to stress again that it is intensive, which means that you need knowledge of the cloud-native products, strategies and architecture, and to implement all of them together.

2. “Lift and Shift”: This is the less intensive way of migrating to the cloud. In this strategy, components are moved into the cloud with only minor modifications. Many companies follow this strategy, and the clouds provide Platform as Service (PaaS) solutions to support them. This approach requires a lot less cloud knowledge and architecture than the “Improve and Move” approach, and implementation is relatively fast. But while you benefit from the scale benefits of the cloud (cost-wise), you do not benefit from modern, cloud-native components.

3. “Remove and Replace”: This strategy takes full advantage of cloud benefits, but also requires the most knowledge and work. It is a full reboot - the current platform is simply replaced by a novel, cloud-native platform.

4. A mix of all the above:  An example of mixing the strategies can be considered. For example, one can “Lift and Shift '' several ETLs, now writing their data to cloud storage. However, the existing on-premise data server is retired (“Remove and Replace”) and a new cloud-native data lake house can be built. However, the A/B testing platform might need minor modifications, and therefore should be moved into a serverless cloud function setup (“Lift and Shift”).

• Lack of expertise: There is a shortage of cloud knowledge in the market. Carrying out a cloud migration requires a wide variety of expertise domains, such as security, project management, data engineering, architecture and cloud engineering. 

• Getting solid migration timelines and evolving business requirements: Because of the many dependencies, getting concrete about deadlines is difficult. For instance, a data engineering team cannot move forward without a source connection, whose highest priority is keeping the business rolling - not a migration to a new platform.

• Data security: When migrating to a new platform, you should treat security as a first-class citizen. Make sure to block sufficient resources to carry out vulnerability scans, automatic software deployment and comprehensive testing. 

• Overhead of running two data platforms at the same time: Only after the new data platform is up and running can a migration happen. A cloud migration, however, is a tedious process that might require a long time. After all, it involves migrating the infrastructure, processes and data to the cloud. Moreover, both the old and new platforms need to be running at the same time, since the business might rely on the data in the original platform. In addition to this, data is not allowed to be lost and the ingestion pipelines need to stay up and running. Consequently, a migration will probably demand resources to keep the old platform alive in production, while working and integrating the same sources onto the new one.

• Validation of functionality in the new platform: When building a platform on the cloud, it often means that operations will be done differently than in the past; services used in the old platform might be deprecated, outdated or just not best practice anymore. This means that different tools and services have to be used. 

• Insights into current limitations: When migrating a platform to the cloud, the old platform needs to be evaluated in full. This might be the first time this review is done in quite some time. As a result, limitations in the current data, business processes or strategies might arise. This can radically affect further project planning and business operations.

Consequently, a validation process is crucial to make sure the data in the new platform is identical to that on the old platform. This step is critical, since there is a big chance the users of the old platform know the data in and out, and will not trust the data on the new platform if it is any different from what it used to be.

• Cost reduction: This applies to infrastructure management costs, as well as management costs, which will be further reduced due to the ease of setting up and outsourcing management to cloud vendors. 

• Scalability: Migrating to the cloud unlocks scalability in a way that an on-premise system simply cannot accommodate. By migrating their work to the cloud, companies can increase their computational capacity 10 times with one click, which, working on-premise, would require ordering new hardware, waiting for it to be delivered, and then connecting it.

• Flexibility: Suppose you need a temporary scale increase - for instance, your organization needs a lot more resources for a short period. With an on-premise data center, you have to invest in hardware that stays. However, with a cloud setup, your scale may be increased temporarily and the computational capacity can be downscaled again, either automatically or by changing a few settings. Besides decreasing costs, migration increases flexibility, as you can explore new machine learning or big data analytics more often because they do not require additional setups or investments.

• Reduced incident management: With cloud vendors providing high service level agreements (SLAs) on mission-critical infrastructure, incident management (restoring normal operations in the fastest way possible after an interruption with the least possible impact) will become a thing of the past. This guarantees (almost) no downtime of your heavily accessed databases or authentication mechanisms.

• Financial costs: There are considerable costs to migrating to a cloud that have nothing to do with the charges of a cloud vendor themselves. Such costs include finding and/or investing in the necessary human resources that are familiar with your cloud of choice. Chances are that your current system administrators or IT-teams will be knowledgeable on the presently-used systems, but not on new cloud platforms.

• Time costs: Migrating operationally critical applications requires extensive testing to ensure that functionality doesn’t break, which, depending on the type and number of operations, might add up to several months or even years of full-time attention.
• Complete infrastructure overhaul: This requires (among others) changes in software, network configurations, deployment pipelines and security frameworks (except in the case of a Lift & Shift).
• Looming vendor lock-in: By committing to a single provider to host your cloud data platform, you become highly reliant on that provider. By putting such trust in a provider, you make your organization vulnerable to its sudden price increases, and make the operationality of your services and their maintenance and quality strongly linked to the cloud vendor. This lock-in situation is mainly due to the high costs of migrating away from a provider. There are several ways of avoiding a vendor lock-in, such as making your infrastructure interoperable over clouds by preferring cloud-agnostic solutions over provider-managed solutions.

There is no one-size-fits-all answer to this question. The best cloud provider for your business is the one that fits its requirements best. Fundamentally, there are three components to be on the lookout for when determining the suitability of a cloud provider with your business:

• Price: Xomnia recommends considering different solution architectures and checking and verifying them externally. We recommend making extensive use of the pricing calculators to estimate your cloud bill, which are provided by cloud providers such as AWS, Azure, GCP. Because of the pricing structure, it might be that one cloud is cheaper for your use case. Do not forget when making the calculations to factor in scalability and initial migration costs. 

• Setup: Which cloud platform offers the most easy migration process for your applications and data? And which allows you to get started quickly and rapidly prototype? The answer to those questions is important in determining the best cloud provider for your business. 

• Expertise: Ask yourself questions like: What platform is my technical personnel most familiar with? Which one would they like to learn? Can it be easily integrated into existing enterprise software? Is there perhaps a platform where more people work, offering a larger basis of (future) personnel? The answer to those questions is important in determining the best cloud provider for your business. 

• Use case(s): Certain providers have more mature offerings for specific use cases than others. For example, AWS offers 2 solutions for big data platforms (Redshift and Athena), which tie into different use-cases. On the other hand, Azure offers a more integrated Data Lake(house) service (Synapse). GCP, however, offers a high level of integration with various (third-party) services and managed hosting, e.g. Apache Airflow hosted as GCP Composer. This ties back into expertise, since your team will want to tackle the use case in the space they are most familiar with.

There are obviously other considerations to take into account when choosing the best cloud provider, such as the availability of professionals who are experienced in a certain cloud provider, the ease of attracting talent that has experience with a provider, the accessibility and comprehensiveness of documentation on setting up and configuring cloud services, or the possibility of integrating a cloud with the other tools that are used in the organization. A common example for the last is the Microsoft Suite, which ties in nicely with the Azure ecosystem.

Any data platform that sits on top of modern infrastructure is just seconds away from tapping into its immense potential through machine learning. Most cloud vendors offer managed machine learning services that will ease the complex machine learning lifecycle of training, evaluating and deploying models. 

Given that your platform adheres to the principles of scalability, security (i.e. AI is used responsibly) and governance, machine learning engineers and data scientists can start exploring the potential of your organization’s data. 

Components of a platform that are required by a machine learning solution are:

• An automated MLOps pipeline: It is responsible for the training, deploying and monitoring algorithms. A solid pipeline will allow data scientists to easily prototype and perform testing. Moreover, they could also monitor model performance and corresponding metadata or model drift. This is the “machine learning in production” feature: The slow - or sometimes fast - changes in the world are reflected in the input data of your model, the model performance and other measures of the model's fit. 
• Scalable data sources to access data from: These could include data warehouses, data streams or file storage. Machine learning is a data-hungry process, and the more data, the more valuable it can be - as well as more challenging to implement effectively.
• An integrated environment to collaboratively develop machine learning models: Whether this is an integrated environment (such as Databricks, Azure ML, Google Colab or AWS Sagemaker) or a combination of local development and CI/CD pipelines, code management should not be a blocker for data scientists.
• Sufficient computational resources to train memory-hungry machine learning models: The automation of deployment and monitoring of costs are essential for making machine learning profitable for your organization.
• A big data processing engine: It performs large-scale transformations on raw data to prepare it for the machine learning pipeline.