Database migration can be a challenging task for companies using legacy software and implementing age old IT practices. Planning an IT refactoring strategy, shifting from traditional on prem database to cloud, adopting new tools, languages or even changing the architecture come as additional responsibilities associated with a successful migration.
When is it high time for database migration
However database migration is inevitable for companies using on prem legacy data warehouse which will soon become inadequate to store large and diverse proprietary data generated on a daily basis. Unstructured data generated by business is a treasure trove of business intelligence. Therefore, it is essential therefore to take stock of the existing data management scenario at an organization for the following signs and assess the need of data migration.
1. Data retention is becoming an issue
As more and more services get created by the organization the pressure to segregate, filter, and manage data increases, not to mention the increased variation of data. At one point of time the struggle is to effectively scale and retain data. Legacy warehouses are really bad at scaling while maintaining speed. Lot of optimization and A/B testing becomes a constant partner to the process and makes it overwhelming.
2. You are in ETL hell
Often legacy data warehouses maintain a relational database like SQL database. Joins and aggregations while dealing with billions of records in a SQL database can be a painful task. Even joining two tables several times of the day can slow the database. A surge in data can lead to failure of ETL jobs. All this contributes to overall downtime of the database, degrading query performance and making users unhappy. This is a time to look for data migration to a more practical solution.
3. You’re constantly resizing and reconfiguring your DB cluster
Databases are an expensive IT commodity. The IT budget of an organization may not be sufficient enough to constantly buy new server space to accommodate the pilling up data. One hack that is frequently used is to compress the data while adding additional machines. Of course, this is easier said than done as adding another machine is no mean feat. This will require the majority of workload of DBA being occupied with constant reconfiguring of DB clusters and load-testing of servers. An alternative approach is to figure out how to store raw data and analyzing it later adopting a data lake.
4. The business expects better latency than you can deliver
Latency is a killer when it comes to the world of IT. Often if you can’t use data in real-time the insight becomes useless. Queries performed in batch processes have limitations to the amount of data that can be delivered in realtime. If the services require uninterrupted processing and supply of data at a high speed physical on prem data warehouses may not be the ideal solution anymore.
5. Infrastructure has become your sole focus
If coming up with new hacks to manage the infrastructure is the sole task with which a the DBA or DevOps team is tied with, life can become quite miserable. While maintenance and troubleshooting are a part of the DBA roles but it can’t be the only task. This is the time to look for an alternative data management solution that offers hassle free scaling and optimization.
Conclusion: Planning for your database migration
Database migration is a step-by-step process that requires meticulous planning and careful execution to not disrupt anything. It is always wise to approach a cloud consulting company if the internal team is not adequate enough to undertake such tasks. The general outline to planning a database migration can be as following:
Understanding the source database
Understanding the source database that will populate the migrated database is important even before starting to migrate. The following factors should be considered like the size of the source database, data types involved, presence of large tables.
Assessing the data
This stage involves profiling the source database and defining data quality rules to remove inconsistencies, duplicacies, and incorrectness. This stage helps to sort and migrate only quality data.
Converting database schema
Heterogenous database migration require conversion of schema while migrating from the source to destination database. This can be achieved manually but the better path is to use some data migration tools supporting database schema migration.
Testing the migration build
It is a good idea to iteratively migrate data. This involves creating a small subset of data and running them through all the stages of migration. Once the database schema has been converted with the help of some migration tool it is important to test it. Following a successful testing of the subset the data volume can be increased gradually to build a single workflow.
Executing the migration
Planning also involves when is the ideal window for downtime. Data synchronization and CDC tools can help to plan a interruption free migration.
A successful data migration plan also encompasses retirement policies for the old database. Checks should also be placed to validate the incoming data and monitor the quality of migrated data.
