This article defines data migration as the movement or copying of data from one system or database to another. Triggers for data migration projects range from retirement of legacy systems to enterprise resource planning (ERP) upgrades. Business scenarios that create these and other triggers include:
Merger or acquisitions: multiple legacy information technology (IT) systems and datasets left in the wake of mergers or acquisitions, requiring consolidation for operational efficiencies and cost reductions.
Application standardization: mergers, acquisitions, or ungoverned purchases created multiple software platforms serving identical purposes; reduction and standardization lower licensing costs, elevates support efficiencies and reduces employee training requirements.
Modernization: a strategic decision to sunset existing systems in order to increase business competitiveness, achieve higher operating efficiencies, and/or optimize customer experience.
Regardless of what triggers data migration, a successful implementation requires precision in planning, implementation, and transition. Significant risk is always present because once migration is done the source system or database goes dark—there’s no going back or second chances.
RPA and data migration
Robotic Process Automation (RPA) capabilities position this technology for consideration as a data migration tool because highly structured, rules-based migration activities fit the profile of what robots do best. That profile is apparent in the fundamental extract, transform, load (ETL) data migration methodology.
Extract Design: well-defined requirements for the manner in which data will be extracted, held, and verified.
Transform: solution design rules guide data transformation for the targeted to-be data structure.
Load: clearly defined steps dictate how extracted and transformed data is mapped into the target structure.
Test and Recovery: specific unit and integration test plans, along with exit criteria, reporting, roll-back, and recovery procedures for every migration stage.
By spelling out rules, sequential activities, and required outcomes for every data migration step, this ETL methodology makes a clear case for using RPA. Further, since data migration often involves legacy systems lacking API access, RPA’s unique user interface (UI)-level integration, which avoids impacting underlying systems and databases, makes it a low-risk choice.
Use case example
This insurance industry use case example illustrates the value of RPA in relatively modest migration projects.
The need to migrate data in the insurance industry is commonly tied to the acquisition of a “book of business”—or block of policies—by one company from another. It’s not unusual for these books to have been written years ago and administered on old legacy systems supplemented by spreadsheets. Acquired books of business are typically at least several hundred thousand policies and the practical difficulties of extracting and cleansing this volume of data from legacy systems and spreadsheets is not hard to imagine.
Acquired policies are “live” and must be administered, so migration time is of the essence. While that sounds like an obvious role for automation, this scale of a legacy migration rarely justifies the time and expense of a systems integration project.
Before RPA emerged as a rapid, low-cost automation option, acquiring companies had few options beyond deploying a migration team of several small groups; typically leads and analysts from the business unit tasked with servicing the acquired policies. These groups worked through extraction, quality review, and cleansing. Then, the IT department would take on transforming, loading, and testing activities.
This labor-intensive approach to data migration meant a book of several hundred thousand policies could take up to 12 weeks and cost in the neighborhood of $350,000—but a bargain in time and money compared to a systems integration project.
Now, with proper planning and execution, RPA can change both timelines and costs.
First step: designing and automating a rules-driven migration process for all three steps: extracting, transforming, and loading. To be effective, this workflow must clearly define rule-driven steps appropriate for automation and logical exception points where human intervention is necessary.
Second step: a reassignment of roles and responsibilities which leverages the different strengths of robotic automation and the business analysts.
Third step: integrating the automated processes with unit and integration test scripts.
Unattended robots do their work invisibly in the background, and analysts often remain involved—but in a different role. With error-free automation taking over the role of manual data movement, analysts can focus on documenting, archiving, and resolving errors in data properties and activity outcomes. If the migration is large enough to justify it, automating many of those activities will be done as well.
With effective process modeling, automation design, and training, RPA can cut the migration time and costs by 50% and 40% respectively. Additional benefits include complete extraction accuracy and detailed, archived log files on all transactions for operating and compliance purposes.
UiPath Enterprise RPA Platform and data migration
While the previous use case makes a compelling argument for RPA, it does so for a modest data migration scope. To be effective beyond that scope, the RPA solution must scale to hundreds of robots leveraging complex work queues in parallel processing execution. While UiPath is one of the few platforms capable of scaling to that level, the reality is large scale data migration requires enterprise-grade software tools specifically designed for the job.
The value UiPath brings to these large data migration projects is tied to our unique drag-and-drop REST API & webhooks—features that create unmatched, dynamic integration with other automation solutions: for example, ERP, business process management (BPM) and data migration tools such as our partner SiriusIQ.
By dynamically integrating with SiriusIQ at the UiPath Orchestrator and Robot level, UiPath and the SiriusIQ data migration tool become a seamless, powerful solution for customers facing large scale, one-time data migration and system configuration projects involving massive amounts of data: typically, hundreds of terabytes.
Within this solution, Orchestrator moves the vast majority of activities left for humans by SiriusIQ to unattended Robots. These tasks go beyond monitoring and error-handling to include the validation of migrated transactions. You can see this powerful UiPath/SiriusIQ large scale data migration solution in action:
This approach has been used so successfully in large-scale SAP migrations that EY is working with UiPath to roll out a S/4HANA upgrade migration offering. You can see more information on how UiPath is transforming large scale data migration, particularly for the SAP S/4HANA migration, below:
Automated data conversion, testing, and custom code remediation are examples of where RPA, and aligned technologies such as machine learning, will be combined in this S/4HANA upgrade approach. Bill Hale, digital automation leader at EY, projects this methodology will cut upgrade cost and time in half.
As both use cases—one with modest migration scope and the other with hundreds of terabytes—make clear, the use of RPA in the migration of data delivers compelling cost and performance benefits.
It’s a testimony to the power and flexibility of the UiPath Enterprise RPA Platform it brings the same benefits of rapid deployment, return on investment (ROI), and performance scaling to such widely divergent scenarios, effectively bringing automation to activities that otherwise would be relegated to employees.
David Eddy is the Director of Strategic Marketing at UiPath.
Note: This post is an updated version of my 2015 post How Robotic Process Automation Fits Data Migration
