MLens is Knowledge Lens’ one-stop solution for enterprises’ big data requirements. With Big Data Backup, Automated Disaster Recovery, Data Ingestion, Compression, Encryption, and Archival capabilities, our solution has proven to deliver absolute control over your enterprise data, its storage, analysis and recovery.
This seven-part blog series will take you through the different scenarios where MLens was deployed in our clients’ organizations, to solve their business challenges, and add value to all on board. In this post, we will look into our client’s requirement for Data Migration from a Live Cluster.
Business Challenge
During data migration, our client’s source cluster was a live cluster with continuous changes such as file addition, deletion, and renaming being made to it. For large scale data migration, the process could take hours, even days to run. Identifying all changes made to the cluster, was a complicated process.
Our client required a solution wherein multiple additions, deletions and modifications of files could be done while the migration was in process.
The Knowledge Lens Solution
Hadoop stores all operations that are performed on a cluster in its Edit Log files. The MLens team designed a highly efficient parser for the Edit Log files generated by Hadoop. Parsing the edits logs and identifying changes turned out to be simple and time-saving.
“Millions of Hadoop operations within a specific time interval were detected within a few minutes. The edit log parser was then integrated with MLens’ Data Migrator tool to perform incremental migration.”
Why Knowledge Lens?
With the unique combination of the edit log parser and the Data Migrator tool, we successfully implemented full and incremental data migration of our client’s live Hadoop cluster, meeting their requirements.
Looking for more resources? Read our previous posts in the series here:
How a Global Payments Company achieved Secure, Scalable Data Masking and Data Transfer
How our client achieved Secure Data Transfer across multiple Hadoop Clusters
Read our customer success stories here-
High Speed Backup and Efficient Data Recovery
Reshaping the Creation of Enterprise Data Lakes
At Knowledge Lens, we constantly work towards improving our Lenses, so your business can do more for you. Visit us here to learn how you can grow your business operations through data- driven decision making, starting today.
Contributors: Rupak Das, Technology Lead, Knowledge Lens
