MLens is Knowledge Lens’ one-stop solution for enterprises’ big data requirements. With Big Data Backup, Automated Disaster Recovery, Data Ingestion, Compression, Encryption, and Archival capabilities, our solution has proven to deliver absolute control over your enterprise data, its storage, analysis and recovery.

This seven-part blog series will take you through the different scenarios where MLens was deployed in our clients’ organizations, to solve their business challenges, and add value to all on board. In this post, we will look into our client’s requirement for Data Migration from a Live Cluster.

Business Challenge

During data migration, our client’s source cluster was a live cluster with continuous changes such as file addition, deletion, and renaming being made to it. For large scale data migration, the process could take hours, even days to run. Identifying all changes made to the cluster, was a complicated process.

Our client required a solution wherein multiple additions, deletions and modifications of files could be done while the migration was in process.

The Knowledge Lens Solution

Hadoop stores all operations that are performed on a cluster in its Edit Log files. The MLens team designed a highly efficient parser for the Edit Log files generated by Hadoop. Parsing the edits logs and identifying changes turned out to be simple and time-saving.

“Millions of Hadoop operations within a specific time interval were detected within a few minutes. The edit log parser was then integrated with MLens’ Data Migrator tool to perform incremental migration.”

Why Knowledge Lens?

With the unique combination of the edit log parser and the Data Migrator tool, we successfully  implemented full and incremental data migration of our client’s live Hadoop cluster, meeting their requirements.

Explore MLens

Request Free Demo

Looking for more resources? Read our previous posts in the series here:

How a Global Payments Company achieved Secure, Scalable Data Masking and Data Transfer

How our client achieved Secure Data Transfer across multiple Hadoop Clusters

Read our customer success stories here-

High Speed Backup and Efficient Data Recovery

Reshaping the Creation of Enterprise Data Lakes

At Knowledge Lens, we constantly work towards improving our Lenses, so your business can do more for you. Visit us here to learn how you can grow your business operations through data- driven decision making, starting today.

Contributors: Rupak Das, Technology Lead, Knowledge Lens


Sneha Mary Christall

Marketing and Brand Executive, Knowledge Lens.

Leave a Reply

Your email address will not be published. Required fields are marked *