MLens is Knowledge Lens’ one-stop solution for enterprises’ big data requirements. With Big Data Backup, Automated Disaster Recovery, Data Ingestion, Compression, Encryption, and Archival capabilities, our solution has proven to deliver absolute control over enterprise data, its storage, analysis and recovery.
This seven-part blog series will take you through the different scenarios where MLens was deployed in our clients’ organizations, to solve their business challenges, and add value to all on board. In this post, we will look into our client’s requirement for HDFS Snapshot Backup and Recovery.
Snapshots are read-only copy of HDFS data that are created to protect sensitive data from user modifications. Although snapshot data cannot be modified by any user, snapshots as a whole can be deleted. Also, the snapshot data lies in the same HDFS cluster where the actual data is present. So, in case of a disaster where the entire HDFS cluster fails, the entire snapshot data could be lost.
It is a common practice in industries to take snapshots of production data to prevent it from accidental modification by users or programs. This raises the need for a backup of snapshots that is taken in a separate storage, other than the cluster.
Backup of HDFS cluster data to an external storage like Amazon S3 can be done by Distcp, a tool provided by Apache Hadoop. But due to the way in which snapshots are stored, it cannot be read directly by the Distcp tool. Thus, the backup and restore of snapshots became a huge challenge and it was practically impossible to do, manually.
The Knowledge Lens Solution:
MLens provides a snapshot disaster recovery tool which can perform an automated backup and restore of snapshots, using Amazon S3 as a backup storage. It has an efficient snapshot listing algorithm that searches for all available snapshots within a given HDFS path and backs up all the data and metadata of the snapshots.
The most important feature of this tool is that it can also restore all the snapshot data and metadata, in the correct order in which the snapshots were created originally.
This tool uses MLens’ distributed framework to perform high speed distributed parallel data transfer, which enables a highly robust snapshot backup and restore process. The MLens edit log parser is updated to detect incremental changes in snapshot operations like Create, Delete, Rename, and is integrated with this tool.
Why Knowledge Lens?
“MLens provided the unique advantage of supporting the incremental backup of snapshot data. This eliminated the need for cluster downtime and the copying of the same huge data sets, over and over again.”
Thus, the MLens Snapshot Disaster Recovery tool provided a unique solution for high speed distributed backup and restore of HDFS snapshots.
Looking for more resources? Read our previous posts in the series here:
How a Global Payments Company achieved Secure, Scalable Data Masking and Data Transfer
How our client achieved Secure Data Transfer across multiple Hadoop Clusters
How our client achieved full and incremental Data Migration from a Live Cluster
Achieve Real-time Replication of Data across Multiple Live HBase Clusters
Read our customer success stories here-
High Speed Backup and Efficient Data Recovery
Reshaping the Creation of Enterprise Data Lakes
At Knowledge Lens, we constantly work towards improving our Lenses, so your business can do more for you. Visit us here to learn how you can grow your business operations through data- driven decision making, starting today.
Contributors: Rupak Das, Technology Lead, Knowledge Lens
http://canadianorderpharmacy.com/ · June 3, 2019 at 7:00 pm
We are a gaggle of volunteers and starting a brand new scheme in our community. Your website provided us with helpful info to work on. You have done a formidable job and our entire community can be thankful to you.
Sneha Mary Christall · June 13, 2019 at 6:23 pm
Hi, we are super pumped to hear that our website could be of use to you. Subscribe to our Blogs to be in the loop, or mail us at email@example.com if you’d like to know more from us.
Mini Excavation R. Beaudet · June 14, 2019 at 12:22 pm
I simply could not go away your site before suggesting that I actually loved the standard info a person provide for your visitors?
Is gonna be again incessantly to check out new posts|
I need to to thank you for this excellent read!! I certainly enjoyed
every little bit of it. I have you book-marked to check out new stuff you post? https://www.miniexcavationrbeaudet.com
Sneha Mary Christall · June 26, 2019 at 9:39 am
Thank you very much, keep visiting our blog for more updates!
Thaddeus · July 18, 2019 at 6:01 am
Thanks for sharing your thoughts on website. Regards
Sneha Mary Christall · July 18, 2019 at 11:08 am
We’re glad you enjoy our posts, keep coming back for more! You can also e-mail me at firstname.lastname@example.org, for all further queries.