Forum Discussion
Upload large data to storage in different zones
Hi all,
First of all please accept my apologise for my question. This is my first question in the tech form. I have a case where I need to upload large data to the cloud storage and I need a guide to solve this. So, we have a large single object with size around 800 GB, which I need upload this to the storage. My first question for the large single object like this, which storage (Blob or Data lake) you suggest? My second question is related also to this. I want to upload this file as multi part. Moreover, I need to use for this storages which are in the different zones (this is also project requirement) and measure the uploading time. I would like to know your suggestions for this purpose. Thank you for reading
1 Reply
Hi Agil Yolchuyev.
For your first question, it depends of what the purpose of this large object, after all, Data Lake is a Optimized storage for big data analytics workloads, if this object it's gonna be used to Big Data Analyses, i wold recommend use Data Lake, if not, use the Blob.
(Reference: https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage)
For your second question, since the object has a large size and i don't know your internet bandwith, i recommend look for Azure Import/Export service, where you send a external Disk to Azure and they upload your objects to Azure Storage.
(Reference: https://docs.microsoft.com/en-us/azure/storage/common/storage-import-export-service)
But if you have a good link, you can use AzCopy
(Reference: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-migrate-on-premises-data)Looking foward to hearing from you
All the best. JP
