Forum Discussion

Fjorgego's avatar
Fjorgego
Copper Contributor
Jul 17, 2023

Read large volume of files from Blob Storage


Hi!
 
I have a Blob Storage containing approximately 1 billion documents, split across different folders. I'm currently facing challenges in finding an ideal solution to retrieve the files, read their metadata, and update a tag to indicate whether the document has been processed or not. Could you provide any suggestions on how to address this problem? Currently, I am using the blobContainerClient.listBlobsByHierarchy(delimiter, options, null) method to retrieve the files. However, there are cases where I am unable to retrieve all the documents within a folder.
 
Full implementation (Java):
   ...
 
        List<Map<String, Map<String, String>>> listOfMetadata = new ArrayList<>();
        String delimiter = "/";
        log.info("Setting details");
        BlobListDetails blobListDetails = new BlobListDetails().setRetrieveMetadata(true).setRetrieveTags(true);
        log.info("Setting options");
        ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix).setDetails(blobListDetails).setMaxResultsPerPage((int) listLimit);
 
      
        blobContainerClient.listBlobsByHierarchy(delimiter, options, null).stream().parallel().limit(listLimit).forEach(blob -> {
            Map<String, String> mapOfTags = blob.getTags();
            if (mapOfTags.containsKey("wasRead") && mapOfTags.get("wasRead").equals("false")) {
                Map<String, Map<String, String>> mapOfBlobs = new HashMap<>();
                mapOfBlobs.put(blob.getName(), blob.getMetadata());
                listOfMetadata.add(mapOfBlobs);
            }
        });
 
        return listOfMetadata;
...
azure blob storage

2 Replies

Sort By
  • ProSolutions's avatar
    ProSolutions
    Iron Contributor
    Jul 23, 2023

    Fjorgego  Processing a large volume of files from Blob Storage can be a challenging task. To improve the efficiency and reliability of your implementation, you can consider the following suggestions:

     

    1. **Optimize ListBlobsOptions:**
    - Make sure to set the `setFlatListing(true)` option in `ListBlobsOptions` to retrieve all blobs, regardless of their hierarchy.
    - Increase the `setMaxResultsPerPage` value to fetch a larger batch of blobs per request. This reduces the number of API calls and improves performance.

     

    2. **Use Continuation Token:**
    - When working with a large number of blobs, you might encounter continuation tokens, which indicate that there are more blobs to retrieve. You should handle these tokens properly to paginate through the results.

     

    3. **Parallelization:**
    - It appears that you are already using parallel streams to process blobs concurrently. Make sure to adjust the level of parallelism based on your hardware capabilities and the rate limits of your Blob Storage to achieve optimal performance.

     

    4. **Batch Processing:**
    - Instead of processing blobs one by one, consider processing them in batches. Fetch a reasonable number of blobs at a time, process them, and then move on to the next batch. This can improve efficiency and reduce overhead.

     

    5. **Asynchronous Processing:**
    - For an even more efficient solution, you can use asynchronous programming to retrieve and process blobs concurrently.

     

    6. **Caching and Local Storage:**
    - Depending on your use case, you could consider caching metadata locally to reduce the number of requests to Blob Storage. However, be mindful of the storage space and data consistency when employing caching.

     

    7. **Optimize Metadata and Tags:**
    - Ensure that the metadata and tags associated with each blob are streamlined and minimized to reduce the amount of data that needs to be fetched.

     

    8. **Azure Functions or Durable Functions:**
    - If you have access to Azure Functions, you can leverage them to process blobs in a serverless and scalable manner. Durable Functions can help with managing state and handling retries in case of failures.

     

    9. **Partitioning and Sharding:**
    - Depending on your specific scenario, you could consider partitioning or sharding your data in Blob Storage to spread the processing workload and improve parallelism.

     

    Remember that large-scale data processing can be complex, and it's essential to consider the cost, scalability, and performance trade-offs based on your specific use case. Continuously monitor your implementation and make iterative improvements as needed.

     

    Lastly, consider whether you need to update the "wasRead" tag for each blob individually, or if you can batch update tags for multiple blobs at once to reduce API calls. This can significantly improve the efficiency of the tag updating process.

Resources