User Profile
Fjorgego
Copper Contributor
Joined 2 years ago
User Widgets
Recent Discussions
Read large volume of data from Storage
Hi folks! I have 1 billion of blob files in an azure storage that I need to read and update a tag. What solution do you guys recommend to use? Azure Databicks, Azure Data Factory or using the Azure Storage SDKs and do a custom app? Thanks, Best Regards,401Views0likes0CommentsRead large volume of files from Blob Storage
Hi! I have a Blob Storage containing approximately 1 billion documents, split across different folders. I'm currently facing challenges in finding an ideal solution to retrieve the files, read their metadata, and update a tag to indicate whether the document has been processed or not. Could you provide any suggestions on how to address this problem? Currently, I am using the blobContainerClient.listBlobsByHierarchy(delimiter, options, null) method to retrieve the files. However, there are cases where I am unable to retrieve all the documents within a folder. Full implementation (Java): ... List<Map<String, Map<String, String>>> listOfMetadata = new ArrayList<>(); String delimiter = "/"; log.info("Setting details"); BlobListDetails blobListDetails = new BlobListDetails().setRetrieveMetadata(true).setRetrieveTags(true); log.info("Setting options"); ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix).setDetails(blobListDetails).setMaxResultsPerPage((int) listLimit); blobContainerClient.listBlobsByHierarchy(delimiter, options, null).stream().parallel().limit(listLimit).forEach(blob -> { Map<String, String> mapOfTags = blob.getTags(); if (mapOfTags.containsKey("wasRead") && mapOfTags.get("wasRead").equals("false")) { Map<String, Map<String, String>> mapOfBlobs = new HashMap<>(); mapOfBlobs.put(blob.getName(), blob.getMetadata()); listOfMetadata.add(mapOfBlobs); } }); return listOfMetadata; ...3KViews0likes2CommentsAzure Integration - Knowledge Cluster
Today, I come to you with an exhilarating initiative brewing at my company, and I'm eager to hear your invaluable opinions and insights. We are gearing up to create an Azure Integration Knowledge Cluster that will revolutionize the way we navigate the Azure integration landscape. And guess what? Your input is always welcome 🙂 Our foundation for this knowledge cluster rests on five essential pillars, each representing a critical facet of Azure Integration: Azure Architecture: Embark on a journey to unravel the principles and best practices that underpin solid Azure architectures. Discover how to design scalable, resilient, and highly available integration solutions. Azure Services: Explore the vast array of Azure services at our disposal for seamless integration scenarios. From Logic Apps to Service Bus, Event Grid to API Management, we'll unravel their intricacies and unleash their full potential. Integration Platforms: Delve into the world of integration platforms, both cloud-based and on-premises. Uncover how they facilitate efficient data flow and provide a unified environment for managing integrations. Integration Patterns: Immerse yourself in the art of integration patterns, gaining insights into battle-tested approaches for connecting systems and orchestrating workflows. From request-reply to publish-subscribe, we'll delve into a variety of patterns to suit diverse integration needs. API Design, Messaging, and Event-Driven: Master the art of designing well-structured APIs that enable smooth communication between applications. Dive into messaging and event-driven architectures, leveraging asynchronous integration to build highly responsive systems. But wait, there's more! We'll also explore Hybrid Integrations, where on-premises systems seamlessly interact with the cloud, ensuring a harmonious coexistence between legacy infrastructure and Azure's cutting-edge capabilities. And, of course, we won't forget the crucial aspects of Security & Performance, safeguarding our integrations and optimizing their efficiency. Now, here's where your expertise comes in! I kindly invite you to share your thoughts on this knowledge cluster endeavor: Which of these pillars excites you the most? Have you encountered any integration challenges or roadblocks that you believe could be addressed through this knowledge cluster? Please share your ideas, suggestions, and opinions in the comments below 🙂 With sincere gratitude, Jorge467Views0likes0Comments
Groups
Azure Tech Bites
Azure Tech Bites event series help inspiring digital transformation and cloud adoption by empowering technical communities to achieve more by building an echo system of technical community (customers, partners and experts) focused on building customer trust to drive tech intensity in solving business problem by effectively using Azure products and services and accelerate cloud adoption for a better customer-connected experience
Latest Activity: Jul 01, 2024Recent Blog Articles
No content to show