Forum Discussion

Wavel's avatar
Wavel
Copper Contributor
Mar 10, 2021

Word search through thousands of pdf's?

Is this the appropriate product to use if I want to create a word search index for thousands of pdf files, and then query it from my asp.net application?  
Luis Cabrera-Cordon's avatar
Luis Cabrera-Cordon
Former Employee
Mar 10, 2021
Yes, absolutely. You could put your thousands of pdf files in a repository like blob storage, then index it and then query for the information in those PDFs. Note that there are many different types of PDFs. If you have scanned PDFs, you may want to add a skillset to your indexer that extracts text from the images embedded in the PDFs. The easiest way to do all of this (in just a couple of minutes) is to follow this tutorial : https://docs.microsoft.com/en-us/azure/search/cognitive-search-quickstart-blob By the end of that quickstart, you will have an index that you can query so you can find any PDFs that meet the query requirements. Thanks, Luis Cabrera, Azure Cognitive Search Team
  • Wavel's avatar
    Wavel
    Copper Contributor
    Mar 10, 2021
    Last time I checked though, the system only indexed the first 30k or so of each document. I need the entire pdf (10k-2MB) searchable. These are all text pdf's by the way. No images.
    • Luis Cabrera-Cordon's avatar
      Luis Cabrera-Cordon
      Former Employee
      Mar 10, 2021

      Wavel You should be able to index all of the content of the document.

       

      Note that different tiers have different limits in terms of the number of characters pushed into the index. See Service limits for tiers and skus - Azure Cognitive Search | Microsoft Docs.  Note that that is not the limit for the size of the actual PDF.

       

      If you have an S2 tier for instance,8MB of characters get indexed, but it would take a pretty big PDF to generate 8MB of characters (about 2700 pages full of characters).

      • Wavel's avatar
        Wavel
        Copper Contributor
        Mar 10, 2021
        So the issue becomes pricing. S2 is almost a $1000/mo and in my case, we're talking about 20k documents. I think the pricing doesn't work for a small number of documents that might be larger than 30k or so.
        My pdf total size is under the 2 Gig limit of Basic but because some are large, I'm forced into S1 or even S2. Not affordable in this case.
    • David_Bluebox's avatar
      David_Bluebox
      Copper Contributor
      Mar 10, 2021
      I have similar requirements where a client subscribes to a number of Market Research (in education) sources that provide regular PDF's containing text and images. My client wishes to perform a search across those documents e.g. all research documents related to students coming from China. The documentation I have read seems to also refer to the text limit as mentioned by Wavel. How can we execute such an indexing operation? All of the files currently reside in SharePoint.

Resources