Forum Discussion

Search720's avatar
Search720
Copper Contributor
Mar 10, 2021

Ingest/Index static and dynamic web pages

What would the recommended method be to index/ingest standard classic HTML and client-side Javascript rendered web page content? Is there a native web crawler/indexer for "dynamic" web page content? ...
DerekLegenzoff's avatar
DerekLegenzoff
Former Employee
Mar 11, 2021

Search720 there's no built-in indexer for crawling web pages so customers often leverage an open-source crawler such as Apache Nutch to extract content from web pages. From there, you can land the content in a supported data source such as Blob storage/Cosmos DB/ADLS Gen2 and index it. You can also push the data directly to the index via the Push API as described here.

 
  • stejacob's avatar
    stejacob
    Copper Contributor
    Apr 20, 2021

    Search720 You can use the Norconex HTTP connector for dynamic webpages.

     

    https://opensource.norconex.com/collectors/http/

     

    Cheers.

    • Search720's avatar
      Search720
      Copper Contributor
      Apr 20, 2021
      Thanks!

Resources