How to ingest historical data into ADX

Published Sep 29 2021 08:21 AM 1,450 Views

LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. To learn more about LightIngest, see Use LightIngest to ingest data into Azure Data Explorer.

LightIngest can be particularly useful to load historical data from an existing storage system to Azure Data Explorer. While you can build your own command using the list of Command-line arguments, this article shows you how to auto-generate this command through an ingestion wizard. In addition to creating the command, you can use this process to create a new table, and create schema mapping. This tool infers schema mapping from your data set.

This article shows you how to create a new table, create schema mapping, and generate a LightIngest command for one-time ingestion using the LightIngest tool.

 

 

How to Ingest more than 5000 files/blobs to Azure Data Explorer 

 

  1. Open Web Explorer 
  2. Select ingest new data or Ingest new data from container at Data Page 
  3. Select Cluster, database and table (new or existing) 
  4. Source tab - for more than 5,000 blobs select Historical data 

source-tab-lightingest.png

  1. Select a source or enter SAS URL to source 

 

source-tab-container-from-subscription.png

  1. Select "Advanced parameters" to support: 
    1. Creation Time 
    2. blob name pattern 
    3. Tag
    4. limited amount of files 
    5. Don't wait for the ingestion to be completed (recommended for big amount of data )
    6. 1.PNG3.PNG
  2. Review and adjust the auto-suggested schema structure 
  3. In the Data ingestion completed window, all three steps will be marked with green check marks.

    Copy the generated LightIngest command by clicking on the copy icon to the top right of the command box.

    summary-tab-copy-command.png
  4. To complete the ingestion process, you must run LightIngest using this copied command.

read more on generate lightingest command 

Recommended reading 

  1. "How to Ingest Container with less than 5000"  
  2. Azure Data Explorer Ingestion Overview 
  3. Azure Data Explorer Ingestion 
Version history
Last update:
‎Nov 03 2021 04:05 AM
Updated by: