So you’ve got a shiny new Azure Data Explorer (Kusto) cluster and you want to put some data into it. The easiest way is to import a CSV with Kusto Explorer, but if you want to do it programmatically, there are some nice APIs to get you started. Think of this post a crib notes for the real documentation which covers all of this in detail.
This post will walk through the absolute minimum requirement to ingest data with C#. These steps assume that you have already created a cluster and a database.
- In the Azure portal, go to the query editor for your database and execute: .create table ingestionTest(a:int, b:int, c:int)
- Create a new Console app and reference the Microsoft.Azure.Kusto.Ingest NuGet package. See the docs for more help with the package.
- Go to the C# API samples and copy the code from Ingest From Local File(s) using KustoDirectIngestClient into your new console project.
- Go the the Overview blade for your cluster on the portal and copy the URI value. Pass that value into the KustoConnectionStringBuilder constructor. (Note, normally you would not use direct ingestion. It can be hard on your cluster and requires you to manage a lot of retry logic, etc on your own. Queued ingestion is preferred. In those cases, your connection string would be the "Data Ingestion URI" value.)
- Update the Initial Catalog to the name of your database and also set that as the value for the kustoDatabase variable lower in the code.
- The code sample is configured to use an AAD app to authenticate, but for this example, let's just keep it simple and run with your own AAD account. To do this, simply remove the ApplicationKey and ApplicationClientId fields from the initialize block. All you need to set are the FederatedSecurity and InitialCatalog fields.
- Update the kustoTable variable to be the name of the table that you created in step 1.
- Create a sample CSV file that has three integers on each row. Put that file name into the "files" list in the code.
- Run!
Because this is using direct ingestion, you should immediately be able to go the Query blade in the portal and query your table. So if you called your table "ingestionTest", execute a query that is just "ingestionTest". You should see the contents of your data there.
In a real production scenario, you would be using queued ingestion and you'd probably want to ingest from a blob or a stream. For further reading, checkout a bigger walkthrough of a scenario like I described above and also best practices for ingestion.
Congrats! You have now ingested some data into Azure Data Explorer. Expand that out to a millions rows and witness the power of the Kusto query language!
