This is a notification to let you know that we’re moving forward with these changes that were published 4 weeks ago.

Please read below to see which steps were completed and what are the next steps.

It also includes a new change in Microsoft.Azure.Kusto.Ingest (change #4)

Changes #3 and #4 will be released together as part of the same release of the client library update.

 

This notification includes information about 4 upcoming changes in Azure Data Explorer service. The changes are in the:

1. Serialization of dynamic types in Kusto REST API.
2. Retention policy.
3. .NET version support
4. Microsoft.Azure.Kusto.Ingest

 

-------------------------------------------------------------------------------------------------

Planned change in the serialization of dynamic types in Azure Data Explorer REST API

The change is backwards-compatible for customers who use Azure Data Explorer UX tools (Kusto.Explorer and Kusto.WebExplorer), but it does have an impact on customers using the client libraries or consume the REST API directly. Such customers are advised to read the following carefully and to assess the potential impact on their workload.

Change Details

The change includes the following modifications in the way values of type dynamic are serialized in Azure Data Explorer REST API response.

1.  Short: Dynamic types will be serialized as JSON values (formerly they were serialized as strings).
2. Long: Azure Data Explorer REST API encodes the results of a query or control command as a JSON document. Due to historical reasons, there are two ways in which a scalar value of Azure Data Explorer type dynamic may be encoded in the JSON document – as a string which is the serialization of the value, or as the value itself. It is possible to distinguish between them by examining the schema sub-object that precedes the schema (so readers do not need to know in advance which encoding will be used). This change sets the default serialization behaviour of dynamic types to the latter encoding. This change affects the /v2 REST API only and is aligned with how the /v1 REST API works today.
3. Null dynamic values (that is, values ‘v’ of type dynamic for which isnull(v) is true) will be serialized as JSON null values (formerly they were serialized as empty strings). This change affects both the /v1 and /v2 REST APIs.
4. String dynamic values (that is, value ‘v’ of type dynamic for which gettype(v) is string) will be serialized as strings. Formerly such were casted to their inferred type and serialized accordingly (e.g., if the value was a string that could be parsed as a number, it was serialized as a number, not a string). This change affects both the /v1 and /v2 REST APIs.

 

The following table demonstrates the differences in raw response between current behaviour and future behaviour:

Change	Scope of impact	Query	Former representation	New representation
1	/v2	print dynamic({"a" : 123})	"{\"a\":123}"	{"a":123}
2	/v1, /v2	print dynamic(null)	""	null
3	/v1, /v2	print dynamic("123")	123	"123"
3	/v1, /v2	print dynamic("{}")	{}	"{}"

 

How can I tell if this change affects me? How can I test it before it hits my cluster?

The change will be rolled out gradually, according to the rollout plan outlined below. To allow customers to test the new behaviour before it is enabled on the service side in production, one can enable this new behaviour per-request on the client side by using the following client request properties. Kindly note that these are provided for a limited period, until customers modify their code to work well with all changes enabled.

 

1. response_dynamic_serialization – controls whether dynamic types are serialized as strings or objects – change #1 described above. Valid options are “string” (old behaviour) or “json” (new).

 

2. response_dynamic_serialization_2 – controls the 2 additional changes explained above. Valid options are “legacy” (null values are serialized as empty strings; dynamic strings are serialized according to their inferred types), or “current” (new).

 

We strongly advise you to test and modify your code now to handle the new behaviours using these client request properties, as they will eventually become the default behaviour (json; current).

 

Client Request Property Name	Values	Current Default V1	Current Default V2
response_dynamic_serialization	[string, json]	json	string
response_dynamic_serialization_2	[legacy, current]	Legacy	legacy

 

Schedule & plan 

Please carefully review the rollout plan and ETAs to make sure your code does not break.

 

We are about to move to phase #1 of the dynamic types serialization change. Please make sure you have verified your code with the new behaviour, as explained below. If you are using client request properties or a feature flag to set the behaviour to the legacy one, we strongly recommend that you remove these properties/feature flag and transition to the new behaviour ASAP. The legacy behaviour will be deprecated at the end of the transition, according to the phases outlined below.

Phase #1: The service default behaviour is set to new (json, current) (ETA: week of January 24, 2019)

• We will send another notification prior to making this change.
• Client that will override the default settings using the relevant client request properties (string/legacy) will still be able to use the legacy behaviour.
• Recommended actions: if applicable, remove the client request properties added during testing phase, as these are now the new defaults.  

 

Phase #2: Microsoft.Azure.Kusto.Data major version upgrade (ETA: week of February 04, 2019)

• After completing the rollout of the server-side changes, we will update our .NET client libraries accordingly. This will include type changes for dynamic types returned by the ExecuteQuery APIs. All expected changes will be described in the version changelog.

Phase #3: Remove support for legacy behaviour (ETA: week of March 11, 2019)

• We will send another notification prior to making this change.
• We expect all clients to modify their code so that it handles the new behaviour correctly by this time.
• Clients requesting the legacy behaviour using client request properties (string/legacy) will not be supported and will receive the new behaviour.

 

.NET Client Libraries (Microsoft.Azure.Kusto.Data, Microsoft.Azure.Kusto.Ingest)

The change is fully backward compatible for customers using Azure Data Explorer .NET client libraries, with the following exception:

• A dynamic null value will be returned as a DBNull by the IDataReader. This is aligned with how other null types are represented in the IDataReader. Please make sure your code handles this correctly (formerly, null dynamic values were returned as empty strings).

 

You can test your code’s compatibility with the new changes, using the client request properties, as in the sample below:

 

 

The next major version upgrade of Azure Data Explorer .NET client libraries will include additional breaking changes involving dynamic types. All changes will be described in the changelog of the new version.

REST API Clients

Customers using the REST API directly (either v1 or v2) are potentially impacted by all these changes. The snippet below shows how you can test your code with the new behaviour, using the client request properties:

 

Python / NodeJS / Java Client Libraries

Customers of these client libraries are impacted by the changes to the REST API raw response, as the libraries return the REST API response as-is, and the caller is responsible for parsing it. Note that Java client library uses /v1 protocol (therefore, is not impacted by change #1), while Python and NodeJS use the /v2 protocol. Please search the client library docs on how to set client request properties when sending a request.

 

If you are using Azure Data Explorer python client library with data frames(dataframe_from_result_table), make sure to upgrade the client library to version 0.0.17 or later before rollout of phase #1.

 

 

-------------------------------------------------------------------------------------------------

Planned change in Azure Data Explorer Retention Policy

 

This message includes information about an upcoming change in the Retention policy hard delete setting.

 

Change overview: We started an effort to reduce the storage cost of Azure Data Explorer clusters and we expect that it will result in reduction of 30% of the cost. This effort requires changes to the retention policy implementation, as outlined below.

Customers who are using the Retention policy HardDeletePeriod attribute, should read this message carefully to avoid compatibility issues once the change is deployed.

Change Details

If you’re managing or monitoring the Retention policy, we strongly recommend changing it to comply with the changes detailed below.

Note that the syntax of setting the retention policy will be kept backward compatible, yet the values and the settings returned by the commands will change during the following phases.

The new setting is called Recoverability. Possible values are “Enabled”, “Disabled”

You can start using the new syntax for Retention policy management (.alter and .show commands) now.

Extended syntax to alter Retention Policy:

.alter-merge <entity_type> <database_or_table_name> policy retention

softdelete = <timespan>

recoverability = <enabled|disabled> // new setting

harddelete= <softdelete + 1d | softdelete + 14d>  // kept for backward compatibility

Recommended actions:

• Switch to setting and reading Recoverability setting, note that you must set the HardDeletePeriod to either SoftDeletePeriod + 1d or SoftDeletePeriod + 14d (this is because in phase 1 we will not enforce the new Recoverability setting).
• Remove any read-dependency on Retention policy settings that are obsolete: HardDeletePeriod, ContainerRecyclePeriod, ExtentsDataSizeLimitInBytes, OriginalDataSizeLimitInBytes.

These actions will ensure forward and backward compatibility of your code with upcoming changes.

 

Schedule & plan 

Phase #1: Extension of the Retention Policy object (ETA: Completed)

• Auto-migration: Azure Data Explorer will auto-migrate tables/databases that were not explicitly migrated during phase #1.

The value of the recoverability setting will be derived from HardDeletePeriod setting:

If HardDeletePeriod - SoftDeletePeriod > 3d, Recoverability will be set to ‘Enabled’.

If HardDeletePeriod - SoftDeletePeriod <= 3d, Recoverability will be set to ‘Disabled’. 

• The effective HardDeletePeriod is derived from Recoverability setting:

If Recoverability is ‘Enabled’, HardDeletePeriod=SoftDeletePeriod+14d

if Recoverability is ‘Disabled’, HardDeletePeriod=SoftDeletePeriod+1d 

Recommended action:

Remove code that sets the HardDeletePeriod setting. Continue using only the new Recoverability setting.

 

Phase #2: Removal of obsolete settings (ETA: week of January 21, 2019)

 

• Removal of the following settings of the Retention policy from the results of the “.show/.alter table policy retention” commands

HardDeletePeriod

ContainerRecyclingPeriod

ExtentsDataSizeLimitInBytes

OriginalDataSizeLimitInBytes

 

-------------------------------------------------------------------------------------------------

Planned change in Azure Data Explorer .NET support

This message includes information about an upcoming change in the .NET supported versions.

 

Change overview: Azure Data Explorer SDK will require .NET 4.6.2 as the minimal runtime version.

Earlier versions of .NET are deprecated for over a year and we will also stop supporting them.

Customers who are using Azure Data Explorer .NET SDK with .NET runtime lower than 4.6.2 will not be able to get the new versions of the SDK.

Change Details

Azure Data Explorer SDK will require .NET 4.6.2 as the minimal runtime version.

 

Schedule & plan 

Phase #1: Azure Data Explorer .NET SDK will publish a new preview version requiring .NET runtime version 4.6.2 or higher (ETA: week of January 21, 2019)

Phase #2: Azure Data Explorer .NET SDK will publish a new version requiring .NET runtime version 4.6.2 or higher (ETA: week of February 04, 2019)

Recommended action:

Upgrade your .NET runtime to 4.6.2 or higher

 

-------------------------------------------------------------------------------------------------

Planned change in Microsoft.Azure.Kusto.Ingest

Change details

The interface IKustoIngestClient was remodeled and reduced.

Highlights of the change:

• There are only asynchronous flavors of ingest methods
• All methods now accept a single data artifact (storage object, stream, or DataReader)
• File and blob ingestion methods are unified in a single method IngestFromStorageAsync
• Various data source handling options are unified under a single optional structured parameter ‘sourceOptions’

For the sake of backwards compatibility, all the legacy methods are still available as extension methods on the IKustoIngestion interface.

 

The new IKustoIngestion interface includes the following methods:

 

/// <summary>

/// Ingests data from <see cref="IDataReader"/>. <paramref name="dataReader"/> will be closed when the call completes.

/// </summary>

/// <param name="dataReader">The <see cref="IDataReader"/> data source to ingest. Only the first record set will be used</param>

/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>

/// <param name="sourceOptions">Options for the <see cref="IDataReader"/> ingestion source. This is an optional parameter</param>

/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>

Task<IKustoIngestionResult> IngestFromDataReaderAsync(IDataReader dataReader, KustoIngestionProperties ingestionProperties, DataReaderSourceOptions sourceOptions = null);

 

/// <summary>

/// Ingest data from one of the supported storage providers. Currently the supported providers are: File System, Azure Blob Storage.

/// </summary>

/// <param name="uri">The URI of the storage resource to be ingested. Note: This URI may include a storage account key or shared access signature (SAS).

///  See <see href="https://docs.microsoft.com/en-us/azure/kusto/api/connection-strings/storage"/> for the URI format options.</param>

/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>

/// <param name="sourceOptions">Options for the storage ingestion source. This is an optional parameter</param>

/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>

Task<IKustoIngestionResult> IngestFromStorageAsync(string uri, KustoIngestionProperties ingestionProperties, StorageSourceOptions sourceOptions = null);

 

/// <summary>

/// Ingest data from <see cref="Stream"/>.

/// </summary>

/// <param name="stream">The <see cref="Stream"/> data source to ingest</param>

/// <param name="ingestionProperties">Additional properties to be used during the ingestion process</param>

/// <param name="sourceOptions">Options for the <see cref="Stream"/> ingestion source. This is an optional parameter</param>

/// <returns>An <see cref="IKustoIngestionResult"/> task</returns>

Task<IKustoIngestionResult> IngestFromStreamAsync(Stream stream, KustoIngestionProperties ingestionProperties, StreamSourceOptions sourceOptions = null);

 

Impacted scenarios

Impact is limited to data ingestion scenarios using IKustoIngestClient interface.
 

Required change 

The recommendation when upgrading to the latest version of Microsoft.Azure.Kusto.Ingest is to review usage of IKustoIngestClient interface and adjust to the
new interface.

 

Schedule & plan 

Phase #1: Kusto ingest SDK API changes preview release (ETA: week of January 21, 2019)

Phase #2: Kusto ingest SDK API changes release (ETA: week of February 04, 2019)

 

Please add your questions / comments about these changes as replies to this post.