Azure Data Explorer service notification – breaking changes

[Gabi Lehner]

This notification includes information about 3 upcoming changes in Azure Data Explorer service. The changes are in the:

1. Serialization of dynamic types in Kusto REST API.
2. Retention policy.
3. .NET version support

-------------------------------------------------------------------------------------------------

Planned change in the serialization of dynamic types in Azure Data Explorer REST API

The change is backwards-compatible for customers who use Azure Data Explorer UX tools (Kusto.Explorer and Kusto.WebExplorer), but it does have an impact on customers using the client libraries or consume the REST API directly. Such customers are advised to read the following carefully and to assess the potential impact on their workload.

Change Details

The change includes the following modifications in the way values of type dynamic are serialized in Azure Data Explorer REST API response.

1.  Short: Dynamic types will be serialized as JSON values (formerly they were serialized as strings).
2. Long: Azure Data Explorer REST API encodes the results of a query or control command as a JSON document. Due to historical reasons, there are two ways in which a scalar value of Azure Data Explorer type dynamic may be encoded in the JSON document – as a string which is the serialization of the value, or as the value itself. It is possible to distinguish between them by examining the schema sub-object that precedes the schema (so readers do not need to know in advance which encoding will be used). This change sets the default serialization behaviour of dynamic types to the latter encoding. This change affects the /v2 REST API only and is aligned with how the /v1 REST API works today.
3. Null dynamic values (that is, values ‘v’ of type dynamic for which isnull(v) is true) will be serialized as JSON null values (formerly they were serialized as empty strings). This change affects both the /v1 and /v2 REST APIs.
4. String dynamic values (that is, value ‘v’ of type dynamic for which gettype(v) is string) will be serialized as strings. Formerly such were casted to their inferred type and serialized accordingly (e.g., if the value was a string that could be parsed as a number, it was serialized as a number, not a string). This change affects both the /v1 and /v2 REST APIs.

The following table demonstrates the differences in raw response between current behaviour and future behaviour:

Change	Scope of impact	Query	Former representation	New representation
1	/v2	print dynamic({"a" : 123})	"{\"a\":123}"	{"a":123}
2	/v1, /v2	print dynamic(null)	""	null
3	/v1, /v2	print dynamic("123")	123	"123"
3	/v1, /v2	print dynamic("{}")	{}	"{}"

 

How can I tell if this change affects me? How can I test it before it hits my cluster?

The change will be rolled out gradually, according to the rollout plan outlined below. To allow customers to test the new behaviour before it is enabled on the service side in production, one can enable this new behaviour per-request on the client side by using the following client request properties. Kindly note that these are provided for a limited period, until customers modify their code to work well with all changes enabled.

1. response_dynamic_serialization – controls whether dynamic types are serialized as strings or objects – change #1 described above. Valid options are “string” (old behaviour) or “json” (new).

2. response_dynamic_serialization_2 – controls the 2 additional changes explained above. Valid options are “legacy” (null values are serialized as empty strings; dynamic strings are serialized according to their inferred types), or “current” (new).

We strongly advise you to test and modify your code now to handle the new behaviours using these client request properties, as they will eventually become the default behaviour (json; current).

Client Request Property Name	Values	Current Default V1	Current Default V2
response_dynamic_serialization	[string, json]	json	string
response_dynamic_serialization_2	[legacy, current]	Legacy	legacy

 

Rollout plan

Please carefully review the rollout plan and ETAs to make sure your code does not break.

Phase #1: The service default behaviour is set to new (json, current) (ETA: week of January 14, 2019)

• We will send another notification prior to making this change.
• Client that will override the default settings using the relevant client request properties (string/legacy) will still be able to use the legacy behaviour.
• Recommended actions: if applicable, remove the client request properties added during testing phase, as these are now the new defaults.  

Phase #2: Microsoft.Azure.Kusto.Data major version upgrade (ETA: week of February 04, 2019)

• After completing the rollout of the server-side changes, we will update our .NET client libraries accordingly. This will include type changes for dynamic types returned by the ExecuteQuery APIs. All expected changes will be described in the version changelog.

Phase #3: Remove support for legacy behaviour (ETA: week of March 11, 2019)

• We will send another notification prior to making this change.
• We expect all clients to modify their code so that it handles the new behaviour correctly by this time.
• Clients requesting the legacy behaviour using client request properties (string/legacy) will not be supported and will receive the new behaviour.

.NET Client Libraries (Microsoft.Azure.Kusto.Data, Microsoft.Azure.Kusto.Ingest)

The change is fully backward compatible for customers using Azure Data Explorer .NET client libraries, with the following exception:

• A dynamic null value will be returned as a DBNull by the IDataReader. This is aligned with how other null types are represented in the IDataReader. Please make sure your code handles this correctly (formerly, null dynamic values were returned as empty strings).

You can test your code’s compatibility with the new changes, using the client request properties, as in the sample below:

The next major version upgrade of Azure Data Explorer .NET client libraries will include additional breaking changes involving dynamic types. All changes will be described in the changelog of the new version.

REST API Clients

Customers using the REST API directly (either v1 or v2) are potentially impacted by all these changes. The snippet below shows how you can test your code with the new behaviour, using the client request properties:

Python / NodeJS / Java Client Libraries

Customers of these client libraries are impacted by the changes to the REST API raw response, as the libraries return the REST API response as-is, and the caller is responsible for parsing it. Note that Java client library uses /v1 protocol (therefore, is not impacted by change #1), while Python and NodeJS use the /v2 protocol. Please search the client library docs on how to set client request properties when sending a request.

If you are using Azure Data Explorer python client library with data frames (dataframe_from_result_table), make sure to upgrade the client library to version 0.0.17 or later before rollout of phase #1.

-------------------------------------------------------------------------------------------------

Planned change in Azure Data Explorer Retention Policy

This message includes information about an upcoming change in the Retention policy hard delete setting.

Change overview: We started an effort to reduce the storage cost of Azure Data Explorer clusters and we expect that it will result in reduction of 30% of the cost. This effort requires changes to the retention policy implementation, as outlined below.

Customers who are using the Retention policy HardDeletePeriod attribute, should read this message carefully to avoid compatibility issues once the change is deployed.

Change Details

If you’re managing or monitoring the Retention policy, we strongly recommend changing it to comply with the changes detailed below.

Note that the syntax of setting the retention policy will be kept backward compatible, yet the values and the settings returned by the commands will change during the following phases.

The new setting is called Recoverability. Possible values are “Enabled”, “Disabled”

You can start using the new syntax for Retention policy management (.alter and .show commands) now.

Extended syntax to alter Retention Policy:

.alter-merge <entity_type> <database_or_table_name> policy retention

softdelete = <timespan>

recoverability = <enabled|disabled> // new setting

harddelete= <softdelete + 1d | softdelete + 14d>  // kept for backward compatibility

Recommended actions:

• Switch to setting and reading Recoverability setting, note that you must set the HardDeletePeriod to either SoftDeletePeriod + 1d or SoftDeletePeriod + 14d (this is because in phase 1 we will not enforce the new Recoverability setting).
• Remove any read-dependency on Retention policy settings that are obsolete: HardDeletePeriod, ContainerRecyclePeriod, ExtentsDataSizeLimitInBytes, OriginalDataSizeLimitInBytes.

These actions will ensure forward and backward compatibility of your code with upcoming changes.

Phase #1: Extension of the Retention Policy object (ETA: week of January 7, 2019)

• Auto-migration: Azure Data Explorer will auto-migrate tables/databases that were not explicitly migrated during phase #1.

The value of the recoverability setting will be derived from HardDeletePeriod setting:

If HardDeletePeriod - SoftDeletePeriod > 3d, Recoverability will be set to ‘Enabled’.

If HardDeletePeriod - SoftDeletePeriod <= 3d, Recoverability will be set to ‘Disabled’. 

• The effective HardDeletePeriod is derived from Recoverability setting:

If Recoverability is ‘Enabled’, HardDeletePeriod=SoftDeletePeriod+14d

if Recoverability is ‘Disabled’, HardDeletePeriod=SoftDeletePeriod+1d 

Recommended action:

Remove code that sets the HardDeletePeriod setting. Continue using only the new Recoverability setting.

Phase #2: Removal of obsolete settings (ETA: week of January 21, 2019)

• Removal of the following settings of the Retention policy from the results of the “.show/.alter table policy retention” commands

HardDeletePeriod

ContainerRecyclingPeriod

ExtentsDataSizeLimitInBytes

OriginalDataSizeLimitInBytes

-------------------------------------------------------------------------------------------------

Planned change in Azure Data Explorer .NET support

This message includes information about an upcoming change in the .NET supported versions.

Change overview: Azure Data Explorer SDK will require .NET 4.6.2 as the minimal runtime version.

Earlier versions of .NET are deprecated for over a year and we will also stop supporting them.

Customers who are using Azure Data Explorer .NET SDK with .NET runtime lower than 4.6.2 will not be able to get the new versions of the SDK.

Change Details

Azure Data Explorer SDK will require .NET 4.6.2 as the minimal runtime version.

Phase #1: Azure Data Explorer .NET SDK will publish a new version requiring .NET runtime version 4.6.2 or higher (ETA: week of February 04, 2019)

Recommended action:

Upgrade your .NET runtime to 4.6.2 or higher

 Please add your questions / comments about these changes as replies to this post.