System Table Retention Is Now Your Choice, Not a Fixed Number

Long-term auditing based on system tables has always faced the same challenge: sometimes, the data simply wasn't available for as long as organizations needed it.

Azure Databricks has introduced, in Beta, configurable retention for supported system tables, giving account administrators greater control over how long historical system data is retained.

Instead of relying only on the platform's default retention period, administrators can now configure retention from 30 to 3,650 days — up to approximately 10 years.

This is particularly relevant for organizations using system tables for FinOps, auditing, governance, security investigations, usage analysis, and long-term operational analytics.

Official documentation:

https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/

What changes with configurable retention?

When configurable retention is enabled, supported system tables can use an account-level retention configuration.

The main characteristics are:

• Retention can be configured from 30 to 3,650 days.
• The configuration is managed at the Databricks account level.
• An account administrator is required to manage the setting.
• The configuration applies only to system tables that support configurable retention.
• During the Beta period, configurable retention is available without additional retention storage charges.
• When the feature is enabled, supported system tables receive 395 days of retention by default.

This gives organizations considerably more flexibility when designing long-term governance and observability strategies around system tables.

Configuring retention using the API

In addition to managing retention through the Databricks Account Console, account administrators can configure it programmatically using the Databricks Account Settings API.

The setting responsible for system table retention is called:

st_retention

Administrators can then change the retention period using a PATCH request.

For example, the following configuration sets the retention period to 400 days:

curl --request PATCH \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $token" \
  --data '{
    "integer_val": {
      "value": 400
    },
    "name": "st_retention"
  }' \
  "<account-console-url>/api/2.1/accounts/<account-id>/settings/st_retention"

This is particularly useful for organizations that manage their Databricks environments using automation and Infrastructure as Code practices, since retention policies can become part of a broader governance process.

Official documentation:

https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/

Increasing and decreasing retention behave differently

Azure Databricks also includes an important protection mechanism when changing retention settings.

When administrators increase the retention period, the new configuration takes effect immediately.

However, when administrators decrease the retention period, Azure Databricks provides a seven-day grace period before applying the reduction.

This provides an opportunity to correct an accidental configuration change before historical data is permanently removed.

For example:

Current retention: 1,000 days

New retention: 365 days

Grace period: 7 days

During this grace period, an administrator can reconsider or correct the configuration before the shorter retention policy takes effect.

Records that exceed the configured retention period are generally removed within approximately one week after becoming eligible for deletion.

Increasing retention does not restore deleted data

There is another important behavior to understand.

Increasing the retention period does not restore historical records that have already expired or been removed.

For example, imagine that an organization previously retained system table information for 365 days.

Later, the administrator changes the configuration to:

3,650 days

This does not cause Azure Databricks to recreate or backfill data from previous years.

The new retention period applies to data that is still available and to data generated going forward.

Therefore, organizations that require long-term auditing should define their retention strategy before historical information becomes unavailable.

Not every system table supports configurable retention

Configurable retention does not currently apply to every system table.

According to the current documentation, tables in the following schemas are excluded:

system.data_classification

system.data_quality_monitoring

These system tables continue to follow their own retention policies.

Because this capability is currently in Beta, supported tables and behavior may evolve as the feature moves toward General Availability.

Always check the latest documentation before defining production retention policies:

https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/

What about cost?

During the Beta period, configurable system table retention is available without additional retention storage charges.

However, Databricks states that when the feature becomes Generally Available (GA), data retained beyond 395 days will be subject to storage charges.

This turns retention into more than just a technical configuration. It becomes a FinOps and governance decision.

For example, configuring:

Retention = 3,650 days

means maintaining approximately 10 years of system table history.

That could be extremely valuable for regulatory auditing, historical usage analysis, security investigations, or long-term FinOps analysis.

But organizations should evaluate whether that historical depth provides enough business value to justify the associated storage cost once the capability becomes generally available.

Why does this matter?

System tables provide operational information about the Databricks environment and can support use cases such as:

• Cost and usage analysis
• FinOps dashboards
• Audit analysis
• Security investigations
• Governance monitoring
• Query history analysis
• Compute monitoring
• Marketplace activity
• Data sharing activity
• Operational observability

For many organizations, 395 days may be enough.

For others — especially organizations operating in highly regulated environments — retaining several years of historical information can be an important compliance requirement.

The possibility of configuring up to 3,650 days allows the retention strategy to better reflect the organization's actual governance requirements.

A governance decision, not just a configuration

The most interesting aspect of this feature is that system table retention is becoming an explicit architectural decision.

Instead of simply accepting a predefined retention period, Data Platform teams can ask:

How much history do we actually need?

Which system tables are important for auditing?

What are our regulatory requirements?

How much historical data is useful for FinOps?

What will be the storage impact?

Who should be allowed to change the retention policy?

These questions move system table retention from a platform limitation into the organization's Data Governance and FinOps strategy.

A reasonable approach is to define retention according to the purpose of the data rather than automatically selecting the maximum available value.

Final thoughts

Configurable system table retention may look like a relatively small platform feature, but it solves a very practical problem.

Teams often build dashboards, audit processes, FinOps reports, and operational analytics on top of system tables only to discover later that the historical information they need is no longer available.

With configurable retention, Azure Databricks provides organizations with much more control over this lifecycle.

The possibility of retaining system table information for anywhere between 30 days and 3,650 days makes it possible to align historical availability with business, regulatory, security, and operational requirements.

At the same time, the maximum value should not automatically become the default.

Retention is now a choice — and that choice should be part of your Data Governance strategy.

References

Azure Databricks — System tables:

https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/

Azure Databricks documentation:

https://learn.microsoft.com/en-us/azure/databricks/

Databricks System Tables documentation:

https://docs.databricks.com/aws/en/admin/system-tables

Databricks September 2026 product release notes:

https://docs.databricks.com/aws/en/release-notes/product/2026/september