JSON Structure: A JSON schema language you'll love
We talk to many customers moving structured data through queues and event streams and topics, and we see a strong desire to create more efficient and less brittle communication paths governed by rich data definitions well understood by all parties. The way those definitions are often shared are schema documents. While there is great need, the available schema options and related tool chains are often not great. JSON Schema is popular for its relative simplicity in trivial cases, but quickly becomes unmanageable as users employ more complex constructs. The industry has largely settled on "Draft 7," with subsequent releases seeing weak adoption. There's substantial frustration among developers who try to use JSON Schema for code generation or database mapping—scenarios it was never designed for. JSON Schema is a powerful document validation tool, but it is not a data definition language. We believe it's effectively un-toolable for anything beyond pure validation; practically all available code-generation tools agree by failing at various degrees of complexity. Avro and Protobuf schemas are better for code generation, but tightly coupled to their respective serialization frameworks. For our own work in Microsoft Fabric, we're initially leaning on an Avro-compatible schema with a small set of modifications, but we ultimately need a richer type definition language that ideally builds on people's familiarity with JSON Schema. This isn't just a Microsoft problem. It's an industry-wide gap. That's why we've submitted JSON Structure as a set of Internet Drafts to the IETF, aiming for formal standardization as an RFC. We want a vendor-neutral, standards-track schema language that the entire industry can adopt. What Is JSON Structure? JSON Structure is a modern, strictly typed data definition language that describes JSON-encoded data such that mapping to and from programming languages and databases becomes straightforward. It looks familiar—if you've written "type": "object", "properties": {...} before, you'll feel right at home. But there's a key difference: JSON Structure is designed for code generation and data interchange first, with validation as an optional layer rather than the core concern. This means you get: Precise numeric types: int32 , int64 , decimal with precision and scale, float , double Rich date/time support: date , time , datetime , duration —all with clear semantics Extended compound types: Beyond objects and arrays, you get set , map , tuple , and choice (discriminated unions) Namespaces and modular imports: Organize your schemas like code Currency and unit annotations: Mark a decimal as USD or a double as kilograms Here's a compact example that showcases these features. We start with the schema header and the object definition: { "$schema": "https://json-structure.org/meta/extended/v0/#", "$id": "https://example.com/schemas/OrderEvent.json", "name": "OrderEvent", "type": "object", "properties": { Objects require a name for clean code generation. The $schema points to the JSON Structure meta-schema, and the $id provides a unique identifier for the schema itself. Now let's define the first few properties—identifiers and a timestamp: "orderId": { "type": "uuid" }, "customerId": { "type": "uuid" }, "timestamp": { "type": "datetime" }, The native uuid type maps directly to Guid in .NET, UUID in Java, and uuid in Python. The datetime type uses RFC3339 encoding and becomes DateTimeOffset in .NET, datetime in Python, or Date in JavaScript. No format strings, no guessing. Next comes the order status, modeled as a discriminated union: "status": { "type": "choice", "choices": { "pending": { "type": "null" }, "shipped": { "type": "object", "name": "ShippedInfo", "properties": { "carrier": { "type": "string" }, "trackingId": { "type": "string" } } }, "delivered": { "type": "object", "name": "DeliveredInfo", "properties": { "signedBy": { "type": "string" } } } } }, The choice type is a discriminated union with typed payloads per case. Each variant can carry its own structured data— shipped includes carrier and tracking information, delivered captures who signed for the package, and pending carries no payload at all. This maps to enums with associated values in Swift, sealed classes in Kotlin, or tagged unions in Rust. For monetary values, we use precise decimals: "total": { "type": "decimal", "precision": 12, "scale": 2 }, "currency": { "type": "string", "maxLength": 3 }, The decimal type with explicit precision and scale ensures exact monetary math—no floating-point surprises. A precision of 12 with scale 2 gives you up to 10 digits before the decimal point and exactly 2 after. Line items use an array of tuples for compact, positional data: "items": { "type": "array", "items": { "type": "tuple", "properties": { "sku": { "type": "string" }, "quantity": { "type": "int32" }, "unitPrice": { "type": "decimal", "precision": 10, "scale": 2 } }, "tuple": ["sku", "quantity", "unitPrice"], "required": ["sku", "quantity", "unitPrice"] } }, Tuples are fixed-length typed sequences—ideal for time-series data or line items where position matters. The tuple array specifies the exact order: SKU at position 0, quantity at 1, unit price at 2. The int32 type maps to int in all mainstream languages. Finally, we add extensible metadata using set and map types: "tags": { "type": "set", "items": { "type": "string" } }, "metadata": { "type": "map", "values": { "type": "string" } } }, "required": ["orderId", "customerId", "timestamp", "status", "total", "currency", "items"] } The set type represents unordered, unique elements—perfect for tags. The map type provides string keys with typed values, ideal for extensible key-value metadata without polluting the main schema. Here's what a valid instance of this schema looks like: { "orderId": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "customerId": "7c9e6679-7425-40de-944b-e07fc1f90ae7", "timestamp": "2025-01-15T14:30:00Z", "status": { "shipped": { "carrier": "Litware", "trackingId": "794644790323" } }, "total": "129.97", "currency": "USD", "items": [ ["SKU-1234", 2, "49.99"], ["SKU-5678", 1, "29.99"] ], "tags": ["priority", "gift-wrap"], "metadata": { "source": "web", "campaign": "summer-sale" } } Notice how the choice is encoded as an object with a single key indicating the active case— {"shipped": {...}} —making it easy to parse and route. Tuples serialize as JSON arrays in the declared order. Decimals are encoded as strings to preserve precision across all platforms. Why Does This Matter for Messaging? When you're pushing events through Service Bus, Event Hubs, or Event Grid, schema clarity is everything. Your producers and consumers often live in different codebases, different languages, different teams. A schema that generates clean C# classes, clean Python dataclasses, and clean TypeScript interfaces—from the same source—is not a luxury. It's a requirement. JSON Structure's type system was designed with this polyglot reality in mind. The extended primitive types map directly to what languages actually have. A datetime is a DateTimeOffset in .NET, a datetime in Python, a Date in JavaScript. No more guessing whether that "string with format date-time" will parse correctly on the other side. SDKs Available Now We've built SDKs for the languages you're using today: TypeScript, Python, .NET, Java, Go, Rust, Ruby, Perl, PHP, Swift, and C. All SDKs validate both schemas and instances against schemas. A VS Code extension provides IntelliSense and inline diagnostics. Code and Schema Generation with Structurize Beyond validation, you often need to generate code or database schemas from your type definitions. The Structurize tool converts JSON Structure schemas into SQL DDL for various database dialects, as well as self-serializing classes for multiple programming languages. It can also convert between JSON Structure and other schema formats like Avro, Protobuf, and JSON Schema. Here's a simple example: a postal address schema on the left, and the SQL Server table definition generated by running structurize struct2sql postaladdress.json --dialect sqlserver on the right: JSON Structure Schema Generated SQL Server DDL { "$schema": "https://json-structure.org/meta/extended/v0/#", "$id": "https://example.com/schemas/PostalAddress.json", "name": "PostalAddress", "description": "A postal address for shipping or billing", "type": "object", "properties": { "id": { "type": "uuid", "description": "Unique identifier for the address" }, "street": { "type": "string", "description": "Street address with house number" }, "city": { "type": "string", "description": "City or municipality" }, "state": { "type": "string", "description": "State, province, or region" }, "postalCode": { "type": "string", "description": "ZIP or postal code" }, "country": { "type": "string", "description": "ISO 3166-1 alpha-2 country code" }, "createdAt": { "type": "datetime", "description": "When the address was created" } }, "required": ["id", "street", "city", "postalCode", "country"] } CREATE TABLE [PostalAddress] ( [id] UNIQUEIDENTIFIER, [street] NVARCHAR(200), [city] NVARCHAR(100), [state] NVARCHAR(50), [postalCode] NVARCHAR(20), [country] NVARCHAR(2), [createdAt] DATETIME2, PRIMARY KEY ([id], [street], [city], [postalCode], [country]) ); EXEC sp_addextendedproperty 'MS_Description', 'A postal address for shipping or billing', 'SCHEMA', 'dbo', 'TABLE', 'PostalAddress'; EXEC sp_addextendedproperty 'MS_Description', 'Unique identifier for the address', 'SCHEMA', 'dbo', 'TABLE', 'PostalAddress', 'COLUMN', 'id'; EXEC sp_addextendedproperty 'MS_Description', 'Street address with house number', 'SCHEMA', 'dbo', 'TABLE', 'PostalAddress', 'COLUMN', 'street'; -- ... additional column descriptions The uuid type maps to UNIQUEIDENTIFIER , datetime becomes DATETIME2 , and the schema's description fields are preserved as SQL Server extended properties. The tool supports PostgreSQL, MySQL, SQLite, and other dialects as well. Mind that all this code is provided "as-is" and is in a "draft" state just like the specification set. Feel encouraged to provide feedback and ideas in the GitHub repos for the specifications and SDKs at https://github.com/json-structure/ Learn More We've submitted JSON Structure as a set of Internet Drafts to the IETF, aiming for formal standardization as an RFC. This is an industry-wide issue, and we believe the solution needs to be a vendor-neutral standard. You can track the drafts at the IETF Datatracker. Main site: json-structure.org Primer: JSON Structure Primer Core specification: JSON Structure Core Extensions: Import | Validation | Alternate Names | Units | Composition IETF Drafts: IETF Datatracker GitHub: github.com/json-structure
Azure Event Grid Domain Creation: Overcoming AZ CLI's TLS Parameter Limitations with Workaround
Introduction: The Intersection of Security Policies and DevOps Automation In the modern cloud landscape, organizations increasingly enforce strict security requirements through platform policies. One common requirement is mandating latest TLS versions for example TLS 1.2 across all deployed resources to protect data in transit. While this is an excellent security practice, it can sometimes conflict with the available configuration options in deployment tools, particularly in the Azure CLI. This blog explores a specific scenario that many Azure DevOps teams encounter: how to deploy an Azure Event Grid domain when your organization has a custom policy requiring latest version considering TLS 1.2, but the Azure CLI command doesn't provide a parameter to configure this setting. The Problem: Understanding the Gap Between Policy and Tooling What Is Azure Event Grid? Azure Event Grid is a serverless event routing service that enables event-driven architectures. It manages the routing of events from various sources (like Azure services, custom applications, or SaaS products) to different handlers such as Azure Functions, Logic Apps, or custom webhooks. An Event Grid domain provides a custom topic endpoint that can receive events from multiple sources, offering a way to organize and manage events at scale. The Policy Requirement: Many organizations implement Azure Policy to enforce security standards across their cloud infrastructure. A common policy might look like this: { "policyRule": { "if": { "allOf": [ { "field": "type", "equals": "Microsoft.EventGrid/domains" }, { "anyOf": [ { "field": "Microsoft.EventGrid/domains/minimumTlsVersion", "exists": false }, { "field": "Microsoft.EventGrid/domains/minimumTlsVersion", "notEquals": "1.2" } ] } ] }, "then": { "effect": "deny" } } } This policy blocks the creation of any Event Grid domain that doesn't explicitly set TLS 1.2 as the minimum TLS version. The CLI Limitation: Now, let's examine the Azure CLI command to create an Event Grid domain: az eventgrid domain | Microsoft Learn TLS property is unrecognized with the latest version of AZ CLI version. Current Status of This Limitation: It's worth noting that this limitation has been recognized by the Azure team. There is an official GitHub feature request tracking this issue, which you can find at => Please add TLS support while creation of Azure Event Grid domain through CLI · Issue #31278 · Azure/azure-cli Before implementing this workaround described in this article, I recommend checking the current status of this feature request. The Azure CLI is continuously evolving, and by the time you're reading this, the limitation might have been addressed. However, as of April 2025, this remains a known limitation in the Azure CLI, necessitating the alternative approach outlined below. Why This Matters: This limitation becomes particularly problematic in CI/CD pipelines or Infrastructure as Code (IaC) scenarios where you want to automate the deployment of Event Grid domain resources. Workaround: You can utilize below ARM template and deploy it through AZ CLI in your deployment pipeline as below: Working ARM template: { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "domainName": { "type": "string", "metadata": { "description": "Name of the Event Grid Domain" } }, "location": { "type": "string", "defaultValue": "[resourceGroup().location]", "metadata": { "description": "Azure region for the domain" } } }, "resources": [ { "type": "Microsoft.EventGrid/domains", "apiVersion": "2025-02-15", "name": "[parameters('domainName')]", "location": "[parameters('location')]", "properties": { "minimumTlsVersionAllowed": "1.2" } } ] } Please note I've used latest API version from below official Microsoft documentation : Microsoft.EventGrid/domains - Bicep, ARM template & Terraform AzAPI reference | Microsoft Learn Working AZ CLI command: az deployment group create --resource-group <rg> --template-file <armtemplate.json> --parameters domainName=<event grid domain name> You can store this ARM template in your configuration directory with replacement for Azure CLI command. It explicitly sets TLS 1.2 for Event Grid domains, ensuring security compliance where the CLI lacks this parameter. For example: az deployment group create --resource-group <rg> --template-file ./config/<armtemplate.json> --parameters domainName=<event grid domain name> Disclaimer: The sample scripts provided in this article are provided AS IS without warranty of any kind. The author is not responsible for any issues, damages, or problems that may arise from using these scripts. Users should thoroughly test any implementation in their environment before deploying to production. Azure services and APIs may change over time, which could affect the functionality of the provided scripts. Always refer to the latest Azure documentation for the most up-to-date information. Thanks for reading this blog! I hope you've found this workaround valuable for addressing the Event Grid domain TLS parameter limitation in Azure CLI. 😊Azure Event Grid CLI Identity Gaps & Workarounds with Python REST and ARM Templates
Azure Event Grid has become a cornerstone service for building event-driven architectures in the cloud. It provides a scalable event routing service that enables reactive programming patterns, connecting event sources to event handlers seamlessly. However, when working with Event Grid through the Azure CLI, developers often encounter a significant limitation: the inability to configure system-assigned managed identities using CLI commands. In this blog post, I'll explore this limitation and provide practical workarounds using Python REST API calls and ARM templates with CLI to ensure your Event Grid deployments can leverage the security benefits of managed identities without being blocked by tooling constraints. Problem Statement: Unlike many other Azure resources that support the --identity or ---assign-identity parameter for enabling system-assigned managed identities, Event Grid's CLI commands lack this capability while creating event subscription for system topic at the moment. This means that while the Azure Portal and other methods support managed identities for Event Grid, you can't configure them directly through the CLI in case of system topic event subscriptions For example you can add managed identity for delivery through portal but not through AZ CLI: If you try to use the following CLI command: az eventgrid system-topic event-subscription create \ --name my-sub \ --system-topic-name my-topic \ --resource-group my-rg \ --endpoint <EH resource id> --endpoint-type eventhub \ --identity systemassigned You'll run into a limitation: The --identity flag is not supported or unrecognized for system topic subscriptions in Azure CLI. Also, --delivery-identity is in preview and under development Current Status of This Limitation: It's worth noting that this limitation has been recognized by the Azure team. There is an official GitHub feature request tracking this issue, which you can find at Use managed identity to command creates an event subscription for an event grid system topic · Issue #26910 · Azure/azure-cli. Before implementing any of the workarounds described in this article, I recommend checking the current status of this feature request. The Azure CLI is continuously evolving, and by the time you're reading this, the limitation might have been addressed. However, as of April 2025, this remains a known limitation in the Azure CLI, necessitating the alternative approaches outlined below. Why This Matters: This limitation becomes particularly problematic in CI/CD pipelines or Infrastructure as Code (IaC) scenarios where you want to automate the deployment of Event Grid resources with managed identities. Solution 1: Using Azure REST API with Python request library: The first approach to overcome this limitation is to use the Azure REST API with Python. This provides the most granular control over your Event Grid resources and allows you to enable system-assigned managed identities programmatically. System Topic Event Subscriptions - Create Or Update - REST API (Azure Event Grid) | Microsoft Learn You can retrieve Azure Entra token using below CLI command: az account get-access-token Sample working code & payload: import requests import json subscription_id = <> resource_group = <> system_topic_name = <> event_subscription_name = <> event_hub_resource_id = <> access_token = <> url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.EventGrid/systemTopics/{system_topic_name}/eventSubscriptions/{event_subscription_name}?api-version=2024-12-15-preview" payload = { "identity": { "type": "SystemAssigned" }, "properties": { "topic": "/subscriptions/<>/resourceGroups/<>/providers/Microsoft.EventGrid/systemTopics/<>", "filter": { "includedEventTypes": [ "Microsoft.Storage.BlobCreated", "Microsoft.Storage.BlobDeleted" ], "advancedFilters": [], "enableAdvancedFilteringOnArrays": True }, "labels": [], "eventDeliverySchema": "EventGridSchema", "deliveryWithResourceIdentity": { "identity": { "type": "SystemAssigned" }, "destination": { "endpointType": "EventHub", "properties": { "resourceId": "/subscriptions/<>/resourceGroups/rg-sch/providers/Microsoft.EventHub/namespaces/<>/eventhubs/<>", "deliveryAttributeMappings": [ { "name": "test", "type": "Static", "properties": { "value": "test", "isSecret": False, "sourceField": "" } }, { "name": "id", "type": "Dynamic", "properties": { "value": "abc", "isSecret": False, "sourceField": "data.key" } } ] } } } } } headers = { "Authorization": f"Bearer {access_token}", "Content-Type": "application/json" } response = requests.put(url, headers=headers, data=json.dumps(payload)) if response.status_code in [200, 201]: print("Event subscription created successfully!") Remember that these tokens are sensitive security credentials, so handle them with appropriate care. They should never be exposed in logs, shared repositories, or other insecure locations. Solution 2: Using ARM Templates & deploying it through CLI Another solution is to use Azure Resource Manager (ARM) templates, which fully support system-assigned managed identities for Event Grid. This approach works well in existing IaC workflows. Microsoft.EventGrid/systemTopics/eventSubscriptions - Bicep, ARM template & Terraform AzAPI reference | Microsoft Learn Here's a sample ARM template that creates an Event Grid topic with a system-assigned managed identity: { "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "systemTopicName": { "type": "string", "metadata": { "description": "Name of the existing system topic" } }, "eventSubscriptionName": { "type": "string", "metadata": { "description": "Name of the event subscription to create" } }, "eventHubResourceId": { "type": "string", "metadata": { "description": "Resource ID of the Event Hub to send events to" } }, "includedEventType": { "type": "string", "defaultValue": "Microsoft.Storage.BlobCreated", "metadata": { "description": "Event type to filter on" } } }, "resources": [ { "type": "Microsoft.EventGrid/systemTopics/eventSubscriptions", "apiVersion": "2024-06-01-preview", "name": "[format('{0}/{1}', parameters('systemTopicName'), parameters('eventSubscriptionName'))]", "identity": { "type": "SystemAssigned" }, "properties": { "deliveryWithResourceIdentity": { "destination": { "endpointType": "EventHub", "properties": { "resourceId": "[parameters('eventHubResourceId')]" } }, "identity": { "type": "SystemAssigned" } }, "eventDeliverySchema": "EventGridSchema", "filter": { "includedEventTypes": [ "[parameters('includedEventType')]" ] } } } ] } How to deploy via Azure CLI: az deployment group create \ --resource-group <your-resource-group> \ --template-file eventgridarmtemplate.json \ --parameters \ systemTopicName=<system-topic-name> \ eventSubscriptionName=<event-subscription-name> \ eventHubResourceId="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.EventHub/namespaces/<namespace>/eventhubs/<hub>" Disclaimer The sample scripts provided in this article are provided AS IS without warranty of any kind. The author is not responsible for any issues, damages, or problems that may arise from using these scripts. Users should thoroughly test any implementation in their environment before deploying to production. Azure services and APIs may change over time, which could affect the functionality of the provided scripts. Always refer to the latest Azure documentation for the most up-to-date information. Thanks for reading this blog! I hope you've found these workarounds valuable for addressing the Event Grid identity parameter limitation in Azure CLI. 😊
Vector Image Search using Azure OpenAI & AI Search: A Technical Guide
Discover how AI is transforming image retrieval with our comprehensive guide on combining Azure OpenAI and Azure AI Search. Learn to set up, deploy, and utilize these powerful tools to enhance your applications, from e-commerce to digital asset management. Dive into the core functionalities of the search and vectorize methods in the function_app.py Python file, and see how Azure’s AI services can revolutionize your image search capabilities.Azure Integration Services Quarterly Highlights and Insights 2024'Q1
Welcome to the first in a new series of quarterly blogs that will feature the latest and greatest from Azure Integration Services. With so many exciting new Azure products, Microsoft events, and updates from our partners and customers throughout the year, this series is your chance to revisit the highlights that continue to enable digital transformation with Azure Integration Services.
