Forum Discussion

taggar4's avatar
taggar4
Copper Contributor
Mar 08, 2023

By-pass hub firewall for data centric solution

I have the following high-level solution:

Source Data - flowing via kafka cluster on a series of topics

Destination - Databricks via spark connector landing in ADLS Gen2

The source and destination are in separate subscriptions within the same tenant and private endpoint will be used to ensure data flow is over a private network. VNet peering will be used and possibly NSGs on the respective subnets. However, the hub firewall will be by-passed due to the volume and speed of the data. Security controls will be used to mitigate access on source and destination. I would like to understand the best practice for similar use-cases, should we by-pass the central firewall? The rationale is that the firewall will impact the data flow / performance. 

1 Reply

  • Kidd_Ip's avatar
    Kidd_Ip
    MVP
    Oct 01, 2025

    Take this as best practices for Firewall Bypass Scenarios:

    1. Use Private Endpoints and VNet Peering

    It was done

    1. Apply NSGs and Route Tables Strategically
    • Use Network Security Groups (NSGs) to tightly control traffic at the subnet level.
    • Define User Defined Routes (UDRs) to ensure traffic flows as intended, either through the firewall or directly between VNets.
    1. Segment Workloads with Spoke VNets
    • Keep Kafka and Databricks in separate spoke VNets.
    • Peer them directly if needed, but ensure NSGs and route tables enforce least privilege.
    1. Use Azure Firewall selectively
    • Route management traffic, internet-bound traffic, or non-performance-critical flows through the firewall.
    • Bypass only data-plane traffic that requires high throughput.
    1. Monitor and Audit Extensively
    • Use Azure Monitor, Network Watcher, and Private Link diagnostics to track traffic flows.
    • Enable logging on NSGs and Databricks audit logs to ensure visibility.
    1. Consider Azure Firewall Premium or Third-Party NVA

    If you must route through a firewall but need better performance, consider:

    • Azure Firewall Premium: Offers TLS inspection and better throughput.
    • Third-party Network Virtual Appliances (NVAs): Some offer optimized performance for data-heavy workloads.

Resources