By-pass hub firewall for data centric solution

Take this as best practices for Firewall Bypass Scenarios:

1. Use Private Endpoints and VNet Peering

It was done

2. Apply NSGs and Route Tables Strategically

• Use Network Security Groups (NSGs) to tightly control traffic at the subnet level.
• Define User Defined Routes (UDRs) to ensure traffic flows as intended, either through the firewall or directly between VNets.

3. Segment Workloads with Spoke VNets

• Keep Kafka and Databricks in separate spoke VNets.
• Peer them directly if needed, but ensure NSGs and route tables enforce least privilege.

4. Use Azure Firewall selectively

• Route management traffic, internet-bound traffic, or non-performance-critical flows through the firewall.
• Bypass only data-plane traffic that requires high throughput.

5. Monitor and Audit Extensively

• Use Azure Monitor, Network Watcher, and Private Link diagnostics to track traffic flows.
• Enable logging on NSGs and Databricks audit logs to ensure visibility.

6. Consider Azure Firewall Premium or Third-Party NVA

If you must route through a firewall but need better performance, consider:

• Azure Firewall Premium: Offers TLS inspection and better throughput.
• Third-party Network Virtual Appliances (NVAs): Some offer optimized performance for data-heavy workloads.