Striking the Right Balance: ECN and PFC Thresholds for AI Clusters

Introduction

In large-scale AI training clusters, congestion control is critical to maintaining high GPU utilization and low job completion times. Two key mechanisms in RoCEv2 (RDMA over Converged Ethernet) fabrics are Explicit Congestion Notification (ECN) and Priority Flow Control (PFC). Overly aggressive tuning of ECN marking and PFC pause thresholds – tightening them too much – often backfires. Instead of reducing jitter and tail latency, it can cause instability: pause storms, throughput collapse, oscillations, and “ghost” latency spikes (extreme tail latencies).

ECN and PFC in Congestion Management

Explicit Congestion Notification (ECN) and Priority Flow Control (PFC) are both used to achieve lossless Ethernet fabrics for RDMA (RoCEv2) traffic, but they operate very differently:

• ECN is a proactive congestion signal. When a switch’s queue builds up past a threshold, instead of dropping packets, the switch marks packets with a “Congestion Experienced” (CE) bit in the IP header. The receiving NIC (or host) sees the CE mark and sends a Congestion Notification Packet (CNP) back to the sender, which triggers the sender’s RoCE/DCQCN algorithm to reduce its transmission rate. In essence, ECN asks senders to gently slow down. Importantly, traffic is not stopped; packets continue flowing, but at a throttled rate. This preserves throughput while preventing queues from overflowing. [juniper.net] [cisco.com] [RDMA]
• PFC is a reactive link-level flow control. If a switch egress queue reaches a PFC Xoff threshold (meaning the buffer is close to full), the switch sends a PAUSE frame to upstream devices for that priority class. The upstream NIC halts all packets on that class until a resume (Xon) signal is received. PFC thus prevents packet loss by literally stopping traffic when buffers fill. However, halting traffic has side effects: when transmission resumes, a burst of packets can arrive (since senders had queued more data), causing another congestion episode. In other words, PFC is a blunt instrument – it guarantees no loss, but at the cost of potential jitter and throughput drops if overused. [juniper.net] [RDMA] [cisco.com]

In a well-tuned network, ECN and PFC work together to control congestion: ECN is the first line of defense (gradually throttle sources) and PFC is the last resort (only pause if absolutely necessary to avoid drops). This complementary approach aims to keep traffic flowing as smoothly as possible, without packet loss. Numerous industry designs (from cloud data centers to vendor best practices) emphasize using ECN + PFC in tandem for AI/ML clusters. The key is configuring their thresholds correctly, which we discuss next. 

Correct Congestion Response Sequence: ECN → DCQCN Throttling → PFC (if needed)

In RoCE networks for AI workloads, congestion should ideally be handled in the following sequence:

In summary, the switch first tries “slow down” signals (ECN) and only then resorts to “stop” signals (PFC). Why this order? Because halting traffic (even briefly) can cascade and disturb many flows, whereas marking allows flows to continue even at a slower rate. ECN’s gentle backoff keeps GPUs busy with minimal interruption; PFC’s hard stops are a safety net to avoid drops, not a primary congestion control. All major AI network designs endorse this ordering – it yields higher throughput and lower latency under load. [juniper.net], [engineering.fyi] [cisco.com], [RDMA]

The Pitfall of Aggressive Tuning: When PFC Leads and ECN Lags

In practice, sometimes network operators misconfigure the thresholds such that PFC triggers before ECN has a chance to work. This often happens in an attempt to eliminate nearly all buffering: e.g. setting very low ECN thresholds and even lower PFC Xoff (tiny queue limits). The intention is to minimize queueing delay and quickly signal or pause on any congestion. It sounds logical – “tighten” the control loops to prevent any big queues – but it can dramatically backfire.

What goes wrong when PFC fires too early?

If the PFC pause threshold is set too low, the switch will send a pause frame almost immediately when a burst arrives, before the queue is large enough for ECN marking to kick in. In effect, PFC short-circuits the ECN mechanism. The NIC never gets ECN signals to reduce rate; instead, it gets abruptly paused. When the pause timer expires or the queue drains, the NIC blasts traffic again (since it never slowed down internally) – only to be paused once more. The network ends up in a stop-start cycle.

Below is a conceptual illustration of “wrong ordering” vs “correct ordering” of ECN and PFC:

In the misconfigured case (left), every burst of traffic causes the queue to momentarily hit the tiny Xoff threshold and trigger a pause storm. There is no “grace period” for ECN. The result is a pattern of oscillation: the queue is either near empty (when paused) or spiking (when unpaused), rather than hovering at a moderate level. This oscillation is inefficient and brutal on latency – GPUs will see their message exchanges stall during each pause, then burst, then stall again. These on-off swings manifest as high tail latencies (some iterations take much longer) and underutilized throughput (GPUs can’t keep pipelines full). Operators sometimes refer to these mysterious, periodic delays as “ghost latency spikes” because they are unpredictable and not due to obvious packet loss. 

In the correctly tuned case (right), the ECN marking range might span several hundred KB of queue. That means when a burst arrives, the queue can grow somewhat (tens or hundreds of KB) and the switch will start marking packets with CE well before the buffer is critically full. The RDMA sources quickly get congestion feedback and slow down their sending rate. This keeps the queue below the PFC threshold in most cases – traffic is modulated by ECN/DCQCN continuously (analog control), rather than being slammed by on/off pauses (digital control). Only if a sustained heavy burst overruns the ECN control (e.g. congestion builds faster than feedback can react) will PFC step in to prevent drops. But in a balanced setup, that should be rare. 

In short, aggressive (too low) PFC thresholds combined with early ECN can invert the intended hierarchy. PFC ends up leading and ECN becomes ineffective. This is one of the most common mistakes in initial RoCE deployments for AI clusters and has been observed to cause severe performance issues, as we detail next.

Consequences of Overly Aggressive ECN/PFC Settings

These problems directly translate to poor AI training performance: GPUs wait longer on communications, iteration times become inconsistent (hurting synchronous training), and adding more nodes doesn’t improve throughput as expected (diminishing returns due to network inefficiency). In worst cases, pathological congestion collapse or deadlocks can occur if PFC storms spread (though modern networks often implement PFC watchdogs to guard against deadlock scenarios).

Real-world anecdote: Early deployments of large GPU clusters sometimes intuitively set very small buffer thresholds (to minimize any queuing). The result was often disappointing performance – logs would show incessant PFC pause frames and dramatic swings in network latency. Only after loosening the thresholds to give ECN more breathing room did the networks stabilize and training throughput improve.

A Reasonable Strategy for Tuning ECN and PFC

The key principle for tuning is: Give ECN a chance to work before PFC engages. Practically, this means configuring a substantial ECN marking range and a PFC pause threshold that is higher (in terms of queue depth), but not so high as to risk actual packet drops. Here’s a typical strategy employed by experienced AI network operators:

• On the Switch (ECN/WRED configuration): Enable ECN marking on the RDMA traffic class and set a two-tier threshold:
• ECN Start Threshold: when the RDMA egress queue reaches a moderate occupancy (for example, a few hundred KB per port), begin probabilistic marking of packets with CE. This threshold should be low enough that congestion is signaled early, but not so low that it marks on trivial load. Many deployments choose on the order of 100–300 KB as an ECN onset point for 100 GbE links. [cisco.com]
• ECN Full (100% Marking) Threshold: at a higher queue depth (e.g. approaching tail of the buffer), ramp up to mark all packets. For instance, one might set 100% marking by around 2–3 MB of queue length on a 100 Gb switch. This ensures that if a queue keeps growing, all flows get signaled aggressively before any packet would be dropped. 
• (Optional) ECN Random Early Detection (RED) profile: Tune the marking probability curve (linear or whatever the switch OS uses) between the start and full thresholds to avoid too bursty marking. The goal is a smooth increase in marking rate as the queue builds.

Rationale: These settings create an “ECN region” in the buffer where congestion is increasingly signaled. The NICs receiving CE marks will back off using DCQCN. In effect, the network tries to self-throttle long before buffers overflow.

• On the Switch (PFC threshold configuration): Adjust the PFC Xoff threshold for the lossless queue to just above the ECN full marking threshold – essentially the minimum headroom needed to catch any in-flight packets that are already in the pipe after ECN has signaled. In practice, this Xoff might be quite small when measured in absolute KB:
• For example, in some cloud data centers using 100 Gb links, PFC pause is triggered when a queue reaches only on the order of a few dozen KB beyond the ECN range. This small headroom (tens of KB) is just enough to absorb packets already on the wire or in NIC buffers while the CNP feedback is making its way back to senders. The exact number depends on link speed and RTT – faster links and bigger RTT require slightly more headroom.
• Set the PFC Xon (resume) threshold appropriately below Xoff (accounting for hysteresis) so that once the pause is issued and the queue drains, the resume happens at a safe low level to avoid thrashing.

Following this strategy yields a sequence on congestion like: (1) mark with CE, (2) sources reduce rate, (3) if and only if the rate reduction isn’t quick enough, then pause). In numerical terms, for a 100 Gbps link, an example configuration might be: [cisco.com], [engineering.fyi]

• ECN marking starts at ~150 KB, ramps to 100% by ~3,000 KB (as in one Cisco reference design). [cisco.com]
• PFC Xoff maybe ~3,100 KB (just above that, to catch any excess).
• PFC Xon a bit lower (say ~2,500 KB) to resume once drained sufficiently.

On a different switch with smaller buffers, the absolute numbers might be different, but the principle of separation remains – hundreds of KB (or several microseconds worth of buffering) for ECN, and only a small additional margin for PFC.

Crucially, operators have learned that it is better to allow some buffering (a few hundred KB is trivial relative to typical 12-16 MB buffers on a switch port) in order to achieve stable performance. Trying to eliminate buffering entirely is counterproductive – a bit of queue is not a bad thing if it’s used to signal congestion early via ECN.

Example ECN/PFC Threshold Configuration

# Example ECN/PFC Threshold Configuration (Conceptual - adapt to your switch OS)
# For 100 GbE RoCEv2 traffic class

# ECN Marking Thresholds
ecn-marking-threshold start 150KB
ecn-marking-threshold full 3000KB
ecn-marking-probability linear

# PFC Thresholds (must be > ECN full threshold)
pfc-xoff-threshold 3100KB
pfc-xon-threshold 2500KB

# Verify with: show qos ecn-statistics

Closing: Best Practices and Takeaways

In complex GPU training networks, it may feel natural to try to squeeze out all queueing for lowest latency. However, the job of the network engineer is not to eliminate buffering entirely, but to use buffering intelligently. A small amount of buffering coupled with ECN allows the network to continuously flow at high utilization, while still avoiding packet loss. PFC, on the other hand, is your safety brake – engage it only as a last resort.

To recap the best practices for congestion control in RoCE-based AI clusters:

• ECN is your first responder; PFC is your last resort. Always configure so that ECN marking happens before any PFC pause would trigger. This typically means a larger threshold for ECN and a much smaller one for PFC. Most tail latency explosions in large GPU clusters have been traced to overuse of PFC at the expense of ECN. [juniper.net], [engineering.fyi]
• Don’t overtighten the leash. Aggressive settings (tiny ECN/PFC values) can harm stability. You want a smooth, analog congestion control, not an on/off digital pattern. Some queueing is not only OK, but necessary for the control loop to function.
• Tune using empirical feedback. Start with recommended values (e.g. from a vendor or reference design) and then observe: If you still see frequent PFC frames, you might need to increase ECN marking aggressiveness (or lower the start threshold a bit more). If you see no PFC at all but high switch queue occupancy, you might lower the ECN thresholds slightly or consider if PFC should trigger a bit sooner. The perfect balance can depend on the workload patterns (burstiness, message sizes), so fine-tuning in a test environment with representative traffic is wise.

In conclusion, avoiding the drawbacks of aggressive ECN/PFC tuning comes down to respecting the natural roles of these mechanisms. Use ECN to handle the “moderate” congestion and keep traffic flowing, and reserve PFC only for the “oh no” moments. This hierarchical approach yields far better throughput, lower latency, and more predictable performance for distributed AI workloads. By configuring a generous ECN marking range and minimal necessary PFC headroom, you can largely eliminate the pause-induced tail latency blowups and oscillations that plague poorly tuned RoCE fabrics. The winning pattern emerging across the industry is exactly that: ECN as proactive congestion control, PFC as the backup plan. Following this pattern will help ensure your AI training clusters run at scale with smooth, high-performance networking. [cisco.com], [cisco.com]

Next Steps: Apply These Practices

Getting threshold design right is not a one-time exercise – it requires lab validation, production telemetry, and periodic retuning as link speeds, buffer architectures, and workload patterns evolve.

🔧 Get Started Today: Hands-on Configuration

Map the conceptual threshold model from this article to your switch OS using vendor implementation guides:

• Cisco Nexus QoS Configuration Guide for RoCEv2 – Step-by-step threshold configuration for NX-OS platforms
• Juniper AI Data Center Deployment Guide – Complete ECN/PFC setup examples for Junos-based fabrics

These references will help you translate queueing intent into platform-specific commands, units, and validation steps.

📊 Validate Your Setup

After deployment, verify that your fabric behaves the way you intended:

• Use show interface priority-flow-control (Cisco) or the equivalent command on your switch OS to monitor PFC frame counts
• Track ECN marking rates via switch telemetry or queue statistics dashboards
• Key signal: If PFC triggers frequently but ECN marks are low, your thresholds need adjustment – ECN isn’t getting a chance to act before PFC takes over

💬 Join the Discussion

• Share your tuning experiences in the comments below
• Have a specific topology challenge? Describe it and the community can help troubleshoot

References

1. Juniper Networks – “Introduction to Congestion Control in Juniper AI Networks”, Best Practices for ECN/PFC (2025). [juniper.net], [juniper.net]
2. Cisco – “Validated Design for Networking AI/ML Applications”, QoS Configuration for RoCEv2 (2023). [cisco.com], [cisco.com]
3. Edgecore Networks – “Powering AI at Scale: Role of PFC and ECN for Large GPU Deployments” (Blog, 2025). [cisco.com], [engineering.fyi]
4. Meta (Facebook) – “RoCE Networks for Distributed AI Training at Scale” (Engineering Blog, 2024).
5. Calnex Solutions – “Understanding Network Tail Latency in AI Training Data Centers” (Blog, 2023).
6. Arista Networks – “High-Performance Ethernet for AI Systems” (Deployment Guide, 2023).
7. Hoefler2023-RDMA: “Datacenter Ethernet and RDMA: Issues at Hyperscale,” arXiv:2302.03337, 2023. Available: https://arxiv.org/abs/2302.03337