A system for real-time detection and control of microburst congestion and culprit flows in networks

Background

Modern high-speed networks are increasingly challenged by transient, high-intensity traffic surges known as microbursts. These microbursts, which can last from hundreds of microseconds to a few milliseconds, cause sudden spikes in queue utilization within network switches and routers. The impact is particularly acute in environments such as data centers, backbone networks, and latency-sensitive applications (e.g., VR, gaming, financial trading), where even brief congestion can result in significant packet loss, increased latency, or service degradation. To mitigate these effects, network operators have traditionally resorted to overprovisioning switch buffers or operating links at lower utilization levels, both of which are costly and inefficient. As network speeds and application demands grow, there is a pressing need for more precise, real-time congestion management solutions that can identify and address the root causes of microbursts without excessive hardware investment.

Current approaches to congestion detection and management suffer from several critical limitations. Most network devices only provide coarse-grained queue statistics, typically aggregated over seconds or minutes, which are too slow to capture the fleeting nature of microbursts. Furthermore, these devices lack the ability to pinpoint which specific flows are responsible for sudden queue buildups, making targeted intervention impossible. Existing solutions often rely on indiscriminate packet dropping or require large, expensive buffers to absorb bursts, neither of which addresses the underlying problem of identifying and managing culprit flows. In transit networks, where end-to-end congestion control cannot be coordinated, these shortcomings are particularly problematic. As a result, operators are left with blunt tools that either waste resources or fail to prevent performance degradation during microburst events.

Technology

The described technology is a comprehensive microburst detection and congestion management system designed for network switches, known as ConQuest. It operates by analyzing packet arrival and departure times to detect queuing delays that signal microburst congestion—short, intense traffic surges that can overwhelm network buffers. The system uses a snapshot-based data structure with approximate counting (such as Count-Min Sketch) to efficiently estimate the size of each flow within the congested queue, organizing these estimates into rotating, fixed-length time windows. When a flow’s packets constitute a significant portion of the congested queue, it is flagged as a culprit, enabling immediate corrective actions like marking, dropping, rerouting, delaying, or load balancing its packets. ConQuest can be deployed both inline within programmable switches and off-path by monitoring legacy devices via network taps, ensuring compatibility across diverse network environments.

What differentiates this technology is its ability to provide near real-time, fine-grained visibility into the specific flows responsible for transient congestion, all while operating within the severe memory and processing constraints of modern programmable switch hardware. The innovative use of rotating, time-windowed snapshots allows for efficient memory management and implicit expiration of stale data, eliminating the need for per-packet state tracking. Its approximate counting approach delivers high accuracy (over 90% precision and recall) with minimal resource usage, making it practical for high-speed networks. Furthermore, ConQuest’s targeted congestion control actions—applied specifically to culprit flows—enable operators to reduce buffer requirements, increase link utilization, and maintain strict latency guarantees, all without costly hardware upgrades or indiscriminate packet dropping. This level of actionable insight and control over microburst events is unmatched by traditional network telemetry and congestion management solutions.

Benefits

• Enables near real-time detection and management of microburst congestion with sub-millisecond to millisecond granularity.
• Identifies specific culprit flows responsible for transient queue buildup, allowing targeted congestion control actions.
• Reduces the need for large, expensive buffer provisioning by efficiently managing congestion within existing hardware constraints.
• Supports deployment on programmable switches as well as non-intrusive off-path monitoring of legacy network devices.
• Implements memory-efficient snapshot-based approximate flow size tracking using Count-Min Sketch data structures.
• Improves network utilization by enabling higher link usage without increased packet loss or delay.
• Provides flexibility to apply multiple corrective actions such as marking, dropping, rerouting, delaying, and load balancing of packets.
• Facilitates enhanced network planning and capacity management through detailed congestion and flow size analytics.

Applications

• Real-time microburst congestion mitigation
• Targeted flow-based traffic shaping
• Latency-sensitive application protection
• Legacy network congestion monitoring
• Automated network capacity planning

Additional info

This system detects network microbursts by analyzing packet queuing delays. It identifies culprit traffic flows using a snapshot-based data structure, employing approximate counting techniques like Count-Min Sketch to estimate flow sizes within congested queues. Significant contributors are flagged for immediate corrective actions such as marking, dropping, or rerouting, enhancing network efficiency and reducing buffer requirements.