Benchmarking Methodology for Data Center Bridging Devices

This document is intended to provide a methodology for benchmarking Data Center Ethernet (DCB) switching devices that support Priority-based Flow Control (PFC) as defined in . It extends the methodologies already defined in and . This RFC primarily deals with devices that use priority-based flow control (PFC), as defined in , to actively manage the transmission rate of network endpoints in order to minimize forwarding delay and frame loss.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

The lock-step traffic pattern, as described in section 5.1.3 of , is specifically NOT required for DCB testing for two reasons: Such patterns are unrealistic for high speed Ethernet devices due to the transmission clock variance allowed by the IEEE 802.3 Ethernet specification. Flow control mechanisms would quickly break such patterns when activated.

As the terminology used by is specific to IP layer testing, a number of existing terms require clarification when used in the DCB benchmarking context. Additionally, a number of new terms are also presented to clarify concepts not clearly defined within the scope of . Classification: As stated in , the selection of packets according to defined rules. In the context of DCB benchmarking, the Classification criterion is the value of the 802.1p priority code point field in the 802.1Q VLAN header of an Ethernet frame. Classification Group: A collection of traffic streams that belong to a single Classification. A Conformance Vector MAY be associated with a Classification Group. Classification Profile: The set of all Classification Groups involved in a benchmarking test. Conformance Vector: A set of measurable stream result bounds, e.g. latency, jitter, sequencing, etc., that specify whether a frame is Conformant or Nonconformant. Congestion Management: In the context of DCB benchmarking, Congestion Management occurs when the DUT/SUT transmits priority-based flow control (PFC) Pause frames. Forwarding Congestion: In the context of DCB benchmarking, Forwarding Congestion is extended to include PFC pause frame transmissions from the DUT. Intended Load: In this document, the Intended Load refers to the summation of the Intended Vectors for all Classification Groups. Offered Load: In this document, the Offered Load refers to the summation of the Offered Vectors for all Classification Groups. Queue Congestion: Queue congestion occurs when a DUT/SUT uses Congestion Management on a discrete set of code points. The congestion managed code points correspond to the congested queue in the DUT/SUT. Queueput: The maximum Offered Load than can be transmitted into a DUT/SUT such that every transmitted frame matches a specific Classification rule, the DUT/SUT does NOT use priority-based flow control mechanisms to manage the ingress traffic rate, and all ingress frames are forwarded to the correct egress port. A DUT may have a different Queueput rate for each configured Classification.

This document extends the general test setup described in section 3 of and section 6 of to the benchmarking of Data Center Ethernet switching devices. and describe benchmarking methodologies for networking devices that intentionally drop frames at their performance limits. In DCB networks, the DUT will transmit PFC Pause frames as a Congestion Management method to throttle network endpoints, thus minimizing the probability of frame loss in the network.

Since DCB devices are expected to support multiple traffic Classifications, it is RECOMMENDED to benchmark DCB devices with multiple Classification Groups.

When generating multiple traffic classifications from the same test port, traffic from each class MUST be interleaved in a round-robin fashion.

The recommended trial duration is 300 seconds, however other durations MAY be used. Additionally, a running trial MAY be aborted once the test tool can tell that the currently running trial has failed. Two examples would be if a frame does not conform to the Conformance Vector of that frame's Classification Group, or if the test tool detects loss on a lossless queue.

This testing document does not mandate the use of any particular frame format for testing. Any frame that can be legally forwarded by the DUT/SUT MAY be used provided that the test instrument can make the following distinctions for each frame: The test tool MUST be able to distinguish test frames from non-test frames. The test tool MUST be able to determine whether each test frame is forwarded to the correct egress port. The test tool MUST be able to determine whether each received frame conforms or does not conform to the Conformance Vector of the frame's Classification Group.

Packet Conformance MUST be determined for each and every test frame. The method specified for measuring Latency in , e.g. measuring the latency of a single test frame in a traffic flow, is unsuitable for DCB benchmarking.

Multiple methods exist for measuring the time it takes a test frame to be forwarded by a DUT. However, both of the methods discussed in are unsuitable for testing DCB devices, as many DCB devices alternate between both "store and forwarding" and "bit forwarding" behavior depending upon their queue congestion state. Hence, the only recommended method for measuring the time it takes a DUT to forward a test frame is "Forwarding Delay" as described in .

The recommended frame sizes for Ethernet testing are 64, 128, 256, 512, 1024, 1280, 1518, 4096, 8192, and 9216 as per . Note that this frame size includes the Ethernet CRC. This frame size MUST include a VLAN header, since requires the use of VLAN priority code points to distinguish different Classification Groups.

FCoE test traffic introduces a number of frame size constraints that make the default frame sizes specified in unusable: FCoE frames contain an encapsulated Fibre Channel frame. Due to the method of encapsulation used, all FCoE frames MUST be a multiple of 4 bytes. See . Test tools may need to include a test payload in addition to the encapsulated Fibre Channel frame in order to meet the requirements specified in . The maximum supported frame size for FCoE is 2176 bytes. Due to these constraints, the recommended frame sizes for FCoE testing are 128, 256, 512, 1024, 1280, 1520, 2176, and the smallest FCoE frame size supported by the test tool. This frame size MUST include a VLAN header, since requires the use of VLAN priority code points to distinguish different Classification Groups.

To determine the Queueput for one or more Traffic Classifications for a DUT using priority-based flow control.

The following parameters MUST be defined. Each variable is configured with the following considerations. Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See x for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector SHOULD specify the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector is optional, but must be defined if used. Priority-based flow control - PFC mechanisms MUST be enabled. Background Traffic - Background traffic MAY be present.

A search algorithm is used to determine the Queueput for each Classification Group. If Queue Congestion is detected for a Classification Group during a trial, then the Intended Vector for the Classification Group MUST be reduced for the subsequent trial. If a Conformance Vector is specified for the test and nonconformant frames are received during a trial, then the Intended Vector SHOULD be reduced for the subsequent trial. The algorithm MUST adjust the Intended Vector for each Classification Group. The search algorithms for each Classification Group MAY be run in parallel. The test continues until all Classification Groups in the test have converged on a discrete Queueput value.

The Queueput for each Classification SHOULD be reported in either frames or bits per second. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported. The number of PFC pause frames for each code point in the Codepoint Set MUST be reported.

TBD

To determine the maximum forwarding rate of one or more PFC queues on a PFC capable DUT.

Maximum Forwarding Rate is conceptually similar to the measurement in but works on a per-Classification basis in a DCB context. The following parameters MUST be defined. Each variable is configured with the following considerations. Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MAY be enabled or disabled. Background Traffic - Background traffic MAY be present.

The tester should iterate across all configured permutations of frame size, burst size, and Intended Vector for all Classification Groups.

The forwarding rate of each Classification Group SHOULD be reported as the number of test frames per second or bits per second the DUT correctly forwards to the proper egress port. The maximum forwarding rate for each Classification Group MUST be reported as the highest recorded forwarding rate from the set of all iterations. Both the Intended and Offered Vector of each Classification Group MUST be reported. The number of PFC Pause frames received for each Classification MUST be reported. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported.

TBD

To determine the delta between the maximum forwarding rate of a DUT and the point where the DUT ceases to use PFC to manage priority queues.

The following parameters MUST be defined. Each variable is configured with the following considerations. Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MUST be enabled. Backoff method - The recommended backoff method is to reduce the aggregate traffic load by a fixed amount while still maintaining a fixed load ratio between all Classification Groups.

The initial trial SHOULD begin with an Intended Load equal or greater than the Maximum Forwarding Rate of the DUT/SUT. For each subsequent trial, the aggregate load is reduced until the DUT is observed to complete a trial without activating any Congestion Management methods.

The Intended and Offered Vector for each Classification Group MUST be reported. The number of PFC Pause frames received for each Classification Group MUST be reported. If a Conformance Vector is specified for a Classification Group, the percentage of conformant frames SHOULD be reported.

TBD

To determine the maximum burst size a DUT handle on one or more PFC queues.

The following parameters MUST be defined. Each variable is configured with the following considerations Each Classification Group MUST be listed. For each classification group, the following parameters MUST be specified: Codepoint - For DCB tests, the codepoint is the VLAN priority. Frame Size - The frame size includes both the CRC and VLAN header. See for recommended frame sizes. Burst Size - The burst size specifies the number of frames transmitted with the minimum legal IFG before pausing. A burst size of 1 specifies a constant load. Intended Vector - The intended vector includes the intended rate of test traffic specified as a percentage of port load. Conformance Vector - The conformance vector includes all QoS bounds required for this traffic group. Priority-based flow control - priority-based flow control mechanisms MAY be enabled or disabled. The sum of all Intended Vectors on a transmitting port SHOULD equal the Maximum Offered Load (MOL) of that port.

A search algorithm is used to determine the maximum duration in seconds for which the configured Classification Profile can be forwarded by the DUT without active Congestion Management. If Congestion Management is detected during an iteration, then the duration MUST be reduced for the next iteration.

TBD

A broken PFC implementation could lead to hypothetical denial-of-service attacks where a DUT ignores PFC pause frames and incorrectly forwards traffic as a result, or conversely throttles traffic even though no congestion condition exists. However, the test networks described in Benchmarking Methodology Working Group documents generally SHOULD NOT be reachable by anyone other than the tester(s). The security implications of such incorrect PFC handling are beyond the scope of this performance-measurement document, which is intended for testing purposes in controlled lab conditions.

Testers SHOULD use network addresses assigned by IANA for the purpose of testing networks.