Supplement to InfiniBandTM

Architecture Specification
Volume 1 Release 1.2.1

Annex A17:
RoCEv2

September 2, 2014

Copyright 2010 by InfiniBandTM Trade Association.

All rights reserved.
All trademarks and brands are the property of their respective owners.
This document contains information proprietary to the InfiniBandTM Trade Association. Use or disclosure without
written permission by an officer of the InfiniBandTM Trade Association is prohibited.
InfiniBandTM Architecture Release 1.2.1 RoCEv2 (IP Routable RoCE) September 2, 2014
Volume 1 - General Specifications

1
Table 0 Revision History 2
3
Revision Date 4
1.0 Sept. 2, 2014 General Release
5
6
7
8
LEGAL DISCLAIMER This specification provided AS IS and without any 9
warranty of any kind, including, without limitation, 10
any express or implied warranty of non-infringement, 11
12
merchantability or fitness for a particular purpose.
13
14
In no event shall IBTA or any member of IBTA be liable 15
for any direct, indirect, special, exemplary, punitive, 16
or consequential damages, including, without limita- 17
tion, lost profits, even if advised of the possibility of 18
such damages. 19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

InfiniBandSM Trade Association

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

ANNEX A17: ROCEV2 (IP ROUTABLE ROCE) 1

2
3
4
5
A17.1 INTRODUCTION
6
This document is an annex to Volume 1 release 1.2.1 of the InfiniBand Ar- 7
chitecture, herein referred to as the base specification. This annex is Op- 8
tional Normative, meaning that implementation of the feature described by 9
this annex is Optional, but if present, the implementation must comply with 10
the compliance statements contained within this annex. This specification 11
follows the spirit of the RoCE Annex (Annex A16 to the base specification) 12
in defining a new InfiniBand protocol variant that uses an IP network layer
13
(with an IP header instead of InfiniBands GRH) thus allowing IP routing
14
of its packets.
15
16
A17.2 OVERVIEW 17
A17.2.1 THE INFINIBAND ARCHITECTURE 18
19
The InfiniBand Architecture offers a rich set of I/O services based on an 20
RDMA access method and message passing semantics. Included are a
21
variety of transport services, reliable and unreliable, connected and un-
22
connected, support for atomic operations, multicast and others.
23
24
InfiniBand defines a layered architecture that specifies the first four layers
25
of the OSI reference stack including the physical, link, network and trans-
port layers as well as an accompanying management framework. In addi- 26
tion, the IB specification defines a software interface and its 27
accompanying verbs which are designed to allow smooth access to the 28
services provided by the InfiniBand Architecture. 29
30
31
32
33
A17.2.2 RDMA OVER CONVERGED ETHERNET (ROCE)
34
RDMA over Converged Ethernet (RoCE) is an InfiniBand Trade Associa- 35
tion Standard designed to provide InfiniBand Transport Services on 36
Ethernet Networks4. RoCE preserves the InfiniBand Verbs Semantics to- 37
gether with its Transport and Network Protocols and replaces the Infini- 38
Band Link and Physical Layers with those of Ethernet. The network 39
management infrastructure for RoCE is also that of Ethernet. 40
4. http://www.infinibandta.org/content/pages.php?pg=about_us_RoCE 41
42

InfiniBandSM Trade Association Page 1 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Figure 1 InfiniBand and RoCE Protocol Stacks 19
20
A17.2.3 THE NEED FOR (IP) ROUTABLE RDMA 21
5
RoCE packets are regular Ethernet frames that carry an Ethertype 22
6 allocated by IEEE which indicates that the next header is a RoCE 23
value
GRH. 24
25
26
27
28
29
30
Figure 2 RoCE Packet Format
31
32
Since RoCE traffic doesn't carry an IP header, it can't be routed across the 33
boundaries of Ethernet L2 Subnets using regular IP routers. Under this 34
scheme, RoCE provides RDMA services for communication within an 35
Ethernet L2 domain. 36
37
38
39
5. Including VLANs and all other Ethernet header variations as defined by IEEE
802 40
6. 0x8915 41
42

InfiniBandSM Trade Association Page 2 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.2.4 ROCEV2 (IP ROUTABLE ROCE)

1
RoCEv2 is a straightforward extension of the RoCE protocol that involves 2
a simple modification of the RoCE packet format. Instead of the GRH, 3
RoCEv2 packets carry an IP header which allows traversal of IP L3 4
Routers and a UDP header that serves as a stateless encapsulation layer 5
for the RDMA Transport Protocol Packets over IP.
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Figure 3 RoCEv2andRoCEFrameFormatDifferences
23
24
25
RoCEv2 packets use a well-known UDP Destination Port (dport) value
that unambiguously distinguishes them in a stateless manner. 26
27
28
As an additional benefit, following common practices in UDP encapsu-
lated protocols, the UDP Source Port (sport) field of RoCEv2 packets 29
serves as an opaque flow identifier that can be used by the networking in- 30
frastructure for packet forwarding optimizations - see Section 17.9.4, 31
ECMP for RoCEv2, on page 21. 32
33
Since this approach exclusively affects the packet format on the wire, and 34
due to the fact that with RDMA semantics packets are generated and con- 35
sumed below the API, applications can operate over any form of RDMA 36
service (including RoCEv2) in a completely transparent way7 (see Figure 37
4).
38
39
40
7. WidespreadRDMAAPIsareIPbasedforallexistingRDMAtechnologies 41
42

InfiniBandSM Trade Association Page 3 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Figure 4 RoCEv2 Protocol Stack 22
23
A17.3 ROCEV2 PACKET FORMAT 24
25
The RoCEv2 Packet format is shown in Figure 5.
26
27
28
29
30
31
32
33
34
Figure 5 RoCEv2 Packet Format
35
36
37
A17.3.1 ETHERTYPES AND IP HEADER FIELDS
38
RoCEv2 supports both IPv4 and IPv6. The corresponding Ethertype 39
values as well as IPv4 and IPv6 header fields for RoCEv2 packets are de- 40
scribed in Section 17.3.1.1, RoCEv2 with IPv4, on page 5 and
41
Section 17.3.1.2, RoCEv2 with IPv6, on page 6 respectively.
42

InfiniBandSM Trade Association Page 4 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

CA17-1: RoCEv2 Ports shall support both RoCEv2 with IPv4 and
1
RoCEv2 with IPv6 packet formats.
2
CA17-2: RoCEv2 Packets shall conform to the format depicted in Figure 3
5 with individual fields set as mandated by either Section 17.3.1.1, 4
RoCEv2 with IPv4, on page 5 or Section 17.3.1.2, RoCEv2 with IPv6, 5
on page 6. 6
7
A17.3.1.1 ROCEV2 WITH IPV4
8
The Ethertype value for IPv4 as assigned by IEEE is 0x0800.
9
The format of the IPv4 header and its fields are specified by the IETF in 10
RFC791, RFC2474 and RFC3168. The sub-sections below define the 11
values for relevant fields in the IPv4 header of RoCEv2 packets. 12
13
A17.3.1.1.1 INTERNET HEADER LENGTH (IHL) 14
CA17-3: For RoCEv2 packets with IPv4, the IHL field shall be set to 5. 15
16
A17.3.1.1.2 DIFFERENTIATED SERVICES CODEPOINT (DSCP)
17
CA17-4: For RoCEv2 packets with IPv4, the DSCP field shall be set to the
value in the Traffic Class component of the RDMA Address Vector asso- 18
ciated with the packet. 19
20
A17.3.1.1.3 EXPLICIT CONGESTION NOTIFICATION (ECN) 21
RoCEv2 makes use of the ECN field in the IPv4 header for signaling of 22
congestion as defined by the IETF in RFC3168. See Section 17.9.3, 23
RoCEv2 Congestion Management, on page 20. 24
25
For HCAs that support RoCEv2 Congestion Management, the ECN field
in the IPv4 header of a RoCEv2 packet may be set to 01 or 10 to indi- 26
cate that the packet is subject to marking in the network to indicate con- 27
gestion. 28
29
CA17-5: For HCAs that dont support RoCEv2 Congestion Management, 30
the ECN field in the IPv4 header of a RoCEv2 packet shall be set to 00.
31
A17.3.1.1.4 TOTAL LENGTH 32
CA17-6: For RoCEv2 packets with IPv4, the Total Length field shall be set 33
to the length of the IPv4 packet in bytes including the IPv4 header and up 34
to and including the ICRC. 35
36
A17.3.1.1.5 FLAGS 37
CA17-7: For RoCEv2 packets with IPv4 the Flags field shall be set to 010 38
(dont fragment bit is set). 39
40
41
42

InfiniBandSM Trade Association Page 5 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.3.1.1.6 FRAGMENTOFFSET
1
CA17-8: For RoCEv2 packets with IPv4 the Fragment Offset field shall be 2
set to 0.
3
4
A17.3.1.1.7 TIME TO LIVE
5
CA17-9: For RoCEv2 packets with IPv4 the Time to Live field shall be set 6
to the value in the Hop Limit component of the RDMA Address Vector as-
7
sociated with the packet.
8
A17.3.1.1.8 PROTOCOL
9
10
CA17-10: For RoCEv2 packets with IPv4 the Protocol field shall be set to
11
0x11 (UDP).
12
A17.3.1.1.9 SOURCE AND DESTINATION IP ADDRESSES
13
14
CA17-11: The Source IP Address of RoCEv2 packets with IPv4 shall be
15
set to the IPv4 address encoded in the Port GID entry referenced by the
port and SGID index components of the Address Vector associated 16
with the packet. 17
18
CA17-12: The Destination IP Address of RoCEv2 packets with IPv4 shall 19
be set to the IPv4 address encoded in the DGID component of the Ad- 20
dress Vector associated with the packet. 21
22
A17.3.1.2 ROCEV2 WITH IPV6
23
The Ethertype value for IPv6 as assigned by IEEE is 0x86DD. 24
25
The format of the IPv6 header and its fields are specified by the IETF in
26
RFC2460, RFC2474 and RFC3168. The sub-sections below define the
values for relevant fields in the IPv6 header of RoCEv2 packets. 27
28
A17.3.1.2.1 DIFFERENTIATED SERVICES CODEPOINT (DSCP) 29
30
CA17-13: For RoCEv2 packets with IPv6, the DSCP field shall be set to
the value in the Traffic Class component of the Address Vector associated 31
with the packet. 32
33
A17.3.1.2.2 EXPLICIT CONGESTION NOTIFICATION (ECN) 34
RoCEv2 makes use of the ECN field in the IPv6 header for signaling of 35
congestion as defined by the IETF in RFC3168. See Section 17.9.3, 36
RoCEv2 Congestion Management, on page 20. 37
38
For HCAs that support RoCEv2 Congestion Management, the ECN field 39
in the IPv6 header of a RoCEv2 packet may be set to 01 or 10 to indi- 40
cate that the packet is subject to marking in the network to indicate con-
41
gestion.
42

InfiniBandSM Trade Association Page 6 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

CA17-14: For HCAs that dont support RoCEv2 Congestion Manage-

1
ment, the ECN field in the IPv6 header of a RoCEv2 packet shall be set
to 00. 2
3
A17.3.1.2.3 PAYLOAD LENGTH 4
CA17-15: For RoCEv2 packets with IPv6, the Payload Length field shall 5
be set to the length of the IPv6 packet payload starting from the first byte 6
after the IPv6 header (i.e. the BTH) up to and including the 4 bytes of the 7
ICRC 8
9
A17.3.1.2.4 NEXT HEADER
10
CA17-16: For RoCEv2 packets with IPv6 the Next Header field shall be 11
set to 0x11 (UDP).
12
A17.3.1.2.5 HOP LIMIT
13
14
CA17-17: For RoCEv2 packets with IPv6 the Hop Limit field shall be set
to the value in the Hop Limit component of the Address Vector associated 15
with the packet. 16
17
A17.3.1.2.6 SOURCEANDDESTINATIONIPADDRESSES 18
CA17-18: The Source IP Address of RoCEv2 packets with IPv6 shall be 19
set to the IPv6 address in the Port GID entry referenced by the port and 20
SGID index components of the Address Vector associated with the 21
packet. 22
23
CA17-19: The Destination IP Address of RoCEv2 packets with IPv6 shall
be set to the IPv6 address in the DGID component of the Address Vector 24
associated with the packet. 25
26
A17.3.2 UDP HEADER FIELDS 27
The UDP header format is defined by IETF in RFC768. The sub-sections 28
below define the values for relevant fields in the UDP header of RoCEv2 29
packets. 30
31
A17.3.2.1 SOURCE PORT 32
The Source Port field in the UDP header of a RoCEv2 packet may be used 33
by network devices as a component in the selection of a route among mul- 34
tiple possible alternative routes - see Section 17.9.4, ECMP for RoCEv2, 35
on page 21. For that reason, HCAs should use a fixed value across 36
packets where ordering matters between them (e.g. packets of a con-
37
nected QP).
38
A17.3.2.2 DESTINATION PORT 39
CA17-20: The Destination Port field in the UDP header of RoCEv2 40
41
packets shall be set to the value allocated by IANA8 for use with RoCEv2.
42

InfiniBandSM Trade Association Page 7 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.3.2.3 LENGTH 1
CA17-21: The Length field in the UDP header of RoCEv2 packets shall 2
be set to the number of bytes counting from the beginning of the UDP 3
header up to and including the 4 bytes of the ICRC.
4
5
A17.3.2.4 CHECKSUM
6
The Checksum field in the UDP header of RoCEv2 packets should be set 7
to 0.
8
9
A17.3.3 ICRC FOR ROCEV2 PACKETS
10
RoCEv2 implements a 32b end-to-end CRC (denoted ICRC) that covers 11
all invariant fields of the packet and offers protection beyond the coverage 12
of the Ethernet Frame Checksum (FCS) that is usually updated hop-by-
13
hop in the fabric.
14
CA17-22: The rules for generation/checking of the ICRC of RoCEv2 15
packets follow the ICRC calculation in RoCE and InfiniBand as defined in 16
Volume 1 of the InfiniBand Specification Section 7.8.1 subject to: 17
18
(a) The ICRC calculation starts with 64 bits of 1.9 19
20
(b) The ICRC calculation continues with the entire IP datagram starting
21
with the first byte of the IP header up until and including the last IB Pay-
load byte right before the ICRC field itself. 22
23
(c) The variant fields in the IP header are replaced with 1s for the purpose 24
of the ICRC calculation/check so that changes to these fields along the 25
way dont affect the calculated ICRC value. 26
27
For RoCEv2 over IPv4 the fields replaced with 1s for the purpose of ICRC
calculation are: 28
29
Time to Live 30
Header Checksum 31
32
Type of Service (DSCP and ECN).
33
For RoCEv2 over IPv6 the fields replaced with 1s for the purpose of ICRC 34
calculation are:
35
Traffic Class (DSCP and ECN) 36
37
Flow Label
38
8. Once allocated by IANA will be updated to include the actual value 39
9. ThisistomakeitequivalenttotheRoCE(v1)ICRCthatruns64bitsof1(dummy 40
LRH)priortotheGRHthroughtheICRCmachinefollowingthespiritoftheIBICRC
calculation(IBSpecVol.1Section7.8.1) 41
42

InfiniBandSM Trade Association Page 8 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

Hop Limit.
1
(d) UDP Checksum field is replaced with 1s for the purpose of the ICRC 2
calculation/check.
3
4
5
A17.3.4 ROCEV2 INBOUND PACKET VALIDATION 6
CA17-23: RoCEv2 packets shall undergo validation as mandated by the 7
Base Specification subject to the explicit modifications defined in 8
Section 17.4, InfiniBand Transport Protocol Spec Considerations, on 9
page 9. 10
11
In addition,
12
CA17-24: Received RoCEv2 packets, that dont conform to the rules set 13
in Section 17.3.1, Ethertypes and IP Header Fields, on page 4, 14
Section 17.3.2, UDP Header Fields, on page 7 and Section 17.3.3, 15
ICRC for RoCEv2 Packets, on page 8 shall be silently dropped. 16
17
18
19
A17.4 INFINIBAND TRANSPORT PROTOCOL SPEC CONSIDERATIONS 20
This section describes adaptations to elements of normative behavior de- 21
fined in the InfiniBand Transport Protocol Specification as they apply to 22
RoCEv2.
23
CA17-25: An HCA containing a RoCEv2 port which claims compliance to 24
this annex shall be compliant with the InfiniBand transport as defined in 25
Chapter 9 of the base specification, subject to the adaptations and excep- 26
tions explicitly called out in this section. 27
28
A17.4.1 ROCEV2 ADDRESSING 29
A17.4.1.1 L3 ADDRESSES 30
For simplicity in the interpretation of the IB Base Specification text, 31
RoCEv2 L3 Addresses are interchangeably referred to as GIDs. As GIDs 32
have the same format as IPv6 addresses, for RoCEv2 with IPv6, the cor-
33
responding IPv6 Source IP (SIP) and Destination IP (DIP) Addresses are
simply referred to as SGID and DGID. For RoCEv2 with IPv4, the corre- 34
sponding IPv4 Source IP (SIP) and Destination IP (DIP) Addresses are 35
encoded into the SGID and DGID respectively following common rules for 36
IPv4-mapped IPv6 addresses namely: GID =::ffff:<IPv4 Address>. 37
38
A17.4.1.2 L2 ADDRESSES 39
All references in the Base Specification to the LRH and its fields are Not 40
Applicable to RoCEv2 ports.
41
42

InfiniBandSM Trade Association Page 9 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.4.2 ADDRESS VECTOR

1
The InfiniBand Specification defines an Address Vector that is used to de- 2
note a remote destination and the path parameters selected to communi- 3
cate with it. (InfiniBand Specification Vol.1 Rev 1.2.1 Section 11.2.2 and 4
Section 11.2.4.2). 5
6
The Address Vector fields are manipulated via the Software Transport In-
7
terface and passed down to the Transport Layer for the generation of out-
going and validation of incoming RoCEv2 packets. 8
9
RoCEv2 Address Vectors include L2 and L3 address information as well 10
as other path components (e.g. QoS, Hop Limit, etc). The Address Vector 11
components retain their specified behavior with the exceptions explicitly 12
called out in this section:
13
14
As with RoCE, the Send Global Routing Header Flag in the Address
Vector is always set to TRUE for RoCEv2. 15
16
Source L3 Addresses (SGIDs) are not explicitly stored in the Address 17
Vector but obtained by reference. The Address Vector includes a SGID 18
Index that points to an entry in a Source GID Table that is maintained per 19
port as described in Section 17.4.3, Port GID Table, on page 11. The 20
GID type as obtained from the selected SGID entry determines the pro-
21
tocol for outbound packet generation - see Section 17.8, Interoperability
with RoCE Endnodes, on page 18. 22
23
As with RoCE, the mechanism for RoCEv2 L3 to L2 Address Resolution 24
is outside of the scope of this specification. It is assumed that implemen- 25
tations follow common practices and use existing OS services to perform 26
this task (e.g. use ARP and routing infrastructure in the host). 27
28
For RoCEv2, the DLID component in the Address Vector is generalized to
become the Destination L2 Address and may be used to carry an al- 29
ready resolved DMAC across the Software Transport Interface (Create 30
Address Handle verb). In this case, the implementation may use the pro- 31
vided Destination L2 Address as-is for the generation of packets associ- 32
ated with the Address Vector in question10. Alternatively, by setting the 33
Destination L2 Address component of the Address Vector to the value 34
0xFFFFFFFFFFFF, the L3 to L2 Address Resolution is effectively carried 35
out by the implementation of the corresponding verb (Create Address
36
Handle and Modify QP).
37
The Source Path Bits component of the Address Vector is not applicable 38
for RoCEv2 ports. The Source L2 Address for RoCEv2 packets is implied 39
40
10. ThisallowsL3toL2AddressResolutiontohappenbeforethegenerationofthe
AddressVectorasisthecasewiththecurrentRDMACMimplementationinLinux 41
42

InfiniBandSM Trade Association Page 10 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

by the selected SGID of the Address Vector and obtained by the imple-
1
mentation using common services of the underlying OS infrastructure.
2
The SL component in the Address Vector is used to determine the 3
Ethernet Priority of generated RoCEv2 packets. SL 0-7 are mapped di- 4
rectly to Priorities 0-7, respectively. SL 8-15 are reserved. 5
6
A17.4.3 PORT GID TABLE 7
Every RoCEv2 port maintains a port GID table that contains all L3 Ad- 8
dresses that have been configured to the port as described in section 9
10.2.2.1 of the InfiniBand Specification. 10
11
Addresses in the RoCEv2 Port GID Table can be of type IPv4, IPv6 or
12
IB GID11. A new GID type attribute is added to the Port GID Table En- 13
tries of RoCEv2 ports to denote the L3 Address type.
14
CA17-26: RoCEv2 Port GID Table entries shall have a GID type attribute 15
that denotes the L3 Address type among IPv4, IPv6 and IB GID. 16
17
Protocol selection for outbound packet generation is based on the GID 18
type of the selected GID Table entry as described in Section 17.8, In- 19
teroperability with RoCE Endnodes, on page 18.
20
The software stack is responsible for maintaining the GID table following 21
creation/removal of L3 addresses to the port. This is typically achieved 22
through interaction (e.g. subscription to callback/event services) with the 23
OS and its host administrative interfaces. 24
25
A17.4.4 GRH CHECKS 26
The Base Specification (InfiniBand Specification Vol.1 Rev 1.2.1 Section 27
9.6.1.2) defines the rules for checking of the GRH (L3 header) of received 28
InfiniBand packets. As RoCEv2 packets carry an IP header instead of the 29
GRH the following rules replace those mandated by the base specifica- 30
tion. 31
32
All RoCEv2 packets carry an IP header and hence C9-43.1.1 and C9-
43.1.2 in the Base Specification are not applicable for RoCEv2. 33
34
RoCEv2 packets have a Next Header / Protocol field set to 0x11 (UDP) 35
and hence C9-44 of the Base Specification is not applicable for RoCEv2. 36
37
A17.4.4.1 IP VERSION 38
Compliance statement C9-45 of the Base Specification is replaced by: 39
40
11. For interoperability with RoCE as described in Section 17.8, Interoperability
with RoCE Endnodes, on page 18 41
42

InfiniBandSM Trade Association Page 11 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

CA17-27: For RoCEv2 with IPv6, if the version number is anything other
1
than 6, the packet shall be silently dropped. For RoCEv2 with IPv4, if the
version number is anything other than 4, the packet shall be silently 2
dropped. 3
4
A17.4.4.2 ADDRESS VALIDATION RULES 5
The Base Specification mandates L3 Address validation rules for inbound 6
packets (InfiniBand Specification Vol. 1 Rev 1.2.1 Section 9.6.1.2.3). 7
These rules apply to RoCEv2 packets. For the purpose of these checks, 8
the Source and Destination GIDs of RoCEv2 packets with IPv6 are the 9
IPv6 SIP and DIP addresses respectively. For RoCEv2 with IPv4, the
10
SGID and DGID are respectively obtained from the IPv4 SIP and DIP ad-
dresses following the common practice used to map an IPv4 address into 11
an IPv6 one namely: GID =::ffff:<IPv4>. 12
13
In addition, the DGID check is amended to include verification of protocol 14
type as detailed in Section 17.8, Interoperability with RoCE Endnodes, 15
on page 18
16
17
A17.4.5 UNRELIABLE DATAGRAM (UD)
18
A17.4.5.1 UD COMPLETION QUEUE ENTRIES (CQES)
19
For UD, the Completion Queue Entry (CQE) includes remote address in- 20
formation (InfiniBand Specification Vol. 1 Rev 1.2.1 Section 11.4.2.1). For 21
RoCEv2, the remote address information comprises the source L2 Ad-
22
dress and a flag that indicates if the received frame is an IPv4, IPv6 or
RoCE packet. 23
24
A17.4.5.2 SCATTERING OF THE L3 HEADER IN UD 25
The first 40 bytes of user posted UD Receive Buffers are reserved for the 26
L3 header of the incoming packet (as per the InfiniBand Spec Section 27
11.4.1.2). In RoCEv2, this area is filled up with the IP header. IPv6 header 28
uses the entire 40 bytes. IPv4 headers use the 20 bytes in the second half 29
of the reserved 40 bytes area (i.e. offset 20 from the beginning of the re- 30
ceive buffer). In this case, the content of the first 20 bytes is undefined.
31
32
A17.4.6 IB RAW DATAGRAMS
33
The InfiniBand Architecture defines a Raw service which does not use the
34
InfiniBand transport (InfiniBand Specification Vol.1 Rev 1.2.1 Section
9.8.4). The Raw services as defined in the base specification are provided 35
by the InfiniBand link layer. Similarly to RoCE, since RoCEv2 does not use 36
the InfiniBand link layer, IB RAW datagrams, namely Raw Ethertype and 37
Raw IPv6, are not applicable for RoCEv2. 38
39
CA17-28: An implementation of an HCA claiming conformance to this 40
annex shall not support the concept of IB Raw Datagrams on a RoCEv2
41
port.
42

InfiniBandSM Trade Association Page 12 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

As follows from the above, all references to Raw Packets in the Base
1
Specification are not applicable to RoCEv2 ports.
2
A17.4.7 INFINIBAND PARTITIONING 3
4
Methods to populate the P_Key table associated with a RoCEv2 port are
outside the scope of this annex. Note that this annex relies on the partition 5
table being initialized at power on time with at least the default P_Key as 6
described in Chapter 10 (Software Transport Interface) of the base spec- 7
ification. The P_Key contained in the BTH is validated for inbound packets 8
as required by the packet header validation protocols defined in Chapter 9
9 of the base specification. 10
11
A17.4.8 INFINIBAND CONGESTION CONTROL
12
Congestion Management for RoCEv2 is specified in Section 17.9.3, 13
RoCEv2 Congestion Management, on page 20. InfiniBand Congestion 14
Control as defined in Annex A10 of the base specification is not applicable
15
to RoCEv2 ports. Thus, a CA claiming compliance to Annex A10 for its In-
finiBand ports is not required to support any of the port attributes, counters 16
or controls required by Annex A10 for its RoCEv2 ports. 17
18
CA17-29: The B (BECN) and F (FECN) bits in the BTH devoted to con- 19
gestion control as defined in Annex A10 of the base specification are un- 20
used and shall be ignored by a RoCEv2 port.
21
22
A17.4.9 INFINIBAND QOS
23
QoS Management as defined in Annex A13 of the base specification is 24
based on InfiniBand Link Layer capabilities that are not applicable to
25
RoCEv2 ports. Thus, a CA claiming compliance to Annex A13 for its In-
finiBand ports is not required to support any of the port attributes counters 26
or controls associated with its RoCEv2 ports. 27
28
29
30
A17.5 INFINIBAND VERBS CONSIDERATIONS 31
The following sections specify modifications to InfiniBand verbs required 32
for RoCEv2 ports. 33
34
CA17-30: RoCEv2 HCAs shall adopt the modifications to verbs described 35
in this section. 36
37
A17.5.1 QUERY HCA
38
The Port Attribute List Output Modifier for this verb when associated with 39
a RoCEv2 port is changed as follows: 40
41
The Base LID & LMC fields are unused and shall be ignored.
42

InfiniBandSM Trade Association Page 13 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

The maximum number of virtual lanes supported by this HCA is un-

1
used and shall be ignored.
2
The Optional InitTypeReply value is unused and shall be ignored.
3
The Subnet Manager address information for each RoCE port of this 4
HCA is unused and shall be ignored. 5
The CapabilityMask bits IsSM, IsSMDisabled, IsSNMPTunnelingSup- 6
ported, IsClientReregistrationSupported are unused and shall be ig- 7
nored.
8
A new value (RoCEv2) shall be added for the port-type attribute of 9
the Port Attributes list to denote that the port is of RoCEv2 type 10
A new RoCE Supported capability bit shall be added to the Port At- 11
tributes list. This capability bit applies exclusively to ports of the new 12
RoCEv2 type. When set, it denotes that the port is capable of oper-
13
ating simultaneously in RoCEv2 and RoCE modes. See
Section 17.8, Interoperability with RoCE Endnodes, on page 18. 14
15
A17.5.2 MODIFY HCA 16
The Port Attribute List Input Modifier for this verb when associated with a 17
RoCEv2 port shall be changed as follows: 18
19
The CapabilityMask bits IsSM, IsSNMPTunnelingSupported are un- 20
used and are reserved. 21
22
A17.5.3 CREATE/MODIFY/QUERY ADDRESS HANDLE
23
The Address Vector component shall be modified in accordance with 24
Section 17.4.2, Address Vector, on page 10. 25
26
For CREATE/MODIFY ADDRESS HANDLE, the Invalid Address Vector
error may be returned due to the use of a reserved SL value (SL 8-15 are 27
reserved) when the QP is associated with a RoCEv2 port. 28
29
A17.5.4 MODIFY/QUERY QUEUE PAIR / MODIFY/QUERY XRC TARGET QP 30
31
The Address Vector and Alternate Path components shall be modified in 32
accordance with Section 17.4.2, Address Vector, on page 10. 33
34
For MODIFY QUEUE PAIR / MODIFY XRC TARGET QP, the Invalid Ad-
dress Vector error may be returned due to the use of a reserved SL value 35
(SL 8-15 are reserved) when this QP is associated with a RoCEv2 port. 36
37
A17.5.5 MODIFY/QUERY EE CONTEXT 38
39
The Primary Address Vector and Alternate Path components shall be 40
modified in accordance with Section 17.4.2, Address Vector, on
41
page 10.
42

InfiniBandSM Trade Association Page 14 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

For MODIFY EE CONTEXT, the Invalid Address Vector error may be re-
1
turned due to the use of a reserved SL value (SL 8-15 are reserved) when
this EE CONTEXT is associated with a RoCEv2 port. 2
3
A17.5.6 ATTACH/DETACH QP TO/FROM MULTICAST GROUP 4
If the QP is associated with a RoCEv2 port, the Input Modifiers for this 5
verb shall be changed as follows: 6
7
The Multicast group MLID is unused and shall be ignored. 8
The Output Modifiers for this verb when associated with a RoCE port shall 9
be changed as follows: 10
11
Invalid multicast MLID is removed as a valid Verb Result
12
A17.5.7 POLL FOR COMPLETION 13
The output modifier of the Poll for Completion shall be changed as follows: 14
15
If the port is a RoCEv2 port, the remote port address and QP infor- 16
mation returned for datagram services (shown in Table 97 of the base
17
specification) shall be modified in accordance with Section 17.4.5.1,
UD Completion Queue Entries (CQEs), on page 12 18
19
A17.5.8 GET SPECIAL QP
20
Since there is no QP0 associated with a RoCEv2 port, and since a 21
RoCEv2 port does not support either Raw datagram type, for a RoCEv2
22
port, Get Special QP only applies to the GSI QP (QP1).
23
Thus compliance statement C11-13 in Section 11.2.5 of the base specifi- 24
cation does not apply to a RoCEv2 port with respect to QP0, and compli- 25
ance statement o11-1 does not apply. An attempt to call GET SPECIAL 26
QP on a RoCEv2 port for a QP other than QP1 shall return an Invalid 27
Special QP Type error. 28
29
A17.5.9 POST SEND REQUEST
30
The Post Send Request verb shall be modified to eliminate Raw as one
31
of the possible service types. The Operation Type Matrix under the
POST SEND REQUEST verb in the base document is modified to effec- 32
tively eliminate the row of the table governing the Raw service type. 33
34
A17.5.10 UNAFFILIATED ASYNCHRONOUS EVENTS 35
CA17-31: A RoCEv2 port shall not support Client Reregistration. 36
37
CA17-32: A RoCEv2 port shall not support the optional Port Change 38
Event 39
40
41
42

InfiniBandSM Trade Association Page 15 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.6 INFINIBAND MANAGEMENT CONSIDERATIONS

1
The following management classes specified in the InfiniBand Architec- 2
ture as well as their associated normative statements are not applicable
3
to RoCEv2 ports: Subnet Management, Subnet Administration, Perfor-
mance Management, Device Management, Baseboard Management, 4
SNMP Tunneling, Vendor specific, Application specific classes, Conges- 5
tion Control, Boot Management and BIS. Instead, RoCEv2 ports are at- 6
tached to IP subnets that are managed using common Ethernet/IP 7
management practices and standards that are out of scope of this speci- 8
fication. 9
10
In the same spirit, the special Queue Pair, QP0 that is defined solely for
communication between subnet manager(s) and subnet management 11
agents is not relevant for RoCEv2 ports. 12
13
CA17-33: A packet arriving at a RoCEv2 port containing a BTH with the 14
destination QP field set to QP0 shall be silently dropped. 15
16
It is expected that there is no InfiniBand management communication be-
tween an Ethernet and an InfiniBand management domain. Therefore, 17
any InfiniBand method/attribute combination that refers to a RoCE port 18
may return error code 7 in the Code for invalid field of the MAD Common 19
Status field (One or more fields in the attribute or attribute modifier con- 20
tains an invalid value. Invalid values include reserved values and values 21
that exceed architecturally defined limits). 22
23
A17.6.1 COMMUNICATION MANAGEMENT
24
RoCEv2 utilizes the InfiniBand Architecture Communication Management
25
protocol as defined in Chapter 12 of the base specification. Modifications
to the specific MADs required to eliminate references to local addresses 26
are contained in this section. 27
28
Communication Management packets for RoCEv2 connections are reg- 29
ular RoCEv2 packets of the same type (IPv4 vs. IPv6) as the intended 30
connection. 31
32
Communication Management packets for RoCE connections involving
RoCEv2 ports that are also capable of generating and processing RoCE 33
packets (Section 17.5.1, QUERY HCA, on page 13), are RoCE packets. 34
35
A17.6.1.1 REQ MESSAGE 36
CA17-34: When a connection is being established between RoCEv2 37
ports, the Primary Local Port LID, Primary Remote Port LID, Alternate 38
Local Port LID and Alternate Remote Port LID fields of the REQ message 39
are Reserved and shall be ignored on receipt. The value of these fields
40
shall not be checked or validated by a recipient of a REQ message.
41
42

InfiniBandSM Trade Association Page 16 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.6.1.2 REJ MESSAGE

1
CA17-35: When a connection is being established between RoCEv2
2
ports, the following reject reason codes shall not be sent as part of a REJ
message: 3
4
code 13: Primary Remote Port LID rejected 5
code 19: Alternate Remote Port LID rejected 6
7
A17.6.1.3 LAP MESSAGE
8
CA17-36: When alternate paths are being established between RoCEv2
9
ports, the Alternate Local Port LID and Alternate Remote Port LID fields
are Reserved and shall be ignored on receipt. The value of these fields 10
shall not be checked or validated by a recipient of a LAP message. 11
12
A17.6.1.4 APR MESSAGE 13
CA17-37: When alternate paths are being established between RoCEv2 14
ports, the following APR status code shall not be sent as part of an APR 15
message: 16
17
code 7: Proposed Alternate Remote Port LID rejected.
18
A17.6.1.5 SAP MESSAGE
19
CA17-38: The value of the Alternate Local Port LID is reserved and shall 20
be ignored on receipt.
21
A17.7 CHANNEL ADAPTERS 22
23
The base specification defines specific hardware entities such as channel
24
adapters and switches which implement all layers of the InfiniBand Archi-
tecture including the InfiniBand-defined physical and link layers. Chapter 25
17 of the base specification sets forth specific requirements for a channel 26
adapter. Most of the compliance statements contained in Chapter 17 of 27
the base specification apply to a CA which supports one or more RoCEv2 28
ports. This section describes the exceptions. 29
30
A17.7.1 LOADING THE P_KEY TABLE
31
Compliance statement C17-7 describes requirements for setting the 32
P_Key table based on an assumption that the P_Key table is set directly
33
by a Subnet Manager. However, RoCEv2 ports do not support InfiniBand
Subnet Management. Therefore, compliance statement C17-7 does not 34
apply to RoCEv2 ports. 35
36
Methods for setting the P_Key table associated with a RoCEv2 port are 37
not defined in this specification, except for the requirements for a default 38
P_Key described elsewhere in this annex.
39
40
41
42

InfiniBandSM Trade Association Page 17 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.7.2 LOCALLY ROUTED PACKETS

1
Compliance statement C17-9 refers to locally routed packets which dont 2
exist with RoCEv2 Thus, C17-9 is not applicable to RoCEv2 ports.
3
A17.7.3 BACKPRESSURE AND DEADLOCK PREVENTION 4
5
Compliance statements C17-19 and C17-20 place specific limitations on
6
a CAs ability to apply backpressure. For a CA claiming compliance to this
annex, these requirements do not apply to RoCEv2 ports. 7
8
A17.7.4 INBOUND PACKET CHECKING 9
Compliance statement C17-21 is replaced as follows: 10
11
CA17-39: For RoCEv2 ports, the CA shall check for transport layer errors 12
in all incoming packets 13
14
A17.7.5 SUPPORT FOR QP0
15
Compliance statement C17-24 is not applicable to RoCEv2 ports. 16
17
A17.8 INTEROPERABILITY WITH ROCE ENDNODES
18
RoCEv2 and RoCE Ports can interoperate with each other. For this pur- 19
pose, RoCEv2 ports are optionally capable of generating and processing
20
RoCE packets. (Section 17.5.1, QUERY HCA, on page 13)
21
22
Protocol selection (RoCE vs RoCEv2) is controlled through the GID type
23
attribute in the corresponding entry of the RoCEv2 Port GID table (See
Section 17.4.3, Port GID Table, on page 11). 24
25
26
CA17-40: Connected QPs on RoCEv2 ports that support RoCE shall op-
erate in the mode (RoCE or RoCEv2) as determined by the GID type at- 27
tribute in the port GID Table entry configured for the QP (MODIFY_QP). 28
29
CA17-41: For inbound packets subject to DGID checks, as mandated by 30
the transport protocol (InfiniBand Specification Vol.1 Rev 1.2.1 Section 31
9.6.1.2.3), if the GID type of the Port GID table entry against which the 32
DGID check is performed doesnt match the protocol of the received 33
packet the check shall be considered as failed. 34
35
CA17-42: UD QPs on RoCEv2 ports that support RoCE shall generate 36
packets in the mode (RoCE or RoCEv2) as determined by the GID type 37
attribute in the port GID Table entry referenced by the Address Vector (AV)
38
in the WQE.
39
40
CA17-43: UD QPs on RoCEv2 ports that support RoCE shall accept as
41
valid both RoCE and RoCEv2 packets.
42

InfiniBandSM Trade Association Page 18 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.9 ROCEV2 NETWORK CONSIDERATIONS

1
A17.9.1 LOSSLESS NETWORK 2
As with RoCE, the underlying networks for RoCEv2 should be configured 3
as lossless. In this context, lossless doesnt mean that packets are abso- 4
lutely never lost. Moreover, the Transport Protocol in RoCEv2 includes an 5
end-to-end reliable delivery mechanism with built-in packet retransmis-
6
sion logic. This logic is typically implemented in HW and is triggered to re-
cover from lost packets without the need for intervention by the software 7
stack. The requirement for an underlying lossless network is aimed at pre- 8
venting RoCEv2 packet drops as a result of contention in the fabric. 9
10
In Data Center Ethernet networks, this kind of lossless behavior is typi- 11
cally achieved through the use of Link-Layer Flow-Control. IEEE
12
802.1Qbb specifies per-priority link-layer flow-control for Ethernet Net-
works and is ideally suited for RoCEv2 traffic. Other methods to attain 13
lossless network behavior besides 802.1Qbb may also be used. 14
15
In order to ensure end-to-end lossless behavior for RoCEv2 packets that 16
traverse multiple L2 subnets, the intervening L3 routers should be config- 17
ured to generate an L2 priority for RoCEv2 packets that is set to a value 18
configured for Link Layer Flow Control (802.1Qbb) enabled on the
19
subnet where the packet is about to be injected. This is normally obtained
through regular configuration mechanisms of the L3 Routers that control 20
the mapping of the Class of Service for traversing packets. 21
22
This lossless recommendation does not impose a constraint on non- 23
RoCEv2 traffic that could coexist with RoCEv2 on a converged network 24
scenario. Such traffic should be configured to use a distinct set of priorities
25
where Link Level Flow Control could be disabled. Network devices (L2
switches and L3 routers) typically allocate separate queues and buffer re- 26
sources to traffic on distinct priorities. This creates an effective isolation 27
that decouples the RoCEv2 lossless traffic from the other flows (e.g. TCP) 28
that usually operate in lossy mode. 29
30
A17.9.2 ROCEV2 QOS 31
RoCEv2 traffic can take advantage of IP/Ethernet L3/L2 QoS. Given some 32
of the most prevalent use cases for RDMA technology (e.g. low latency, 33
high bandwidth), the use of QoS becomes particularly relevant in a con- 34
verged environment where RoCEv2 traffic shares the underlying network 35
with other TCP/UDP packets. In this regard, RoCEv2 traffic is no different 36
than other IP flows: QoS is achieved through proper configuration of rele- 37
vant mechanisms in the fabric such as the Enhanced Transmission Selec-
38
tion (ETS) defined in 802.1Qaz. Packet/flow identification follows
39
standard practices of IP/Ethernet networks (i.e. DSCP/802.1Q) and is
40
controlled through the Traffic Class parameter in the Address Vector - see
41
Section 17.4.2, Address Vector, on page 10.
42

InfiniBandSM Trade Association Page 19 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

A17.9.3 ROCEV2 CONGESTION MANAGEMENT

1
RoCEv2 Congestion Management (RCM) provides the capability to avoid 2
congestion hot spots and optimize the throughput of the fabric. With RCM, 3
incipient congestion in the fabric is reported back to the sources of traffic 4
that in turn react by throttling down their injection rates thus preventing the 5
negative effects of fabric buffer saturation and increased queuing delays.
6
Congestion Management is also relevant for co-existing TCP/UDP/IP 7
traffic. However, assuming the intended use of a distinct set of priorities 8
for RoCEv2 and the other traffic, each set of priorities having a bandwidth 9
allocation12, the effects of congestion and the reaction (or lack of it) 10
shouldnt impact one another. 11
12
For signaling of congestion, RCM relies on the mechanism defined in 13
RFC3168 (ECN). Upon congestion that involves RoCEv2 traffic, network 14
devices mark the packets using the ECN field in the IP header
15
Section 17.3.1.1.3, Explicit Congestion Notification (ECN), on page 5
and Section 17.3.1.2.2, Explicit Congestion Notification (ECN), on 16
page 6. This congestion indication is interpreted by destination end-nodes 17
in the spirit of the FECN congestion indication flag of the Base Transport 18
Header (BTH). In other words, as ECN marked packets arrive to their in- 19
tended destination, the congestion notification is reflected back to the 20
source which in turn reacts by rate limiting the packet injection for the QP
21
in question.
22
RCM is optional normative behavior. RoCEv2 HCAs that implement RCM 23
shall follow the rules specified in this section: 24
25
CA17-44: If RoCEv2 Congestion Management is supported, upon re- 26
ceiving a valid RoCEv2 packet with a value of 11 in its IP.ECN field the 27
HCA shall generate a RoCEv2 CNP formatted as shown in Figure 6 on
28
page 21 directed to the source of the received packet. The HCA may
choose to send a single CNP for multiple such ECN marked packets on a 29
given QP. 30
31
CA17-45: If RoCEv2 Congestion Management is supported, upon recep- 32
tion of a RoCEv2 CNP the HCA shall reduce the rate of injection for the 33
QP indicated in the RoCEv2 CNP. The amount of rate change is deter- 34
mined by a rate reduction parameter, whose configuration is outside the
35
scope of this specification.
36
CA17-46: If RoCEv2 Congestion Management is supported, the HCA 37
should increase the injection rate on a QP when a configurable amount of 38
elapsed time and/or a configurable number of bytes have been trans- 39
mitted on that QP since the reception of the most recent RoCEv2 CNP for 40
12. IEEE 802.1Qaz ETS or similar mechanisms 41
42

InfiniBandSM Trade Association Page 20 Proprietary and Confidential

InfiniBandTM Architecture RoCEv2 (IP Routable RoCE) September 2, 2014
VOLUME 1 - GENERAL SPECIFICATIONS

that QP. The configuration for the amount of time, number of bytes trans-
1
mitted and rate of increase are outside the scope of this specification.
2
The RoCEv2 Congestion Notification Packet format is shown in Figure 6. 3
4
5
6
MAC Header 7
IPv4/IPv6 Header 8
UDP Header 9
BTH 10
DestQP set to QPN for which the RoCEv2 CNP is generated 11
12
Opcode set to b10000001 13
PSN set to 0 14
15
SE set to 0
16
M set to 0 17
18
P_Key set to the same value as in the BTH of the ECN packet marked
19
20
(16 bytes) - Reserved. MUST be set to 0 by sender. Ignored by receiver 21
ICRC 22
FCS 23
24
Figure 6 RoCEv2 CNP Format
25
26
27
A17.9.4 ECMP FOR ROCEV2
28
Data Center IP networks usually implement path selection mechanisms 29
for load balancing and improved utilization of the fabric topology. Equal 30
Cost Multiple Paths (ECMP) is one prevalent method to achieve this goal. 31
For a given packet, L3 Routers select among the possible different paths 32
using a hash on some of the packet fields. The choice is aimed at allowing
33
multiple paths while preserving the ordering requirements of individual
flows. 34
35
RoCEv2 packets carry an opaque flow identifier in their UDP Source Port 36
field Section 17.3.2.1, Source Port, on page 7 which is part of said hash 37
for UDP packets. Consequently, RoCEv2 endnodes set this field so that 38
packets in a sequence that has ordering constraints (e.g. packets from a 39
connected QP) will all carry a constant value. For packets that have no or- 40
dering constraints with respect to each other, the UDP Source Port field
41
can be set to different values.
42

InfiniBandSM Trade Association Page 21 Proprietary and Confidential