EtherChannel considerations

By stretch | Monday, January 18, 2010 at 4:04 a.m. UTC

EtherChannel is Cisco's term for bundling two or more physical Ethernet links for the purposes of aggregating available bandwidth and, to a lesser extent, providing a measure of physical redundancy. Under normal conditions, all but one redundant physical link between two switches will be disabled by STP at one end.

With EtherChannel configured, multiple links are grouped into a port-channel, which is assigned its own configurable virtual interface. The bundle is treated as a single link.

EtherChannel Negotiation
An EtherChannel can be established using one of three mechanisms: PAgP - Cisco's proprietary negotiation protocol LACP (IEEE 802.3ad) - Standards-based negotiation protocol Static Persistence ("On") - No negotiation protocol is used

Any of these three mechanisms will suffice for most scenarios, however the choice does deserve some consideration. PAgP, while perfectly able, should probably be disqualified as a legacy proprietary protocol unless you have a specific need for it (such as ancient hardware). That leaves LACP and "on", both of which have a specific benefit. LACP helps protect against switching loops caused by misconfiguration; when enabled, an EtherChannel will only be formed after successful negotiation between its two ends. However, this negotiation introduces an overhead and delay in initialization. Statically configuring an EtherChannel ("on") imposes no delay yet can cause serious problems if not properly configured at both ends. To configure an EtherChannel using LACP negotiation, each side must be set to either active or passive; only interfaces configured in active mode will attempt to negotiate an EtherChannel. Passive interfaces merely respond to LACP requests. PAgP behaves the same, but its two modes are refered to as desirable and auto.

Only a single line is needed to configure a group of ports as an EtherChannel: S1(config)# interface range f0/13 -15 S1(config-if-range)# channel-group 1 mode ? active Enable LACP unconditionally auto Enable PAgP only if a PAgP device is detected

http://packetlife.net/blog/2010/jan/18/etherchannel-considerations/

Page 1

desirable on passive

Enable PAgP unconditionally Enable Etherchannel only Enable LACP only if a LACP device is detected

S1(config-if-range)# channel-group 1 mode active Creating a port-channel interface Port-channel 1 As noted, a virtual port-channel interface Port-channel1 has been created to represent the logical link. Switchport configurations applied to this interface are replicated to the physical member interfaces. We can inspect the health of the EtherChannel with the show etherchannel summary command: S1# show etherchannel summary Flags: D - down P - bundled in port-channel I - stand-alone s - suspended H - Hot-standby (LACP only) R - Layer3 S - Layer2 U - in use f - failed to allocate aggregator M - not in u w d use, minimum links not met unsuitable for bundling waiting to be aggregated default port

Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------1 Po1(SD) LACP Fa0/13(D) Fa0/14(D) Fa0/15(D) The opposite side of the LACP EtherChannel will typically be configured as passive, however it can be active as well. S2(config-if-range)# channel-group 1 mode passive Creating a port-channel interface Port-channel 1 When the member ports on both sides of the EtherChannel are enabled, the port-channel interface also transitions to the up state. However, note the timing of the system messages: *Mar *Mar *Mar *Mar 1 1 1 1 00:45:50.647: 00:45:50.683: 00:45:50.691: 00:45:53.487: %LINK-3-UPDOWN: %LINK-3-UPDOWN: %LINK-3-UPDOWN: %LINK-3-UPDOWN: Interface Interface Interface Interface FastEthernet0/14, changed state FastEthernet0/13, changed state FastEthernet0/15, changed state Port-channel1, changed state to to up to up to up up

Almost a full three seconds elapsed between the member ports transitioning to the up state and the port-channel interface coming up. Once it did, we can see the state of the EtherChannel has changed to "in use": S1# show etherchannel summary Flags: D - down P - bundled in port-channel I - stand-alone s - suspended H - Hot-standby (LACP only) R - Layer3 S - Layer2 U - in use f - failed to allocate aggregator M - not in use, minimum links not met u - unsuitable for bundling http://packetlife.net/blog/2010/jan/18/etherchannel-considerations/ Page 2

w - waiting to be aggregated d - default port Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------1 Po1(SU) LACP Fa0/13(P) Fa0/14(P) Fa0/15(P) Note the S indicating layer two operation; on multilayer platforms, EtherChannel interfaces can be configured for routed operation as well. For comparison, let's reconfigure the EtherChannel to function without a negtiation protocol ("on" mode): S1(config)# no interface po1 S1(config)# interface range f0/13 -15 S1(config-if-range)# channel-group 1 mode on Creating a port-channel interface Port-channel 1 S1(config-if-range)# no shutdown This time we observe that the port-channel interface is enabled as soon as its first member port comes up, as there is no delay imposed by negotiation: *Mar *Mar *Mar *Mar 1 1 1 1 00:56:12.271: 00:56:12.287: 00:56:12.291: 00:56:12.307: %LINK-3-UPDOWN: %LINK-3-UPDOWN: %LINK-3-UPDOWN: %LINK-3-UPDOWN: Interface Interface Interface Interface FastEthernet0/13, changed state Port-channel1, changed state to FastEthernet0/14, changed state FastEthernet0/15, changed state to up up to up to up

In the Campus Network High Availability Design Guide, Cisco recommend forgoing the use of a negotiation protocol and configuring EtherChannels for static "on/on" operation; however they also caution that this approach offers no protection against the effect of misconfigurations.

EtherChannel Load-Balancing
Another consideration to make when implementing EtherChannels is the type of load-balancing in effect. EtherChannel provides load-balancing only per frame, not per bit. A switch decides which member link a frame will traverse by the outcome of a hash function performed against one or more fields of each frame. Which fields are considered is dependent on the switch platform and configuration. For example, a Catalyst 3550 can match only against a frame's destination or source MAC address: S1(config)# port-channel load-balance ? dst-mac Dst Mac Addr src-mac Src Mac Addr The show etherchannel load-balance command reveals that source MAC address load-balancing is default on the Catalyst 3550: S1# show etherchannel load-balance EtherChannel Load-Balancing Configuration: src-mac EtherChannel Load-Balancing Addresses Used Per-Protocol: Non-IP: Source MAC address IPv4: Source MAC address http://packetlife.net/blog/2010/jan/18/etherchannel-considerations/ Page 3

More powerful platforms can match against IP address(es) or layer four port(s). Generally speaking, higher layer fields are more favorable as they tend to be more dynamic, resulting in a more granular distribution of traffic across member links. Direction of flow is also an important detail. For example, consider the following topology:

Routed packets entering the subnet from S1 are always sourced from the MAC address of the VLAN interface. If source MAC load-balancing is in use, these frames will be forwarded down only one member link, because the outcome of the hash function will always be the same. Configuring destination MAC load-balancing on S1 is recommended to achieve a more varied distribution of frames and make better use of the available bandwidth. The opposite is true on S2: Since all frames entering the EtherChannel from LAN hosts are destined for the MAC address of the gateway (VLAN interface), source MAC address load-balancing works better here.

EtherChannel Bandwidth and Costs

Finally, remember that the perceived bandwidth of a port-channel interface is equal to the sum of its active member links. For example, an EtherChannel with three active 100 Mbps members will show a bandwidth of 300 Mbps. Because members can still fail individually, the bandwidth of a port-channel interface can fluctuate without going down. For more information on EtherChannel bandwidth and spanning tree considerations, see Etherchannel costs and failover. Posted in Switching

http://packetlife.net/blog/2010/jan/18/etherchannel-considerations/

Page 4