e

Strategic Technology Infrastructure for Regional Competitiveness in the Network Economy

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

eCorridors Program

2003 Virginia Polytechnic Institute and State University, Blacksburg, VA. All rights reserved.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Preface
This series of reports, entitled Strategic Technology Infrastructure for Regional Competitiveness in the Network Economy and packaged in eleven Volumes, is the culmination of a dedicated effort of the following individuals and organizations. Each Volume can be viewed as a stand-alone publication; however, it should be noted that each Volume was written in the context of the overall project. The project utilized the Southside and Southwest Virginia regions as a model for a low-cost Geodesic Mesh network design and viable financial model that could be replicated in any region of the U.S.

Volumes
1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) Rationale, Environment, and Strategic Considerations Connecting the Regional Infrastructure to National and International Networks A Fiber Optic Infrastructure Design for Southside and Southwest Virginia Fiber Optic Infrastructure Design Guide Financial Feasibility and Investment Rationale Leveraging Advanced Optical and Ethernet Technologies Speculative and Alternative Technologies Community, Applications and Services Demographics for Southside and Southwest Virginia Health Information Technology and Infrastructure Education in the 21st Century

Volume 1: Rationale, Environment, and Strategic Considerations

emntsckowldgA
The following individuals and organizations contributed to the development and preparation of this series of reports.
Allen, Morgan Arellano, Christian Aughenbaugh, John Bevis, Jeff Blythe, Erv Bohland, James Bottom, Beth Bowden, Phillip Brown, Eric Charlton, Garland Chen, Daniel Cohen, Marc Colbert, Joy Croasdale, Hud Crowder, Jeff Dalton, Jody de Vries, Marten Dwyer, Sharon Fisher, Tommy Franklin, Nancy Gaylord, Clark Hach, Richard Hall, Shannon Hares, Glynn Harris, Carl Hey, Bryan Hoover, Maynard Horton, Helen Jones, Brian Jones, Doug Kanter, Theresa Kidd, Jeff Lee, Steven Lilly, Judy L. Martin, David Martin, Evelyn Mathai, Mathew McCann, Jessica Morrison, Brandon Neidigh, Brenda Nichols, John Pelt, Ranson Perry, Mike Pheley, Al Plymale, V. Jean Plymale, Bill Pollard, John Rodgers, Pat Sanghvi, Harsh Shepherd, Scott Sheppard, Scott Shumaker, Richard Stewart, Jeb Stock, Doris Tyree, Charles Waddell, Bobby Wenrich, John Woods, Cindy Zirkle, Mary

ii

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Organizations and Companies

3com Adelphia Advance Fiber Optics Advanced Network Infrastructure & Services, VA Tech AEP AFL Telecommunications Alcatel Anderson & Associates Asia Venture Partners AT&T Avante Bristol Virginia Utilities Board Celion Center for Wireless Telecommunications, VA Tech Chamber of Commerce, Richlands Chilson Enterprises Cisco Systems Corning Cable Systems Cox Communications Danvilleonline.com Dominion Telecom Economic Development Assistance Center, VA Tech EngHouse Systems Enterasys Extreme Networks Economic Development Assistance Center, VA Tech Floyd County High School Floydva.com Force 10 Foundry Future of the Piedmont Foundation Gamewood, Inc. GeoTel Grant County Public Utility District Hatteras Hewlett-Packard IBM Institute for Advanced Learning and Research, VA Tech Institute for Connecting Science Research to the Classroom, VA Tech ION Consulting KMI Corporation LENOWISCO Level 3 Communications, Inc. MapInfo Manticom Marketing Dept., VA Tech Micrologic, Inc. Nexans Nortel Old Dominion Electric Coop Pirelli Prince Edward County Office of Economic Development Progress Telecom Qwest RACO, Inc. Rinderva.com Riverstone Salira Sprint Terabeam Urban Affairs and Planning Dept., VA Tech Valleynet Verizon Wiltel Worldcom Worldwide Packets

iii

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Table of Contents
Introduction .....................................................................................................................1 Historical Perspective......................................................................................................3 What is Metropolitan/Regional Ethernet? ........................................................................6 Ethernet in the First Mile...........................................................................................7 Backbone Networks for Non-Ethernet Broadband Access......................................11 Interconnecting Service Provider Networks ...................................................................14 Cross-connects as Meet Points ..............................................................................14 Packet Switching or Multiplexing as Meet Points ....................................................16 Collocation at Meet Points......................................................................................16 Ethernet-Based Internet Exchange Points ..............................................................17 Cost and Manageability Benefits of Metro Ethernet .......................................................19 Limitations of Current Metro Ethernet Technology.........................................................22 Slow Recovery From Link Failures .........................................................................22 Lack of Remote Fault Isolation ...............................................................................24 Lack of In-Service Performance Monitoring and OAM ............................................25 Limited VLAN Tag Space .......................................................................................25 Spanning Tree Inefficiencies on Highly Meshed Networks .....................................26 Lack of End-to-End Service Guarantees.................................................................26 Is Metro Ethernet Ready for Prime Time?...............................................................27 Best Practices for Metro Ethernet Networks ..................................................................29 Architecture: Link Layer or Network Layer Switching..............................................29 Spanning Tree Configuration..................................................................................30 Forwarding Table Considerations...........................................................................31

iv

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Protocol Filtering ....................................................................................................32 Rate Limits on Broadcast and Multicast Frame Flooding ........................................32 IP Multicast Frame Flooding and Rate-Limiting ......................................................33 Quality-of-Service Controls.....................................................................................33 Acronym Glossary .........................................................................................................35 Acknowledgements .......................................................................................................38 References....................................................................................................................39

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

List of Figures
Figure 1: MSAP extending access network .....................................................................8 Figure 2: MSAPs connecting Ethernet rings ....................................................................9 Figure 3: Schematic of traditional DSL access network .................................................11 Figure 4: MSAP connecting multiple ISPs and access networks ...................................12 Figure 5: Schematic of cross-connect pedestal as meet point .......................................15 Figure 6: Collocation at meet points ..............................................................................17 Figure 7: Link Aggregation between MSAPs .................................................................23

vi

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Introduction
Telecommunications services delivery over the last ten years has been dominated by digital circuit-based technologies such as Time Division Multiplexing (TDM) and Synchronous Optical Network (SONET). These technologies are readily adapted to large-scale voice communications, as they effectively multiplex resources with fixed units. For the same reasons, they are less suited for data communications, where the resource demands of the applications are highly variable. Packet switching technologies such as Frame Relay and ATM played an important role in the unprecedented growth in the data telecommunications market, ushered in by the rise of the Internet. With the realization of digital voice and video transmission, the efficiencies of packet switching can also be applied to these applications. Inarguably, all of these technologies have been extraordinarily effective. Yet, they where designed and developed for a very different telecommunications economy than that which exists today. In todays telecommunications market, service providers must be agile to seize opportunities. The Internet and all things around it move at a very fast pace. Service providers must be able to scale capacities upward to meet surging demands in order to remain competitive. Todays service provider cannot assume a decades-long return on capital investments in switching, multiplexing, and line termination equipment. While SONET, ATM, and Frame Relay will continue to play an important role in telecommunications for many years to come, increasingly, service providers are looking to Ethernet technology as the platform for cost-effective delivery of converged voice, video, and data telecommunications services. Originally developed for use in local area network environments, Ethernet has in recent years evolved such that it is a costeffective, robust, scalable, manageable platform for metropolitan and regional telecommunications. The ubiquity of Ethernet technology in enterprise networks and its increasing role in metropolitan networks has created an enormous and highly competitive market for Ethernet equipment. This, along with the relative simplicity of the technology, has led to twenty-year history of ten-fold increases in link capacities for approximately three times the cost of the preceding generation of equipment.[4] 1

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Furthermore, the near-total market adoption of each new generation of Ethernet technology has ultimately driven down costs even further. In this report, we examine the role that Ethernet technology can play delivering telecommunications services on a metropolitan and regional scale. We examine the case in favor of so-called metro Ethernet networks, and consider the foremost limitations of the current generation of metro Ethernet equipment.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Historical Perspective
Early wide area data telecommunications networks utilized modems on dial-up or leased telephone lines. The point-to-point nature of such services greatly limited scalability, since a new line was needed for each concurrent data connection to another remote location. True scalability in wide-area data telecommunications networks was ushered in by packet switching. Packet switching protocols such as X.25 became popular because a single dial-up or leased line connection to the network allowed communication with multiple remote locations via the concept of virtual circuits. The first packet switching protocols were burdened with the need to operate effectively over analog leased lines with very high bit-error rates. The X.25 protocol and its many predecessors and relatives were greatly complicated by error detection and correction mechanisms. With the advent of digital transmission lines, the need for a lightweight packet switching protocol resulted in the development and large scale deployment of Frame Relay. Frame Relay networks retained the fundamental label-swapping techniques of X.25 virtual circuit switching while dispensing with the error correction mechanisms. The simplicity and elegance of Frame Relay allowed it to easily operate at speeds up to 1.5 megabits per second, which was quite impressive at the time. Motivated by a desire to converge applications such as voice, video, and data communications on the same network, and to meet the future needs for increased bandwidth, in the late 1980s the telecommunications industry developed specifications for Broadband ISDN (B-ISDN). The B-ISDN specification consisted of two significant components: Synchronous Digital Hierarchy (SDH, known in the U.S. and hereafter in this document as SONET) and Asynchronous Transfer Mode (ATM). SONET provided a robust ring-based architecture for synchronous digital transmission over fiber-optic cable. SONET was critical to meeting the capacity demands imposed by rapid growth and competition in the long distance voice market, as well as the unprecedented demand for bandwidth that followed the privatization and subsequent commercialization of the Internet.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies ATM was intended to address the inefficiencies of time division multiplexing (TDM) that were inherent to SONET and its T1 roots. Through the use of statistical multiplexing and virtual-circuit-level class-of-service parameters, it allowed delay sensitive traffic classes such as voice and video to be mixed freely with other traffic types on a common digital transmission line. Proponents of ATM believed that it would become the fundamental means of supporting voice, video, and data telecommunications. Carrier deployments of ATM in the mid-to-late1990s further fueled the rapid growth of the Internet and the use of the Internet Protocol (IP) for intranet, extranet, and other applications, by providing more bandwidth and more flexibility than had previously been available. ATM also contributed to the continued success of Frame Relay, by providing a scalable backbone network that was interoperable with Frame Relay and would allow carriers to meet the growing demands for Frame Relay service that accompanied the rise of the Internet. While ATM continues to be an important component of carrier networks, it did not achieve the goal of convergence for which it was originally intended. The success of ATM in achieving convergence depended on end-to-end deployment of the technology. In particular, this meant that the enormous installed base of personal computers in enterprise networks would need to be directly connected to local area ATM networks in order to fully leverage the capabilities of ATM. However, in local area networks, there was already a dominant technology that would not be easily displaced: Ethernet. When ATM emerged as a potential LAN technology, it offered significant advantages over Ethernet. At that time, Ethernet was relatively primitive, relying on broadcast media, and providing only 10 megabits per second of bandwidth shared between all users on the LAN. Furthermore, Ethernet provided no capability for differentiated levels of service that would allow mission-critical or time-sensitive applications to be prioritized above routine traffic on the network. Ethernet networks were interconnected using transparent bridges and multi-protocol routers, which partitioned the Ethernet into smaller shared segments. Partitioning the network in this manner added stability and scalability, at the expense of greater end-to-end packet delay, and increased packet loss. Despite these shortcomings, Ethernet was inexpensive and easy to implement and was adopted by enterprise networks of all sizes.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies The installed base of existing shared Ethernet might not have precluded ATM from taking over the LAN environment. However, the development and rapid deployment of Ethernet switching technology and 100 megabit per second Ethernet delivered a crushing blow to any hope that ATM might one day rule the enterprise network environment. Ethernet switching was a relatively simple variation on the transparent bridge. Advances in semiconductor technology enabled the development of inexpensive high-density Ethernet switches that could be used to replace existing shared hubs. This allowed the existing Ethernet base to migrate to networks providing much greater amounts of bandwidth, while not requiring wholesale replacement of existing components. In particular, existing desktop computer hardware and software could continue to be used on a switched Ethernet network. Fast Ethernet, as the 100 megabit per second variant is known, provided the means to increase the bandwidth on backbone segments by an order of magnitude, in addition to providing very high capacity links for network intensive server applications. Enterprise network managers found that by simply eliminating congestion on their networks, virtually all applications could be made to work successfully, without the need for ATMs advanced traffic management capabilities. Any remaining hope for ATM in the enterprise was lost as Ethernet switches evolved to include features such as traffic prioritization, rate limiting, and advanced queuing. These features are critical to supporting real-time applications such as voice and video. The maximum Ethernet link speed also increased by an order of magnitude, to 1000 megabits per second (1 gigabit per second). The extraordinary capacity and advanced traffic management capabilities of Ethernet obviated any need for a technology other than Ethernet at the data link layer of the enterprise. Indeed, by the turn of the century, rapid advances in Ethernet technology made it feasible for use in metropolitan area networks where SONET, ATM, and Frame Relay technologies ruled. Todays Ethernet technology supports line speeds from 10 megabits per second to 10,000 megabits per second (10 gigabits per second). Only SONET itself offers line speeds that are comparable to Ethernet, but the inefficiencies of the time division multiplexing inherent to SONET combined with the historically higher capital costs for SONET equipment make Ethernet a far more cost-effective alternative in metropolitan and regional area networks.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

What is Metropolitan/Regional Ethernet?

A metropolitan or regional Ethernet (known herein as a metro Ethernet) is a packet switched network that employs Ethernet technology for wide-area connectivity, especially within a metropolitan area or at a similar regional scale. Metro Ethernet services are typically used by enterprise networks and access service providers for connectivity to the public Internet and to extend the functionality of corporate networks between geographically separate sites. Nodes in a metro Ethernet are switches operating at either the OSI data-link layer or at the OSI network layer (where the Internet Protocol is the universal network-layer protocol choice). Often, some combination of the services provided by each of these OSI layers is used. Links between nodes in a metro Ethernet are typically a physical point-to-point connection, provisioned over fiber or optical services (such as DWDM), and can operate at any Ethernet line speed (10 megabits per second to 10,000 megabits per second) depending on the service requirements. Typically, metropolitan and regional fiber networks have ring architectures. While metro Ethernet can successfully operate over ring topologies, todays standards-based Ethernet is better suited to mesh topologies. Consequently, much of the emphasis in metro Ethernet development is in efficiently supporting ring topologies with a robust, highly-resilient Ethernet technology. Metro Ethernet network services are provisioned using virtually any combination of logical point-to-point, point-to-multipoint, or multipoint-to-multipoint configurations, over links operating at any Ethernet line speed. Bandwidth can be allocated to such services in increments as small as 1 kilobit per second. Bandwidth can be dedicated or shared between multiple service users. Quality-of-service is implemented in metro Ethernet networks using a combination of many techniques operating at both the data-link layer and the network layer. These techniques include most of the same capabilities found in Frame Relay and ATM

Volume 6: Leveraging Advanced Optical and Ethernet Technologies networks, including packet classification, marking, rate limiting or policing, and transmit queue scheduling with multiple queues. Some examples of common and proposed uses of metro Ethernet networks are: Residential and commercial Internet access (so-called Ethernet in the First Mile) Backbone networks for other broadband access technologies (e.g. DSL, cable modem, wireless broadband) Corporate LAN extension

Ethernet in the First Mile

Residential and commercial Internet access can be delivered using metro Ethernet networks. In this model, Ethernet connections are extended to the customer premise using a 1000Base-X Ethernet connection over fiber-optic cable. Ethernet switches are located in central-office-like facilities known as Multimedia Service Access Points (MSAP) or in fiber termination pedestals located in utility easements. Figure 1 shows an Ethernet switch located in a community MSAP that is providing 1000Base-X Ethernet connections to homes and businesses in the local community.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

MSAP

Internet
Ethernet Switch

Ethernet

1000Base-X

SONET ADM 1000Base-LX ONT 10/100/1000TX Wireless IP Phone Ethernet Switch Router

MSAP

Figure 1: MSAP extending access network

Using the regional fiber architecture recommended in this report, Figure 1 shows the metro Ethernet network extending the access network in the community to a distant MSAP (perhaps in a larger city) where an Internet Service Provider can deliver highcapacity access to the Internet using prevailing SONET technologies. Because of the prevalence of ring topologies in metropolitan and regional fiber networks, Ethernet in the first mile will typically be implemented using a combination of MSAP facilities and smaller distribution switches located in fiber termination pedestals along utility easements. As shown in Figure 2, small Ethernet switches are arranged on fiber rings passing through residential communities and commercial districts. The logical ring topology minimizes the number of fiber pairs consumed on the physical fiber ring. These smaller distribution switches are located close to customers, such that the costs for additional fiber laterals to reach each customer premise are substantially lower.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

1000Base-X

1000Base-X

1000Base-X

Internet

MSAP

MSAP

1000Base-X

1000Base-X

1000Base-X

Figure 2: MSAPs connecting Ethernet rings

A community may have multiple logical Ethernet rings providing access services in different areas of the community. In this case, the MSAP typically acts as the hub of the community-area distribution networks. Leveraging the regional fiber infrastructure proposed in this report, each MSAP is interconnected to other MSAPs, allowing Internet and application service providers to be located wherever it is most advantageous. The IEEE Ethernet in the First Mile working group (IEEE 802.3ah) is drafting specifications to make Ethernet-based access networks scalable, manageable, and fault tolerant. Technical proposals for OAM (operations, administration, and management), customer premise network termination, and for both fiber- and copper-based physical layer specifications are being considered by the working group.

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Efforts are underway in the IEEE 802.17 Resilient Packet Ring working group to define fault-tolerant ring standards for Ethernet that will ultimately provide a robust architecture that meets or exceeds the resiliency of venerable SONET ring architecture. The Multimedia Service Access Point is described in greater detail in the section entitled Interconnecting Service Provider Networks.

10

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Backbone Networks for Non-Ethernet Broadband Access

In small communities, broadband access providers employing DSL, cable modem, or even wireless broadband technologies can exploit a regional fiber infrastructure and metro Ethernet to reduce operating costs, and offer revenue-generating broadband access services even on a small scale. As shown in Figure 3, traditional approaches to deploying DSL technology have employed SONET-based ATM backbone networks interconnecting telco central offices. The DSL access multiplexer (DSLAM) used to provide connections to residential or commercial subscribers is connected to this ATM backbone network. The ATM backbone provides the means of interconnecting service points to backbone resources such as tier 1 Internet service providers and application service providers. Because of the high infrastructure costs associated with the SONET/ATM architecture, these solutions are not cost effective for smaller communities, where the potential subscriber base is small.

Central Office

Internet
DSLAM ATM Switch

ATM

SONET ADM ATM Switch Router IP Phone Wireless

DSL Modem

POP

Figure 3: Schematic of traditional DSL access network

11

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Using the community MSAP model and regional fiber infrastructure described in this report, it is feasible for even a small rural telephone cooperative to cost-effectively deploy DSL Internet access services. In the lower-left of Figure 4, we see the central office continuing to serve as the termination point for residential and commercial DSL connections. Using 1000Base-X Ethernet, the DSLAM connects to the community MSAP, where access customers are interconnected with an Internet service provider. The MSAP containing Internet service providers (labeled ISP A and ISP B in Figure 4) could be co-located in the same building facility as the telephone cooperatives central office, or it could be in some other part of the region, where the costs for traditional highcapacity SONET-based access to the Internet are lower.

1000Base-X

1000Base-X

1000Base-X

MSAP
Internet Internet

ISP A
1000Base-X

ISP B
1000Base-X

Central Office
Broadband Wireless DSLAM Ethernet Switch

Figure 4: MSAP connecting multiple ISPs and access networks

12

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Figure 4 also illustrates Ethernet-based network access, DSL access, and wireless broadband access all coming together at the community MSAP. This emphasizes a key function performed by the MSAP: interconnection.

13

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Interconnecting Service Provider Networks

Both the Internet and the public switched telephone networks are intricate meshes of connectivity between different service provider networks. Interconnection allows service providers to specialize in network access or higher-layer services, extends the potential market for all service providers, provides better performance for traffic between these networks, and reduces the cost of upstream connectivity to higher-tier providers. There are two basic strategies for interconnecting between such networks: private direct connections and connections at common meet points. When there is more than one other service provider to connect with, the expense of having direct connections to each can be significantly more than connecting to all or most of them at a common meet point. One of the motivations for direct connections is the desire for service providers to exercise more control over resources and limit the exposure to risk from other entities managing those connections. In order to engender the trust of numerous service providers, meet points are best operated by neutral entities. Such neutrality addresses the concern over potential competitors having control over a service providers critical interconnection resources. The technologies for these meet points may be variable, both in terms of the physical medium and the link-layer technologies used. While the interconnection medium could be copper for some applications, we will focus on fiber as the principal physical interconnection medium. The choice of link-layer is independent from the perspective of interconnection methodology, though the most cost-effective and suitable choices will be important to successful implementation.

Cross-connects as Meet Points

The simplest form of a common public meet point is a cross-connect pedestal in a public right-of-way. In such a scenario, service providers need to make only one fiber build of sufficient capacity to this meet point. Having made this investment once, a service provider can then connect with any others who have likewise constructed facility to this

14

Volume 6: Leveraging Advanced Optical and Ethernet Technologies point. This may reduce costs significantly over constructing separate facilities to each potential other service provider. The value of the cross-connect meet point may be increased by having numerous such meet points distributed throughout a geographic region. The entity operating the crossconnect point could connect these distributed meet points via fiber optic cable, and lease access to this dark fiber facility to further reduce the costs to service providers who are closer to some of these. This distributed cross-connect extends the reach of service providers beyond what they may have otherwise been able to cost-effectively construct. In the model of cross-connect meet points, each connection between service providers is still a dedicated connection and still consumes resources from switching electronics, lasers, etc. For N service providers connecting at such a meet point to each other, this is a total of N(N-1) such connection points; each of the N service providers would have N-1 connections to deploy this full mesh approach. The cost of the connecting electronics may still dominate the cost for such interconnections. If the meet point could more efficiently use these connections via shared media, this may further reduce the costs associated with service provider interconnection.
Provider A

Pedestal provides full mesh cross-connect between providers. Provider A Provider B Provider C Provider D

Provider B

Patch bay

Provider C

Provider D

Figure 5: Schematic of cross-connect pedestal as meet point

15

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Packet Switching or Multiplexing as Meet Points

Instead of having dedicated facilities for each possible connection between service providers at meet points, the meet point operator could offer packet-switching or multiplexing services. Using such services, the N service providers may require only one connection each in order to exchange traffic with any of the other service providers. The multiplexing technology could be ATM, Ethernet, WDM, IP, or any of a number of other such technologies. All of these would more efficiently use the physical connection to the meet point, with the trade-off that services would need to be compatible with the chosen multiplexing technology. Most multiplexing technologies, such as SONET or WDM, are based on dedicating virtual resources, for example time slices or frequencies, respectively. To use these dedicated virtual resources, these technologies still may have a significant amount of dedicated resources from the connecting service provider. On the other hand, using a packet-switching technology, such as ATM or Ethernet, allows for more flexible multiplexing of virtual connectivity. Of these, the cost advantages of Ethernet are significant. The operation of such multiplexing or switching services would require more involvement by the meet point operator. In this scenario, an enclosed space with power would be required, in addition to regular monitoring and management of the switching service provided by the meet point operator. However, the cost efficiencies of such a scenario may be compelling. Offering switching services at the meet point does not preclude the possibility of having physical cross-connects. Those applications or service providers for whom this is more suitable could still use such a meet point strategy and still derive the cost savings relative to independently constructed facilities.

Collocation at Meet Points

So far we have considered the meet point as an isolated point or distributed points to which the service providers would construct fiber facility, keeping all their electronics at their own facilities. Once a meet point operator has made the investment in building, power, HVAC, etc, to operate a switching service, the expansion of such a space to accommodate equipment owned and operated by the constituent service providers may not be significant. This may also provide opportunities for other service providers who

16

Volume 6: Leveraging Advanced Optical and Ethernet Technologies are better suited by collocation service to connect to the meet points. To more costeffectively connect these collocated providers, the meet point operator may provide copper-based services in addition to the fiber-based services needed to support connections from outside the facility.
Provider A Provider B

WWW Server

Mail Server

WWW Server

Mail Server

Media Server

Media Server

Meet Point 1

Provider A

Provider B

Provider C

WWW Server Meet Point 2 Meet Point 3

Mail Server

Media Server

Provider D

Provider C Figure 6: Collocation at meet points

Ethernet-Based Internet Exchange Points

As discussed in other sections of this report, the cost benefits of Ethernet as a multiplexing and switching technology make it a compelling choice for meet point services. Using Ethernet as a medium for Internet Protocol connectivity between service providers provides a basis for establishing the meet point as an Internet Exchange Point. In such a scenario, connecting service providers advertise their IP addresses to each other and share a common Ethernet network for exchanging Internet traffic. By connecting to the same Ethernet network, service providers can decide what routing policy they want to have with each other, whether they want to provide transit access to upstream Internet service providers, etc. Just as the cross-connect meet point can be distributed over a geographic region, so too can the Ethernet exchange point. By

17

Volume 6: Leveraging Advanced Optical and Ethernet Technologies building multiple locations and interconnecting them via Ethernet switches, the meet point operator can distribute the Ethernet exchange point over a region. Typically, service providers have certain local IP addresses they would like all connecting peers to be able to reach via the Ethernet exchange point. To better enable this strategy, the meet point operator can also operate route servers available to all participating service providers for the purpose of exchanging local routes. In addition to having a common Ethernet broadcast domain available for service providers who wish to exchange IP traffic, an Ethernet based meet point can be used to establish virtual private connections, using Ethernets Virtual LAN technology, 802.1Q. Using VLANs, connecting service providers can use the same physical Ethernet connection to virtually connect to other providers, in much the same fashion as ATM and Frame Relay provide virtual circuit functionality. Such an Ethernet meet point, with available route servers and virtual LANs, has been dubbed a Multimedia Service Access Point (MSAP). In addition to these stated MSAP services, there may be dark fiber and collocation facilities available, as well as management access to collocated equipment. By expanding from the role of simple cross-connect to an open architecture for Internet based services, the MSAP leverages the cost-effective and ubiquitous nature of Ethernet. This allows for the myriad of Internet applications to be offered by providers at the MSAP: electronic mail, web hosting, streaming media, fiber-based residential and business Internet access all these become enabled by having an open, provider-neutral infrastructure for service providers.

18

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Cost and Manageability Benefits of Metro Ethernet

Ethernet links operating at 1 gigabit per second can be extended over fiber optic cable at distances of up to 70 kilometers, without amplifiers or regeneration. For less than US$40,000 in capital expenditures, it is possible to light a fiber span of up to 70 kilometers and immediately provide gigabit Ethernet services between two distant locations, with all of the physical redundancy and fault resiliency of SONET1. The cost of lighting the same fiber span using OC-12 SONET (which operates at only 622 megabits per second) is almost US$130,0002. This difference in capital cost is at least partly reflective of the different economies of scale for the manufacture of components needed by each technology. The nearly ubiquitous adoption of Ethernet in enterprise networks has created a much larger market and far more competitive pricing for Ethernet products than for SONET products. SONET sales are typically limited to the service provider market sector. In addition to the clear cost advantage, Ethernet provides other benefits not available in the SONET model. Constraints imposed by the SONET architecture preclude using the entire capacity of the facility (622 megabits per second) for any particular connection. Typically, only as much as one quarter of the available ring capacity (155 megabits per second) can be provisioned for any logical circuit on the ring. Because SONET uses time division multiplexing with very coarse-grained bandwidth divisions, the capacity of the ring will typically be underutilized even when the ring is fully provisioned. Service providers must charge customers for more capacity than they are actually using because the provider has no means with which to recover unused

Based on an implementation using Cisco Catalyst 3550 gigabit Ethernet switches and 1000Base-ZX optical transceivers, with two fully redundant physical connections between the switches. 2 Based on an implementation using Cisco 15454 SONET multiplexers with dual rings for redundancy. 19

Volume 6: Leveraging Advanced Optical and Ethernet Technologies capacity in a SONET-based service delivered to one customer for use in meeting another customers needs. The Ethernet approach uses statistical multiplexing and highly granular bandwidth allocation, just like its cousins, ATM and Frame Relay. These attributes combine to give the service provider an extraordinary degree of flexibility to squeeze as much revenue as possible out the link capacity. While ATM and Frame Relay can provide similar statistical multiplexing with granular allocation of bandwidth, they cannot compete with Ethernet on the basis of capital cost. The cost for lighting the same fiber span using ATM or Frame Relay (while providing the same level of fault resiliency) includes all of the costs for the SONET solution plus the costs for ATM or Frame Relay switches. Furthermore, neither of these solutions can provide the same bandwidth. As previously noted, the maximum link speed available to ATM or Frame Relay implemented over an OC-12 SONET ring is only 155 megabits per second. In addition to lower capital costs, the Metro Ethernet Forum cites rapid provisioning as a key benefit of Ethernet in the metropolitan/regional space. The present lack of customer-centric flexibility, as well as the coarseness of bandwidth granularity for [SONET- and ATM-based] legacy systems are considered major impediments to providing promising, revenue-generating services[4]. Citing service velocity as a key competitive differentiator, the Metro Ethernet Forum promotes Ethernets ability to offer services with line speeds ranging from 10 megabits per second to 10 gigabits per second, and configurable bandwidth provisioning, provided quickly and on-demand. Metro Ethernet has other advantages when applied to enterprise LAN extension applications. Because of the cost-effectiveness of the metro Ethernet solution (largely a product of lower equipment costs), it is possible for service providers to offer customers much greater bandwidth for the same money. This allows enterprise networks to be distributed over larger geographic regions without the WAN penalty the traditional difference in bandwidth available between the LAN and the WAN, due largely to the cost of WAN bandwidth. Additionally, because the enterprise network has traditionally been implemented using Ethernet technology, the metro Ethernet solution avoids complex, costly, and difficult-to-manage internetworking solutions to adapt applications to

20

Volume 6: Leveraging Advanced Optical and Ethernet Technologies prevailing legacy WAN technologies. Ethernet end-to-end results in enterprise network extension that is far more seamless than traditional WAN solutions.

21

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Limitations of Current Metro Ethernet Technology

The Metro Ethernet Forum has identified the following limitations to the use of Ethernet in metropolitan and regional networks: Slow recovery after link failures Lack of remote fault management Lack of in-service performance monitoring and OAM Limited VLAN tag space Inefficiencies of spanning tree relative to redundant link utilization in highly meshed topologies No end-to-end service guarantees.

In the following sections, we will address each of these limitations in some detail, along with a description of some of the protocols and proposals that seek to overcome these limitations.

Slow Recovery From Link Failures

The Spanning Tree protocol (IEEE 802.1D) used in Ethernet networks detects link failures and makes topological adjustments needed to restore network service with a convergence time that is between 50 and 130 seconds. While adequate for some applications, this is a far cry from the 50 millisecond link failure recovery time of SONETs automatic protection switching (APS). Multiple efforts are underway using different approaches to address this shortcoming. So-called carrier grade services are implemented with the fundamental assumption that service restoration occurs in less than 50 milliseconds. Therefore, many in the metro Ethernet camp feel that it is imperative that Ethernet networks be able to match SONET in this regard. On the near horizon, the Rapid Reconfiguration protocol (IEEE 802.1w) represents an incremental improvement. Using this protocol, Ethernet networks with particular

22

Volume 6: Leveraging Advanced Optical and Ethernet Technologies topological characteristics can recover from link failures in approximately 1 second. The faster recovery provided by the 802.1w protocol greatly increases the number of applications that can be supported on metro Ethernet. Pre-standard support for the 802.1w protocol is appearing in new Ethernet switches from many vendors, allowing service providers to begin leveraging metro Ethernet to deliver services today. Ratification of the 802.1w specification is expected in the near future. The Link Aggregation protocol (IEEE 802.3ad) can also be used to vastly improve the resiliency and recovery time of metro Ethernet networks. By employing parallel links between Ethernet switches, and utilizing diverse fiber paths, the IEEE 802.3ad protocol can provide load sharing between the parallel links when both links are operational. When a link is broken, failover to the remaining link occurs with a convergence time on the order of 500 milliseconds. See Figure 7.

MSAP

MSAP

Ethernet Switch

1000Base-X 802.3ad Link Aggregation

Ethernet Switch

MSAP

Passive Fiber Pass Through

Figure 7: Link Aggregation between MSAPs This is obviously an improvement over 802.1w Rapid Reconfiguration (though an order of magnitude worse than SONET), and is very simple to implement. However, many metro fiber networks have ring topologies that do not lend themselves to using Link Aggregation protocol as a means to improve resiliency. Several vendors (e.g. Extreme, Riverstone) have implemented proprietary approaches to matching or at least approaching the 50 millisecond recovery time in metro Ethernet

23

Volume 6: Leveraging Advanced Optical and Ethernet Technologies networks. These approaches tend to borrow from the architecture of SONET, creating a dual ring topology that transports Ethernet frames. Extreme claims that their Ethernet Automatic Protection Switching (EAPS) dual-ring architecture has a recovery time no worse than 100 milliseconds. The IEEE Resilient Packet Ring working group is presently standardizing such ring-based approaches (as IEEE 802.17), with a goal of matching or even improving upon the SONET benchmark. Proponents of network layer (IP) switching promote Ethernet-over-MPLS (EoMPLS) as the means by which metro Ethernet networks can achieve the resiliency of SONET protection switching, while at the same time addressing other shortcomings in the area of service-level guarantees. The MPLS functions that provide traffic engineering over an MPLS cloud can be used to provide Ethernet transport with guaranteed bandwidth and 50 millisecond recovery from link failures.

Lack of Remote Fault Isolation

The SONET architecture provides a very effective means of isolating faults to the particular SONET path, line, or section that is experiencing a fault, through the use of overhead bytes in the SONET frame, as well as the Remote Defect Indicator (RDI) and Loss of Signal (LOS) indication at each SONET interface. The 10 gigabit Ethernet standard includes, in the wide area network physical interface specification (the socalled WAN PHY for running over a SONET OC-192c carrier), the ability to map SONET fault isolation into meaningful concepts at the logical interface. In general, however, Ethernet does not presently possess analogous functionality. In the long-haul applications for which SONET is often employed, remote fault isolation is imperative to network manageability. In metro applications, the degree to which the lack of these capabilities is a detriment to the manageability of Ethernet technology is debatable. Remote fault isolation is less critical in Ethernet in part because its architecture is far less complex. Large enterprise networks, based on Ethernet and spanning very large campuses, have been operated for many years without remote fault isolation. Few enterprise network managers would argue that the lack of remote fault isolation makes their networks more difficult to manage.

24

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Remote fault isolation is an area of active interest and research in the Metro Ethernet Forum and certain IEEE working groups.

Lack of In-Service Performance Monitoring and OAM

Customer services provisioned over SONET-based services such as DS1, DS3, and OC3c are terminated at a demarcation point (typically on the customer premise) using an intelligent network termination device. This device typically provides the means by which overhead bits in frames traveling on the circuit can be used to direct the termination device to loop back the circuit and report the bit error rate (BER). This capability allows the provider to monitor and test the loop extending to the customer premise, prior to dispatching a technician, at great cost savings to the provider. Two alternative proposals to providing this capability are being considered by the IEEE 802.3ah Ethernet in the First Mile working group. One proposal suggests the use of the Ethernet preamble, and the other offers a frame-based approach. While there is presently no standards-based approach to providing analogous functionality for metro Ethernet customer access loops, many vendors are developing proprietary approaches to solving this problem. For example, Cisco has an Optical Network Terminator device for use with their Cisco Catalyst 4000 series switches that provides remote OAM functionality for metro Ethernet networks.

Limited VLAN Tag Space

The IEEE 802.1Q standard defines a VLAN tag address space of only 4096 tags. This may be insufficient for a large service provider. Many equipment vendors are implementing so-called Q-in-Q approaches to stacking VLAN tags that, along with careful planning and partitioning of the network to allow some tag reuse, should allow networks to grow to reasonably large proportions. While tag stacking approaches are proprietary, various MPLS techniques being considered by IETF working groups hold the promise of providing standards-based approaches to better scalability of metro Ethernet networks.

25

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Spanning Tree Inefficiencies on Highly Meshed Networks

The advent of standards-based virtual LAN support in the IEEE 802.1Q specification was not accompanied by a change in the Spanning Tree protocol (IEEE 802.1D). Standards-based Ethernets with multiple virtual LANs continued to use a single spanning tree, shared by all virtual LANs in a common broadcast domain. Since there can be only one loop-free path in a spanning tree, this limitation can result in inefficient use of the network redundant paths in a meshed network topology must remain completely idle by design of the protocol.3 Per-VLAN Spanning Tree (PVST) is an approach implemented by Cisco and other vendors. With PVST, each VLAN has a distinct spanning tree. Per-VLAN Spanning Tree allows for load balancing across VLAN trunks. Each spanning tree instance has its own configuration messages and other overhead, which can be quite expensive (in terms of CPU cycles) as the number of VLANs increase. By contrast, the single common spanning tree of standards-based Ethernet alleviates concerns about protocol overhead but does not allow VLAN load balancing. The IEEE 802.1s Multiple Spanning Trees specification will standardize the use of multiple spanning trees. The draft 802.1s specification combines the best aspects of per-VLAN spanning tree and the common spanning tree of 802.1Q. In 802.1s, each spanning tree creates a loop-free logical topology for a configured subset of the VLANs in the network. This allows VLAN load sharing on highly meshed networks, while limiting the number of spanning tree instances and associated overhead.

Lack of End-to-End Service Guarantees

Unlike ATM, Ethernet does not have inherent quality-of-service guarantees. This is not to say that an Ethernet network cannot provide engineered quality assurance to specific application subsets. Most Ethernet switches designed for metro Ethernet applications

An alternative approach that allows parallel redundant links to share the network load is the use of the IEEE 802.3ad Link Aggregation protocol. In this case, the parallel redundant links appear to the Spanning Tree protocol as a single logical link. 26

Volume 6: Leveraging Advanced Optical and Ethernet Technologies have the ability to classify and mark 802.1Q frames and/or IP packets for elevated priority, to police traffic classes at particular rates, and to provide multi-queue transmit disciplines such as Weighted Round Robin, and Strict Priority. By implementing appropriate trust boundaries, and using these mechanisms, it is possible to provide an appropriate service level for delay-sensitive applications (e.g. voice, video) in the presence of bursty, best-effort data applications. It is not possible, using current Ethernet technology to, implement quality-of-service guarantees specifying acceptable packet loss, delay, and jitter parameters, with dynamic admission control and optimal path selection. One could argue that while ATM has the capability of signaling QoS requirements in the call setup, it has seen little use in part because of scalability, interoperability, manageability, and policy concerns in the service provider space. Most often, ATM QoS has been applied to permanent virtual circuits, where admission control and optimal path selection are manually determined. Over the long term, proponents of metro Ethernet point to MPLS and its ability to provide QoS guarantees that are analogous to ATM, with dynamic signaling of QoS parameters and constraint-based routing as the solution to service guarantees for metro Ethernet. Using much of the existing hardware and software used to provide traffic classification and transmit scheduling, and by mapping prioritization between the 802.1Q priority bits and the MPLS EXP bits, metro Ethernet can easily be adapted to provide true end-toend QoS.

Is Metro Ethernet Ready for Prime Time?

Given these limitations, service providers might be reluctant to deploy metro Ethernetbased services now. While these limitations and their possible solutions are important to understand, it is also important to note that metro Ethernet, even given these limitations, can support the vast majority of todays applications. Service providers might well choose to implement a simpler, less robust metro Ethernet network today, and begin generating revenues from the enormous array of applications that can tolerate near-carrier-class service at a substantial price advantage. Indeed, the

27

Volume 6: Leveraging Advanced Optical and Ethernet Technologies success of providers such as Yipes, Telseon, and Cogent in metro Ethernet services is a testament to the business case for such an approach. Historically speaking, advances in Ethernet technology have come at a steady pace, and have been quickly adopted by the industry. As the works-in-progress of IEEE working groups and other industry forums come to fruition, the future of Ethernet technology looks very bright, indeed.

28

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Best Practices for Metro Ethernet Networks

Metro Ethernet networks have topological constraints, as well as management and security considerations that are unique to the service provider environment. Enterprise network managers have discovered, through many years of experience with Ethernet technologies, that these networks can provide extraordinary service levels, with very high performance, and excellent resiliency. But, in order to achieve these benefits, careful consideration must be given to the planning and implementation of any Ethernet network. Given that many incumbent and new service providers have not previously deployed Ethernet-based technologies on any scale, this section focuses on best practices and implementation considerations for metro Ethernet networks. It highlights the pertinent techniques and technology decisions that can benefit from the lessons learned from large-scale Ethernet deployments in both the service provider and enterprise network environments.

Architecture: Link Layer or Network Layer Switching

In planning a metro Ethernet network, one of the foremost considerations is whether the majority of the switching nodes in the network will operate at the OSI data-link layer or at the OSI network layer. Link layer Ethernet switches (also known as Layer 2 or pure Ethernet switches) have the functionality needed to perform the role of access network aggregation points in Ethernet-in-the-First Mile deployments, as well as the interconnect capabilities needed in the MSAP. Ethernet switches from a variety of different vendors provide support for QoS mechanisms needed to support the vast majority of applications. Network layer switches (traditionally known as routers) offer much greater functionality, but at a higher cost typically two to four times the cost of link layer switches. Network layer switches from several vendors can support all of the IP and MPLS functionality needed to provide a robust, high-performance, and cost-effective solution for virtually

29

Volume 6: Leveraging Advanced Optical and Ethernet Technologies every customer application. Moreover, network layer switches can work over almost any combination of Ethernet and legacy WAN technologies (ATM, PPP/HDLC over SONET), as well as dark-fiber and optical (e.g. DWDM) networks. Most metro Ethernet service provider networks will employ a combination of switches operating at both the data-link and network layers. In first-mile access and distribution networks, the use of link layer switches interconnected via 1000Base-X Ethernet is a cost-effective and manageable solution. In the core of the network, where access and distribution networks must meet Internet and application service providers, network layer switches provide the greatest flexibility, scalability, and manageability. Additionally, network layer switching solutions support transparent operation over nearly any combination of Ethernet and other link layer environments.

Spanning Tree Configuration

Careful implementation of the Spanning Tree Protocol (IEEE 802.1D) is essential to the success of any metro Ethernet. Even when the nodes of the metro Ethernet are network-layer packet switches (e.g. IP/MPLS switches), there are spanning tree implications whenever Ethernet connections are extended to customer networks. The Spanning Tree protocol is deceptively simple. The implications of the network topology and switch configuration are not always obvious, particularly in larger networks. The perils of giving inadequate consideration to the planning and implementation of Spanning Tree protocol in your network range from inefficient use of valuable (e.g. fiber line) assets, to incredibly disruptive and difficult-to-resolve anomalies known as forwarding loops. The Spanning Tree root bridge election is of critical importance. In many cases, the customers local area network will participate, to some degree, in the spanning tree protocol within your domain. To ensure stability of the network service for each customer, it is imperative that the root bridge be completely under the control of the service provider. The root bridge should be at or near the topological center of the bridging domain, and should be a switch with adequate CPU resources to run multiple spanning tree instances.

30

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Service providers are strongly advised to establish internal practices and reviews that ensure proper control over the root bridge election via the following means: Configuration of root bridge priority parameter values that ensure that the protocol will elect an administratively and topologically appropriate root bridge for each bridging domain. Configuration of the common so-called root guard feature on all customer facing ports. This feature prevents a bridge in the customers network from becoming the root bridge in the service providers domain. Many metro Ethernet networks will implement the Spanning Tree protocol using switches interconnected over physical ring topologies. Over this physical topology, Spanning Tree will create a logical hub-and-spoke topology, where one of the switches on the ring is the hub (the root bridge), and one link in the ring (most distant from the root bridge) will be blocked. In this case, best results are achieved when the hub is a bridge that interconnects many such rings, such as the switch shown in the MSAP in Figure 2. Furthermore, such a topology is an ideal candidate for deployment of the IEEE 802.1w Rapid Reconfiguration protocol, which will ensure restoration of the ring in approximately one second, should a fiber cut occur anywhere along the ring.

Forwarding Table Considerations

The forwarding table in a switch contains a list of MAC addresses and corresponding egress ports, typically on a per-VLAN basis. There are three basic considerations regarding the forwarding table: 1) Overall capacity. Switches used in service provider networks must provide ample capacity to support the maximum number of end systems that might be connected to the providers network. Most carrier class switches support on the order of 65,000 or more entries in the forwarding table per VLAN. 2) Per port capacity. No single port should be allowed to consume all of the available capacity of the forwarding table, since otherwise a denial of service attack is possible by flooding a large number of source addresses into the network from a single port.

31

Volume 6: Leveraging Advanced Optical and Ethernet Technologies 3) MAC address security. Switches used by service providers should provide the capability to accept and lock-down a limited number of source addresses from customer-facing ports. This can, in some cases, address the denial-of-service vulnerability mentioned in (2), above. This capability can also mitigate the effects of undiscovered topological loops (e.g. resulting from Spanning Tree protocol misconfiguration), by preventing addresses from being learned on any port other than the correct port.

Protocol Filtering
While metro Ethernet networks can support any higher-layer protocol that can be encapsulated in an Ethernet frame, service providers may wish to filter unwanted or unsupported protocols received from customer-facing ports. For example, residential access service providers may want to filter all Ethernet frame types other than the conventional encapsulation for IP and ARP. This would ensure that only the protocols needed for supported services are transported on the network and that customers cannot subject the network to protocol families (e.g. AppleTalk) that tend to needlessly consume network resources with overhead traffic, or that otherwise contribute to network instability. Most Ethernet switches include protocol filtering support. Service providers should implement protocol filtering as appropriate to the services delivered.

Rate Limits on Broadcast and Multicast Frame Flooding

Broadcast and multicast frames are, by default, flooded by switches to all ports on the spanning tree except the port from which the frame was received. Switches deployed by service providers must include the ability to apply rate limits to broadcast and multicast frames. Broadcast and multicast rate controls can mitigate the effects of broadcast storms and undiscovered topological loops and thus allow the network to continue to deliver service even in the face of severe misconfiguration and/or misbehavior on the part of switches in the network.

32

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Service provider Ethernet switches must provide independent configurable rate limits for broadcast and multicast frame flooding. Ideally, the limits should be configurable on a per port basis. True broadcast frame traffic (addressed to the all ones destination address) should, under normal conditions, have a very low bit rate per port. Thus, it is desirable to set broadcast rate limits to be a very small portion of the available bandwidth on each port to ensure that broadcast traffic cannot consume valuable resources or destabilize the network. If controls are provided on a per-port basis, it is possible to provide an engineered solution for broadcast frames from edge to core.

IP Multicast Frame Flooding and Rate-Limiting

Since IP multicast (used for many forms of one-to-many multimedia content delivery) uses multicast Ethernet frames, it is not desirable to impose the same highly restrictive rate limiting on IP multicast frames as should be applied to other multicast and broadcast frames. Ideally, switches should provide independent rate-limiting and scope-limiting functionality for IP multicast. Rate-limiting for IP multicast frames could provide an upper bound for all IP multicast frames, or on a per-multicast-group basis, or both. Scopelimiting should provide a means to ensure that most IP multicast frames are flooded only to those ports with downstream IP multicast group receivers, rather than flooding throughout the broadcast domain. In lieu of fully independent IP multicast rate/scope controls, independent rate controls for multicast frames (of all kinds) on a per port basis can suffice by allowing the multicast frame rate limit to be set to a higher limit than broadcast frames. When combined with protocol filtering (to filter non-IP multicast frames), this can achieve the same effect as an independent rate limit for IP multicast, at the expense of other non-IP protocols (which would in this case be summarily discarded by protocol filters).

Quality-of-Service Controls
In order to provide a reasonable foundation for providing differentiated services, appropriate for a mix of multimedia applications, switches used by service providers should have the following capabilities:

33

Volume 6: Leveraging Advanced Optical and Ethernet Technologies Ability to classify traffic at ingress port based on Ethernet frame fields and network layer attributes. In particular, the classification should allow the classification of traffic by 802.1Q priority, source or destination MAC address, IP precedence and/or DSCP, IP source/destination address, IP protocol field, and transport-layer addresses (i.e. ports) for TCP and UDP. Ability to set the 802.1Q priority and optionally (strongly recommended) the ability to set the IP DSCP field of outbound frames. Ability to police at ingress to specified bit rates based on classification as described above. Policing should minimally provide granularity on the order of 100 kilobits per second, and burst sizes ranging from 32 kilobytes up to hundreds of megabytes. At least two fully independent transmit queues per port. If only two queues are provided, the ability to select either strict priority or weighted round robin disciplines is strongly recommended. Additionally, the ability to direct outbound frames to the appropriate queue of the egress port based on classification as described above.

34

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Acronym Glossary
ARP Address Resolution Protocol. A link-layer protocol used to discover the associations between Internet Protocol (IP) addresses and Media Access Control (MAC) addresses in an Ethernet network. ATM Asynchronous Transfer Mode. A cell-switching paradigm developed as part of the ITU Broadband Integrated Services Digital Network (B-ISDN) specification. BER Bit Error Rate BPDU Bridge Protocol Data Unit. Refers to an Ethernet frame containing a Spanning Tree protocol message. DSCP DiffServ Code Point. A IP packet header field defined to contain a quality-ofservice indicator defined by the Differentiated Services (DiffServ) IETF working group. DSL Digital Subscriber Line. A technology used to provide digital services on the copper customer loop extending between a customer premise and a telephone company central office. DWDM Dense Wave Division Multiplexing. EoMPLS Ethernet over MPLS. Refers to the transparent transport of Ethernet frames over an MPLS switching cloud. IEEE Institute of Electrical and Electronic Engineers. IETF Internet Engineering Task Force. IP Internet Protocol. The OSI network layer protocol used on the Internet and in the vast majority of corporate intranets and extranets.

35

Volume 6: Leveraging Advanced Optical and Ethernet Technologies ISDN Integrated Services Digital Network. ITU International Telecommunications Union, formerly CCITT. LAN Local Area Network. LOS Loss Of Signal. A SONET fault isolation indicator. MAC Media Access Control. Most often used in the context of MAC address, which refers to a link layer frame address (e.g. an Ethernet nodes hardware address). MST Multiple Spanning Trees. The approach to multi-VLAN spanning tree specified by the IEEE 802.1s working group, wherein multiple spanning trees are operated, each one providing a loop-free logical topology for a subset of the virtual LANs within the bridged network. MPLS Multi-Protocol Label Switching, also cited as Multi-Protocol Label Swapping. Provides label stack switching of IP packets in manner similar to that used in Frame Relay and ATM networks, but with the ability to work over virtually any link layer protocol (PPP/HDLC over SONET, Frame Relay, ATM, Ethernet, etc). OAM Operations, Administration, and Management. Sometimes specified OAMP, where the P represents Provisioning. OSI Open Systems Interconnect. An idealized model for representing the peer and interface interactions between communications protocols, arranged in a stack. Specified by the International Standards Organization (ISO). PHY A physical electrical or physical optical network interface component. PVST Per-VLAN Spanning Tree. A proprietary approach to implementing the Spanning Tree Protocol (STP) in multi-VLAN networks, wherein each VLAN runs its own instance of STP.

36

Volume 6: Leveraging Advanced Optical and Ethernet Technologies QoS Quality of Service. Refers to the mechanisms, specifications, and/or service-level agreements associated with providing end-to-end or node-to-node service guarantees or assurances on the basis of packet loss, delay, and/or jitter. RDI Remote Defect Indicator. A SONET fault isolation indicator. SONET Synchronous Optical Network. Specified as the Synchronous Digital Hierarchy (SDH) in the ITU Broadband Integrated Services Digital Network (B-ISDN) specification. It provides the basis for synchronous transport services in traditional telco carrier networks and is the underlying transport for both ATM and Frame Relay services, as well as DS1, DS3, OC3c, and higher speed point-to-point services. STP Spanning Tree Protocol. A data-link layer protocol for estabilishing a loop-free logical topology over an arbitrary interconnecting of data-link layer bridges. TDM Time Division Multiplexing. TCP Transport Control Protocol. A transport layer protocol providing reliable bulk data transfer over the Internet Protocol (IP). UDP User Datagram Protocol. A transport layer protocol providing message passing (datagram) capability over the Internet Protocol (IP). VLAN Virtual Local Area Network. A means of supporting multiple distinct bridging domains on a common Ethernet switching network. While developed for local area networks, the VLAN concept is used in metro Ethernet networks to provision distinct services, providing a virtual private network for each customer. WAN Wide Area Network.

37

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Acknowledgements
The authors would like to express their gratitude to Cisco Systems, Inc, and, in particular, Richard Shumaker and Scott Shepard, for their generous contribution of time, effort, and content to this report.

38

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

References
[1] Black, Ulyess and Waters, Sharleen. Sonet and T1: Architectures for Digital Transport Networks. Prentice Hall, 1997. Clark, Kennedy and Hamilton, Kevin. Cisco LAN Switching. Cisco Press, 2001 Goralski, Walter J. Introduction to ATM Networking. McGraw-Hill, 1995. Metro Ethernet Forum. Metro Ethernet Networks: A Technical Overview. 2002 Norton, William B. Interconnection Strategies for ISPs. Equinex, Inc, 1999. Perlman, Radia. Interconnections, 2nd Edition. Addison-Wesley, 2000. Spurgeon, Charles. Ethernet: The Definitive Guide. OReilly, 2000.

[2] [3] [4] [5] [6] [7]

39

Volume 6: Leveraging Advanced Optical and Ethernet Technologies

Quick Reference to Frequently Asked Questions

1) 2) 3) Why is it difficult for an established telecommunications company to make this investment? (Volume 1, Volume 5) There is already too much fiber in the ground. Why not use whats there? (Volume 1, Volume 2, Volume 6) The principal design criterion driving the development of this infrastructure is that every user has the potential to be a producer in the network economy. Is this the same as broadband, as it is currently hyped in the industry? (Volume 1) Can we quantify the potential jobs that will be created if a region invests in building advanced telecommunications infrastructure? (Volume 1) What should be the Tobacco Commissions role in the deployment of first mile technologies? (Volume 1, Volume 3, Volume 5, Volume 7, Volume 8) How can localities ensure that they get early access to the network? (Volume 1, Volume 5, Volume 8) What kind of success have other regions had with the development of network infrastructure for economic development? (Volume 1) What regulatory factors should be considered when investing in wireless technologies? (Volume 1, Volume 7) Why do we need to connect to network points outside of the tobacco regions? (Volume 2) Once the network is in place, what do we do with it? (Volume 2, Volume 8) Since the business model for inter-regional and inter-county infrastructure did not include the use of conduit facilitating blown fiber strands, what are the circumstances in which this technology is appropriate and financially feasible? (Volume 3, Volume 7) How do existing community networks fit into the overall design? (Volume 3, Volume 5, Volume 6) What are some examples for deployment in the first/last mile? (Volume 3, Volume 7) What type of fiber is recommended? (Volume 3) What would a network design for my county look like? (Volume 3)

4) 5) 6) 7) 8) 9) 10) 11)

12) 13) 14) 15)

40

Volume 6: Leveraging Advanced Optical and Ethernet Technologies 16) 17) How much would all this cost? (Volume 3, Volume 5) What is the appropriate organization model for managing and sustaining the Tobacco Commissions investment in critical technology infrastructure? (Volume 5) Tobacco region communities are underserved because the private sector does not see a profitable business case. What makes this feasible from a business perspective? (Volume 5) If the traditional investment model for developing critical technology infrastructure has failed, what is the alternative? (Volume 5) How much would it cost for consumers in the region to use the network? (Volume 5) What technologies enable use of the fiber? (Volume 6) How does the choice of technology to light the fiber impact the cost? (Volume 6) How do wireless technologies fit into this framework? (Volume 7) What is meant by the term open access? (Volume 8) What is the difference between the broadband hype and the next generation networks? (Volume 8) What are some next generation Internet (NGI) applications? (Volume 8)

18)

19) 20) 21) 22) 23) 24) 25) 26)

41