Professional Documents
Culture Documents
Introduction
Peer to Peer (P2P) technology has grown significantly over the past few years to where it now consumes the majority of service provider network capacity. Less of a problem within enterprise networks, P2P is beginning to grow within businesses as well, as corporate network users share content with others within and outside their corporate networks. While P2P has received much negative press coverage due to its penchant for consuming large amounts of network capacity, it is also an extremely efficient way to distribute content to many users, and for that reason its sustained growth remains a virtual certainty unless controlled. Given P2Ps widespread use in networks around the world and its habit of consuming as much network capacity as possible, the challenge is in managing it to where its utility can still be leveraged while also controlling how much network capacity it consumes. To address the growing volume and wide variety of network traffic and the inherently unpredictable nature of network usage in general, Anagran has developed an advanced Flow Rate Manager which can monitor and rate control all network traffic flows across up to four 10 Gbps Ethernet connections at line rate. A flow is simply an end-to-end network activity such as an image download, a voice call, a video stream, or a web page retrieval, which may in fact consist of many flows. Extremely compact and low power, the Anagran Flow Rate Manager is the first system that has perfected the ability to precisely control the rate of every TCP flow individually. Through continual, rapid measurement of the output rate on every port, VLAN, and class, the system can intelligently adjust the rates of every flow to achieve specified SLAs like the Committed Information Rate (CIR) of each of 4,000 classes different relative rates for each type of flow and class control the maximum rate on each VLAN, and port This paper will lay out the current and potential ways this rich control system can be used to minimize the total peak capacity required in the presence of a wide variety of traffic, including totally eliminating the problem with P2P traffic while maintaining fairness for all traffic types.
Application Areas
There are two areas significantly different area of application where minimizing the impact of P2P traffic is important: Local residential or school Internet access, close to the user Peering links, far from the user
Page 1 of 12
Close to the user, the traffic for all flows per address can be collected by the Anagran system and this information is used to control the relative rate of all flows to and from that address, thus adjusting the total capacity provided to each active address. This is very powerful since, given the fact that there is no true defense against P2P, all users get equal value for their money. Splitting file transfers into multiple flows gets the user no more bandwidth. Anagran will shortly have this technique available. However, this technique is not as valuable on peering links where each P2P user may have one or a few flows per link, and the number of addresses may be too high to keep track of. Here, it is more important to minimize the impact of any P2P flows on the peak required capacity, but it must be done flow-by-flow. In both areas there are several techniques that flow rate management can provide to reduce the peak required bandwidth in addition to just P2P control, and perhaps these should be described first. This is because all P2P control must be viewed in the context of how to solve the basic problem; reducing the peak bandwidth while maintaining excellent interactive response time and high quality voice and video. Although goals may vary, it will be assumed for the methods described in this paper that it is not important what traffic fills the otherwise unused capacity, only minimizing the peak capacity which the network must support.
Page 2 of 12
10 8 6 4 2 0 0 100 200
Load - Active Flow s
300
400
10
30
35
40
Figure 1: Laboratory test results with a 100 Mbps path with a RTT of 30 ms With this broad distribution of delays, the web page gets completed three times slower than when the Anagran flow manager is inserted into the path. This is because the Anagran system controls all similar flows to run at the about same rate, with a very narrow distribution of block transfer delay. Thus all blocks arrive at about the same time and the page completes three times faster with the same average and peak traffic. The difference is the flows are treated fairly, and to the end user, the web page retrieval is a much faster, real-time experience. The test results when the test traffic is being sent into a 100 Mbps router port is shown in Figure 1 when the router is alone (blue), and when the Anagran Flow Manager has rate-controlled the traffic before the router to just under 100 Mbps (red). With 40 segments per web page and only 400 active flows, the TCP traffic expands to fill the 100 Mbps and, as expected, is then controlled to about 92 Mbps. However, this is sufficient to cause the average web page to take 3 times as long as it would if the Anagran controlled all the flows to the same rate. Additional experiments showed that if the Anagran controlled the total peak rate to 50 Mbps, the web page response time would be the same as with the conventional router at 100 Mbps. The corollary to this can be observed on the right in Figure 1 where the response time is the same (6 seconds) when the load with Anagran is twice (400 flows) the original load (200 flows). These results demonstrate the power of equalizing the rates of interactive traffic since the peak capacity requirement for interactive traffic can be cut in half.
Page 3 of 12
With conventional network equipment using output buffer discard technology, the rate for transferring a file increases with the length of the file due to slow-start, until it reaches the maximum feasible rate for the TCP buffers and the RTT involved, typically 10-20 Mbps for Windows at modest distances. With Anagran, without any special prioritization rules, Intelligent Flow Discard (IFD) limits the rate of all flows in the same class to about the same rate, independent of the file size. See the example in Figure 2. Thus, without any special attention, the longer files are rate-constrained to fairness with the shorter flows. This insures the short gaming and web access flows grow as fast as possible without any discards. This is already an improvement over conventional operation since short flows are promoted and large flows dont get the majority of the capacity. Then by setting Behavioral Traffic Control (BTC) rules, the priority of the larger file transfers can be decreased slowly as the flow continues. The way this works is that rules are set that say If the flow has transferred more than X bytes set the priority down to Y. Flows with lower priorities get proportionally lower rates. This lets the network insure that Email and PDF files get through as fast as possible (without hurting the interactive) but the true bulk transfers like P2P get deferred until the smaller transfers do not need all the capacity - fill the valleys concept. The BTC rules used in Figure 2 were priority 1 for flows up to 100 KB and then used progressively lower priorities down to 0,1. The effect of any of these strategies is dependent on the momentary mix of traffic. When there is lots of traffic of all sizes, the high priority traffic takes over and the interactive uses the channel. When there is little interactive, the rates of the larger file transfers are increased. Anagran always attempts to keep the channel full. This case is shown in Figure 3 and as a result the rates of large file transfers have been increased since there is no interactive at the moment.
Rate for Block Transfers
Equal short and long size traffic 10 Packet IFD BTC
10
Packet IFD BTC
Mbps
Mbps
0.1
1 100 10,000 Block Size KB 1,000,000
Figure 2
Figure 3
delay and as we discussed above, it also causes the interactive rates to spread which reduces the response time for Web access. When Anagran is added into the path, using IFD and a slight BTC reduction of priority for bulk traffic, the channel capacity can be reduced by 20%.
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 Unused NRT (Bulk) Interactive
Without Anagran
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 8 15 22 29 36
With Anagran
Recovered Capacity
Unused NRT (Bulk) Interactive
43
50
57
64
71
78
85
92
99
Figure 4: 20% Channel Capacity Supporting the Same Traffic As one can see in Figure 4, the bulk traffic has been slowed down when the interactive traffic peaked and sped up during slack periods so as to fill the valleys. Also, the Web access time has been improved.
Without Anagran
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 8 15 22 29 36
With Anagran
Recovered Capacity
Unused NRT (Bulk) Interactive
43
50
57
64
71
78
85
92
99
Figure 5: Reduced Interactive peaks 25% and Capacity Reduced 33% In Figure 5 the interactive traffic averages 23% and the bulk 23% in both graphs. The Interactive peaks have been reduced from 78% to 58% whereas instead of staying at 23% all the time, the bulk has been reduced to 4% when the interactive peaks and increased to 60% when there is spare capacity. This way the total capacity has been reduced by 33% and the unused capacity reduced 33% from 54% to 21%.
Page 5 of 12
Without Anagran
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 8 15 22 29 36
With Anagran
Recovered Capacity
Unused NRT (Bulk) Interactive
43
50
57
64
71
78
85
92
99
T im e (6 seco nd i ntervals)
Page 6 of 12
caused by P2P programs which use multiple flows to obtain more capacity than other users. TCP allows many forms of unfairness. Simply: 1- Unfairness with distance long delay slows TCP 2- Unfairness due to random discards some stall, others thrive 3- Unfairness with multiple flows the more flows, the more capacity consumed Anagran can fix all three of these unfairness issues. 1- Anagran assigns the same rate to all similar flows, independent of distance 2- Anagran eliminates unfairness with discards since all flows receive the same rate 3- Multi-flow unfairness is the one that most P2P capitalizes on today. A race is ongoing between the P2P developers and the Deep Packet Inspection (DPI) systems to try and outsmart each other. However, unless this basic unfairness is fixed, the end game is clear. P2P can and will win against DPI by encrypting everything and making the flows look identical to some interactive traffic. This is still in the future since making flows as short as Web accesses is inefficient and making them as slow as voice is also inefficient but with enough flows, they will eventually become undetectable as P2P, first with DPI and later with BTC. Therefore, a new technique is required where all the traffic from any source address or destination address is measured and used to detect P2P and to control the total rate that user is allowed. This is the next subject.
Rate Equalization
Although this is a simple concept, implementation is much more difficult. It is certainty possible to keep in memory the byte and flow count for 300,000 users which might be using a 10 Gbps link. However, the addresses must be looked up and bytes counted - perhaps every packet. Then there needs to be a smooth and effective way to slow traffic for that user, hopefully one which does not stall or kill the flow. However, this is quite easy in the context of a flow manager where most of these capabilities already exist. Anagrans Flow manager achieves its compact size and power efficiency by streaming packets through and working mainly at the next higher level, flows. The P2P problem with multiple flows is primarily observable at the next higher level, the traffic and flows to or from one user, typically identified by one IP address. Since byte counts for every flow already are kept, it becomes easy to accumulate the byte count per address as well. These are then averaged over a suitable period like 15 minutes (configurable) and used to adjust a priority for that user. The higher the 15 minute traffic rate, the lower the priority. This priority is then combined with the other priorities and used to
Page 7 of 12
control all the flow rates to that user. The result is that all users on one common resource like a cable, DSLAM, Internet access link, or wireless base station receive the same average capacity. Of course, the basic policy of equality can be modified by additional policies and priorities such as the payment level and other factors. Also, since there needs to be an averaging period to account for normal traffic bursts, the effect will be slightly delayed. However, this does not create a loophole; a user cannot get around the impact if they want to download a video. Thus, this new technique, Rate Equalization per Address, provides a major new capability to control P2P and address the inherent unfairness issue. Given the attractiveness of any unfairness, it is likely that this TCP unfairness will be more widely used in the future and it is important to address it now. At least then in the future, software developers will optimize their systems compatibly with the communication system and not capitalize on an unfairness that costs someone else money.
Page 8 of 12
Each Anagran Flow Manager can collect flow records on any specified criteria, and send these to a central computer using Netflow. Then the central computer can sort through all the flows from each user at any given time and find those users who are using multiple flows at the same time. This is now the same process as outlined for the local area. The probable P2P users can then be identified and further processing will lead to solid identification of the P2P users. Then, the central system can send a list of these P2P users IP addresses to the Anagran flow managers on each peering link. Now, when a new flow arrives on the peering link, the Anagran would look up the address in the table which was forwarded about the detected P2P users. New flows that these users send, that trigger a set of BTC rules would then be subject to the lower priority (and therefore lower rate) that the central site had determined. Repeat offenders would get lower and lower priorities. By putting these flows into a P2P class the total P2P can also be limited. Thus, the system would operate much like the local system except that the central system could do even more analysis. The more remaining part is to determine the best criteria and to make the system as fair as possible. Clearly, a users priority should grow back with time and be proportional to their relative consumption of the peering links. All of this is in the hands of the central system and most likely managed by the network operator.
Page 9 of 12
can find the likely P2P data flows. Unless already done before the Netflow records were generated, we can find the qualifying flows using the following: First insure that there are at least 2 flows at > 25 Kbps each, operating at once Flows with a byte count >10 MB and an average packet size > 1000 bytes Flows not to/from known web sites, rather from discrete addresses Flows to/from know other P2P users (over time this will be a known list) Exclude flows that match known voice or video conferencing characteristics Now we have a set of probable P2P flows to/from a potential P2P user. Next, it is valuable to explore the users behavior profile. For each of the following a weight can be assigned: The number of probable P2P flows per day in & out The total bytes in & out per day from probable P2P flows If the number of different addresses per day contacted by a user is >X If the % of flows from a user to not-known-servers out of all flows is >Y% If the users activity level on the peering links is >Z% of the day If the user has probable P2P flows during all four 6 hour periods of the day The weight can be added up for all these criteria (and many others) and the sum will easily rank the user as a heavy, moderate, light, or unclear P2P user. As data is collected, these criteria, the thresholds, and the weights can be refined. As the list of known external P2P sources develops from the ones the certain P2P users are contacting develops, this address list will also improve the ease of recognition greatly. This process will clearly identify the larger P2P users quickly and accurately. Their addresses and a priority based on the heaviness of their use can be returned to all the Anagran flow managers continuously as the processing proceeds. In fact, the feedback would soon be fast enough that the ongoing P2P flows could be re-prioritized. The process would only take seconds to detect a new user once the criteria and known addresses were developed over the first few days of operation. The false positives from this process would be near zero given the range of criteria and evidence developed since to even become a probable P2P user requires multiple large flows be operating at the same time. Very few activities have this property, and those that do are clearly are similar to P2P and need to be controlled. Not many actual P2P cases would be missed either since any user who sets up more than one download at a time over the peering links would be caught. A priority is then assigned based on this count of P2P periods. The users address and the priority is then returned to all the flow managers on the peering links. The central site would also need to separate residential IP addresses from corporate IP addresses. For corporations, different policies would need to be used. However, this technique would not only be quite feasible but quite effective. Even international users could be listed if they were often getting P2P from the country operating the system.
Page 10 of 12
typical web access. The lower priority would then be applied to the new flow and the flow allowed to proceed. As the flow proceeded, if it triggered a BTC command that was intended to lower the priority of P2P ad bulk traffic, the priority might be further reduced. This would mean the short flows of the P2P user would be slower and the long flows would be even slower. An additional option would be to put the P2P users into a P2P class and set a maximum rate for the whole class. This would limit all P2P traffic to any level desired.
Conclusion
Anagrans Products have strong capabilities to apply priorities to the rate of each flow to automatically adjust the mix of traffic to optimize the bandwidth consumed and ensure quality. The key functions involved are: Equal rate per flow decreases the time per web access to 1/3 or allows the peak rate for interactive to be cut in half with the same response time Intelligent Flow Discard (IFD) stops larger flows from getting even higher rates, making a first-order correction on P2P unfairness Tiered Behavioral Traffic Control (BTC) commands can apply lower and lower priorities to a flow as it grows in size Flow Priority is a major new network capability where a low priority flow gets capacity only to fill the higher-priority valleys Applying these basic capabilities to a typical mix of traffic allows anywhere from 20% to 50% reduction of the peak rate which means either more traffic on current trunks, or less capacity required for the same amount of traffic. When P2P is present these functions allow the P2P traffic to be placed at a low priority so that it does not disturb other time-critical traffic or, if desired, to also be capacity-limited. However, in addition, two new functions are being added which totally change how P2P can be managed to permit a one-time total solution to P2P unfairness:
Page 11 of 12
Rate Equalization per Address allows traffic to be distributed equally to all users (addresses), independent of the number or type of flows they are using, eliminating the unfairness of P2P. This totally eliminates the need for DPI in controlling P2P. P2P User Identification allows the P2P users to be very cleanly identified through the total user activity and put into a P2P class. Then their P2P activity can be identified by flow and managed by priority and if desired, the total P2P class could be limited. There is also a proposed offline function to detect P2P offenders on peering links. This couples back into the flow priority function so that once an offender is detected; flows to/from the address will have lower priority, so a repeating offender would get lower and lower rate service. The P2P identification here work much the same as in the local case, but uses a central system to find P2P users across all peering links. Altogether, these features can save substantial capacity, insure voice and video quality, improve response time, and eliminate the P2P problem fairly and cleanly. With Anagran, P2P traffic can finally peacefully co-exist with all other traffic on any dynamic, high capacity, multi-service network.
*** END ***
Filling the Network Performance Gap To achieve required network performance with todays rich, dynamic traffic mix, the gap between Layers 3 and 4 must be addressed. Operating at Layers 3 and 4 to complement routers and WOCs, the FR-1000 optimally grooms the entire traffic mix to protect the network from unplanned traffic surges and all traffic types from each other.
About Anagran Anagrans Fast Flow Technology (FFT) protects networks, routers, and links from overload while also ensuring required performance and quality for all traffic types. Anagrans products eliminate the potentially crippling effects of traffic congestion to enable video, voice, data and wireless traffic to co-exist and grow with flawless quality over converged IP networks. Anagrans FR-1000 fills the performance gap between routers and WAN optimization techniques to consistently deliver the highest levels of guaranteed performance and quality, while significantly reducing overall and recurring network costs and energy consumption. Anagran was founded by Internet pioneer Dr. Lawrence Roberts, recognized as one of the founding fathers of the Internet. Information about Anagran can be found at www.anagran.com.
Content is subject to change without notice. For more information call: +1 650-298-9029 or email: info@anagran.com Anagran, Fast Flow Technology, Intelligent Flow Discard, and Behavioral Traffic Control are trademarks of Anagran, Inc. www.anagran.com Copyright 2005-2008 by Anagran, Inc. Page 12 of 12