Professional Documents
Culture Documents
Martin Brown (mc@mcslp.com) Professional writer Freelance Ken Milberg Writer/site expert Future Tech This three-part series on AIX 7 networking focuses on the challenges of optimizing network performance. With AIX 7 and the new POWER7 architecture, you have to think about the performance of your system, and also the effects of your network performance on both LPAR and WPAR environments. In the first installment, you'll review some best practices for network configuration and utilize those practices for efficient monitoring of your hardware by taking advantage of network tools such as netstat, netpmon, entstat, and nmon. You'll also examine certain tools used to look at your interfaces to see how to configure them. View more content in this series 23 November 2010
Introduction
The first thing that usually comes to mind when a system administrator hears that there might be some network contention issues is to run netstat. netstat, the equivalent of using vmstat or iostat for your memory reports, is a quick way to get an overview of how your network is configured. Unlike vmstat or iostat, the defaults usually do not give you as much information as you probably would like. You need to understand the correct usage of netstat and how best to utilize it when monitoring your system.
Copyright IBM Corporation 2010 Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware Trademarks Page 1 of 17
developerWorks
ibm.com/developerWorks/
is really not a monitoring tool in the sense of vmstat and iostat. You can use other tools more suitable (discussed later in the article) to help monitor your network subsystem. At the same time, you can't really start to monitor unless you have a thorough understanding of the various components related to network performance. These components include your network adapters, your switches and routers, and how you are using virtualization on your host logical partitions. If you determine you are indeed having a network bottleneck, fixing the problem might actually lay outside of your immediate host machine. There is little you can do if the network switch is improperly configured on the other end. Of course, you might be able to point the network team in the right direction. You should also spend time gathering overall information about your network. How are you going to be able to understand how to troubleshoot your network devices unless you really understand your network? In this article, you'll look at specific AIX network tracing tools, such as netpmon, and how they can help you isolate your bottlenecks.
netstat
Finally, no matter which subsystem you are looking to tune, you must think of systems tuning as an ongoing process. As stated before, the best time to start monitoring your systems is at the beginning, before you have any problems and users aren't screaming. You must have a baseline of network performance so that you know what the system looks like when it is behaving normally. Finally, when making changes, be careful to make changes only one at a time so that you can really assess the impact of your change.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 2 of 17
ibm.com/developerWorks/
developerWorks
Figure 1 illustrates that there is more to network monitoring than running netstat and looking for collisions. From the application layer through the media layer, there are areas that need to be configured, monitored, and tuned. At this point, you should notice some similarities between this illustration and the Open Systems Interconnection Basic Reference Model (OSI Model). The OSI Model has seven layers (bottom to top): Physical Data-link Network Transport Session Presentation Application
Perhaps the most important concept to understand is that on the host machine each layer communicates with its corresponding layer on the remote machine. The actual application programs transmit data using either User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) transport layer protocols. They receive the data from whatever application you are using and divide them into packets. The packets themselves differ, depending on whether it is a UDP or TCP packet. Generally speaking, UDP is faster, while TCP is more secure. There are many tunable parameters to look atyou'll get to these parameters during subsequent phases of the series. You might want to start to familiarize yourself with the no command, which is the utility designed to make the majority of your network changes. From a hardware perspective, it is critical that you understand the components that need to be configured appropriately to optimize
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware Page 3 of 17
developerWorks
ibm.com/developerWorks/
performance. Though you might work together with the network teams that manage your switches and routers, it is unlikely that you will be configuring them (unless you are a small shop or a oneperson IT department). The most important component you will be working with is your network adapter. Gigabit Ethernet is now the standard network interface, although 10Gbit network cards and infrastructure is becoming more accessible.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 4 of 17
ibm.com/developerWorks/
developerWorks
Where is the jumbo frames option? In this case, you cannot make the change. The reason for this is because you are only using the Virtual I/O Ethernet on this systemthis topic is discussed in more detail later. On a system where you have direct access to the network hardware, check the system using Listing 2.
Media speed
Most modern network switches and environments can take advantage of auto-negotiation to provide the best speed, especially as full-duplex network switches have become the standard. However, you force a particular configuration if the auto-negotiation fails to configure a speed that you know your network switch is capable of supporting. The lsattr command gives you the information that you need. The en prefix displays your driver parameters, while the ent prefix displays your hardware parameters. Let's display your hardware parameters (see Listing 3).
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 5 of 17
developerWorks
ibm.com/developerWorks/
In this case, your interface is set as auto-negotiate. You should also check your firmware levels to make sure they are up to date. We've seen many network problems fixed when updating to the latest levels of firmware. The lscfg command gives you the firmware information (see Listing 4).
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 6 of 17
ibm.com/developerWorks/
developerWorks
Product Specific.(B7).......000000000000000000000000 Product Specific.(B1).......00215EEB4680002000215EEB46A00020 Version.....................ipzSeries Physical Location: U78C0.001.DBJ3226-P2-C8 4 X 1GB ETHERNET: Record Name.................VINI Flag Field..................XXET Hardware Location Code......U78C0.001.DBJ3227-P2-C8 Customer Card ID Number.....2BC4 Serial Number...............YL10D9360028 CCIN Extender...............1 Product Specific.(VZ).......04 FRU Number..................46K5965 Part Number.................46K6484 Product Specific.(HE).......0001 Product Specific.(CT).......30910008 Product Specific.(HW).......0001 Product Specific.(B3).......000000000001 Product Specific.(B4).......00 Product Specific.(B7).......000000000000000000000000 Product Specific.(B1).......00215EEB4140002000215EEB41600020 Version.....................ipzSeries Physical Location: U78C0.001.DBJ3227-P2-C8
See the Resources section at the end of the article for a link to the most current release information for your adapter. Though the series focuses on tuning in subsequent parts, you might want to start to familiarize yourself with the memory management facility of network subsystems. What you need to know at this point is that it relates to data structures called mbufs. These are used to store kernel data for incoming and outbound traffic. The buffer sizes themselves can range from 32 to 16384 bytes. They are created by making allocation requests to the Virtual Memory Manager (VMM). In an SMP box, each memory pool is split evenly for every processor. The monitoring section below shows you how to view mbufs. An important concept to note is that processors cannot borrow from the memory pool outside of its own processor. Two other concepts you should be familiar with are virtual Ethernet and shared Ethernet. Virtual Ethernet: Virtual Ethernet allows for inter-partition- and IP-based communications between logical partitions on the same frame. This is done by the use of a virtual I/O switch. The Ethernet adapters themselves are created and configured using the HMC. If you recall, you tried to change an adapter earlier that was configured with virtual Ethernet. Shared Ethernet: Shared Ethernet allows for the use of Virtual I/O servers (VIOs), where several host machines can actually share one physical network adapter. Typically, this is used in environments that do not require substantial network bandwidth. While the scope of this series is not on virtualization, you should understand that if you are using virtualization, there might be other reasons for your bottleneck outside of what you are doing on your host machine. While virtualization is a wonderful thing, be careful not to share too many adapters from your VIO server, or you might pay a large network I/O penalty. Keep in mind as well that with workload partitions (WPAR), the network configuration of the host environment is used and shared by each partition; you cannot configure WPAR network performance individually. Using appropriate monitoring tools should inform you if you have a problem.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware Page 7 of 17
developerWorks
ibm.com/developerWorks/
In addition, application performance can be affected fundamental services, such as domain name service (DNS) used to resolve hostnames and Internet addresses.
Monitoring
This section provides an overview of general network monitoring commands and specific AIX 7 tools available to you. Some of the tools allow you to troubleshoot a performance problem quickly while others capture data for historical trending and analysis. Let's get back to the old standby, netstat, which displays overall network statistics. Probably one of the most common commands you type in is netstat -in (see Listing 5).
Here is what it means: Name: Interface name. MTU: Interface Maximum Transfer Unit size. Network: The actual network address that the interface connects to. Address: Mac and IP address. Ipkts: The total amount of packets received by the interface. Ierrs: The amount of errors reported back from the interface. Opkts: The amount of packets transmitted from the interface. Oerrs: The amount of error packets transmitted from the interface. Coll: The amount of collisions on the adapter. If you are using Ethernet, you won't see anything here.
Another handy netstat flag is the -m option. This flag allows you to view the Kernel malloc statistics; the mbuf memory requests, including the size of the buffers, the amount in use and the failures by CPU (see Listing 6).
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 8 of 17
ibm.com/developerWorks/
developerWorks
6 256 55 117 4
0 0 0 0 0
3 62 23 76 0
163 0 4 0 102
0 0 0 0 0
******* CPU 1 ******* By size inuse 64 46 128 58 256 51 512 78 1024 62 2048 23 4096 1 8192 1 16384 0 32768 2 65536 0 131072 0 ******* CPU 2 ******* By size inuse 64 21 128 9 256 9 512 81 1024 27 2048 5 4096 0 8192 0 32768 0 131072 0 ******* CPU 3 ******* By size inuse 64 42 128 15 256 12 512 83 1024 32 2048 23 4096 1 8192 0 16384 0 32768 2 65536 0 131072 0
calls failed 226765 0 152657 0 70035 0 46458768 0 171426 0 3669503 0 891 0 567 0 12 0 17 0 11 0 1 0
delayed 0 2 2 4 12 5 4 3 5 3 6 0
hiwat 5240 2620 5240 6550 2620 3930 1310 327 163 81 81 40
freed 0 0 0 0 0 0 0 0 0 0 0 0
delayed 1 0 0 10 15 0 1 1 1 0
free 43 23 39 55 33 23 5 0 1 16
freed 0 0 0 0 0 0 0 0 0 0
hiwat 5240 2620 5240 6550 2620 3930 1310 327 163 81 81 204
freed 0 0 0 0 0 0 0 0 0 0 0 0
The -m option is particularly useful because it shows the network performance statistics in relation to individual CPUs. When monitoring and managing the performance with LPAR and WPAR environments, the ability to correlate your CPU resources and the network resources can give you valuable information about the correct allocation and distribution of network resources. For Ethernet, you can use the entstat command to display device-driver statistics. This provides a plethora of information (see Listing 7).
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 9 of 17
developerWorks
------------------------------------------------------------ETHERNET STATISTICS (ent0) : Device Type: Virtual I/O Ethernet Adapter (l-lan) Hardware Address: 66:da:93:d1:6b:17 Elapsed Time: 16 days 1 hours 48 minutes 7 seconds Transmit Statistics: -------------------Packets: 79636 Bytes: 15868037 Interrupts: 0 Transmit Errors: 0 Packets Dropped: 0 Max Packets on S/W Transmit Queue: 0 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 0 Broadcast Packets: 5 Multicast Packets: 6171 No Carrier Sense: 0 DMA Underrun: 0 Lost CTS Errors: 0 Max Collision Errors: 0 Late Collision Errors: 0 Deferred: 0 SQE Test: 0 Timeout Errors: 0 Single Collision Count: 0 Multiple Collision Count: 0 Current HW Transmit Queue Length: 0 General Statistics: ------------------No mbuf Errors: 0 Adapter Reset Count: 0 Adapter Data Rate: 20000 Driver Flags: Up Broadcast Running Simplex 64BitSupport ChecksumOffload DataRateSet Virtual I/O Ethernet Adapter (l-lan) Specific Statistics: --------------------------------------------------------RQ Length: 4481 Trunk Adapter: False Filter MCast Mode: False Filters: 255 Enabled: 1 Queued: 0 Overflow: 0 LAN State: Operational Hypervisor Send Failures: 0 Receiver Failures: 0 Send Errors: 0 Hypervisor Receive Failures: 0 Invalid VLAN ID Packets: 0 ILLAN Attributes: 0000000000003002 [0000000000003002] Port VLAN ID: VLAN Tag IDs: 2 None Broadcast Packets: 203036730 Multicast Packets: 215 CRC Errors: 0 DMA Overrun: 0 Alignment Errors: 0 No Resource Errors: 0 Receive Collision Errors: 0 Packet Too Short Errors: 0 Packet Too Long Errors: 0 Packets Discarded by Adapter: 0 Receiver Start Count: 0 Receive Statistics: ------------------Packets: 203054741 Bytes: 30905882351 Interrupts: 201058047 Receive Errors: 0 Packets Dropped: 0 Bad Packets: 0
ibm.com/developerWorks/
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 10 of 17
ibm.com/developerWorks/
developerWorks
Total (KB) I/O Memory VRM Minimum (KB) VRM Desired (KB) DMA Max Min (KB) Transmit Information Transmit Buffers Buffer Size Buffers History No Buffers Virtual Memory
65536 32 0
Collisions are largely a thing of the past with modern network switches, but look for transmit errors and make sure they are not increasing too fast. You need to learn to troubleshoot collision and error problems before you even begin to think about tuning. Alternatively, you can use netstat -v, which provides similar information. Let's look at netpmon. netpmon provides information on CPU usage as it relates to the network, and it also includes data about the network device driver I/O, Internet socket calls, and other various statistics. Similar to its other trace brethren, tprof and filemon, it starts a trace and runs in the background until you stop it with the trcstop command. We like netpmon because it really gives you a detailed overview of network activity and also captures data for trending and analysis (though it is not as useful as nmon for this purpose). Here you'll use a trace buffer size of two million bytes (see Listing 8).
Let's look at the data. Here is just a small sampling of the output (see Listing 10).
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 11 of 17
developerWorks
Network Process (top 20) PID CPU Time CPU % CPU % ---------------------------------------------------------netpmon 12976354 41.7559 24.072 0.000 netpmon 14155800 32.1352 18.526 0.000 inetd 14155804 29.3093 16.897 0.000 xmtopas 14155834 29.2385 16.856 0.000 xmtopas 14155830 29.2381 16.856 0.000 xmtopas 14155822 28.9899 16.713 0.000 xmtopas 14155816 28.9433 16.686 0.000 xmtopas 14155826 28.9390 16.683 0.000 xmtopas 14155828 28.9308 16.679 0.000 inetd 14155824 28.9287 16.677 0.000 xmtopas 14155820 28.9227 16.674 0.000 xmtopas 14155814 28.9158 16.670 0.000 inetd 14155866 23.7606 13.698 0.000 inetd 14155864 23.7131 13.671 0.000 xmtopas 14155846 19.0223 10.966 0.000 inetd 14155848 19.0083 10.958 0.000 inetd 9371742 18.8547 10.870 0.000 inetd 9371658 16.5665 9.551 0.000 inetd 9371886 16.4571 9.488 0.000 xmtopas 9371888 16.4094 9.460 0.000 ---------------------------------------------------------Total (all processes) 838.3315 483.300 0.000 Idle time 78.8376 45.450 ======================================================================== First Level Interrupt Handler CPU Usage Statistics: --------------------------------------------------Network FLIH CPU Time CPU % CPU % ---------------------------------------------------------data page fault 255.0988 147.065 0.000 UNKNOWN 17.9673 10.358 0.000 PPC decrementer 2.7282 1.573 0.000 external device 0.0081 0.005 0.000 instruction page fault 0.0002 0.000 0.000 queued interrupt 0.0000 0.000 0.000 ---------------------------------------------------------Total (all FLIHs) 275.8026 159.001 0.000 ======================================================================== TCP Socket Call Statistics (by Process): --------------------------------------------- Read --------- Write ----Process (top 20) PID Calls/s Bytes/s Calls/s Bytes/s -----------------------------------------------------------------------sshd: 5636334 0.12 1888 0.53 37 nonstop_aix 3539136 0.05 9 0.05 0 java 4260046 0.05 9 0.05 0 -----------------------------------------------------------------------Total (all processes) 0.21 1907 0.62 37 ======================================================================== Detailed TCP Socket Call Statistics (by Process): ------------------------------------------------PROCESS: sshd: reads: PID: 5636334 5
ibm.com/developerWorks/
As you can see, there is little overall network I/O activity going on during this time. The top section is most important, as it really helps you get an understanding of what processes are consuming
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware Page 12 of 17
ibm.com/developerWorks/
developerWorks
network I/O time, and you can use the more detailed per-process output to obtain more specific information. lsattr (used earlier to view the hardware parameters) is another command you will be using frequently to display statistics on your interfaces. The attributes that you see here are configured using either the chdev or no commands. Let's display your driver parameters (see Listing 11).
If you've been following the other series on AIX 7 (see Resources), you know we love nmon and you should also, once you start using it. With nmon (type in n after startup), you have a quick snapshot of everything going on in your network, including adapter details, MTU, error counters and collisions, and megabit rating. Further, you also have the ability to capture data with nmon. Using the nmon analyzer, you can print out graphical reports directly from Microsoft Excel spreadsheets. See Resources for a link to an IBM Wiki for the nmon manual or for downloads.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 13 of 17
developerWorks
ibm.com/developerWorks/
Summary
This article covered the relative importance of the network I/O subsystem, and defined the AIX 7 network I/O layers and how it relates to the OSI Model. You learned some best practices for network configuration, changed your Ethernet settings to support jumbo frame, and viewed interface hardware and driver data. You even examined the monitoring tools available to you and captured data using netpmon and nmon. In the next part of the series, you'll tune NFS, find out more about monitoring utilities, such as nfsstat and nmon, and discover how to tune with nfso.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 14 of 17
ibm.com/developerWorks/
developerWorks
Resources
Learn AIX Wiki: Get the nmon manual or downloads here. Improving database performance with AIX concurrent I/O: Read this white paper for more information on how to improve database performance. Power Architecture: High-Performance Architecture with a History: Read this white paper. "Power to the People; A history of chip making at IBM" (developerWorks, December 2005): This article covers the IBM power architecture. "CPU Monitoring and Tuning" (March, 2002): Learn how standard AIX tools can help you determine CPU bottlenecks. nmon analyser -- A free tool to produce AIX performance reports (Steven Atkins, developerWorks, April 2006): You can download nmon analyser from here. nmon performance: A free tool to analyze AIX and Linux performance (Nigel Griffiths, developerWorks, February 2006): Read this article for information on nmon. Operating System and Device Management from IBM provides users and system administrators with complete information that can affect your selection of options when performing such tasks as backing up and restoring the system, managing physical and logical storage, and sizing appropriate paging space. The AIX 7.1 Information Center is your source for technical information about the AIX operating system. The IBM AIX Version 6.1 Differences Guide can be a useful resource for understanding changes in AIX 6.1. The IBM AIX Version 7.1 Differences Guide can be a useful resource for understanding changes in AIX 7.1. Popular content: See what AIX and UNIX content your peers find interesting. AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills. New to AIX and UNIX?: Visit the New to AIX and UNIX page to learn more about AIX and UNIX. AIX Wiki: Discover a collaborative environment for technical information related to AIX. Safari bookstore: Visit this e-reference library to find specific technical resources. developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts. Podcasts: Tune in and catch up with IBM technical experts. Future Tech: Visit Future Tech's site to learn more about their latest offerings. Get products and technologies IBM trial software: Build your next development project with software for download directly from developerWorks. Discuss Follow developerWorks on Twitter.
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware Page 15 of 17
developerWorks
ibm.com/developerWorks/
Participate in developerWorks blogs and get involved in the developerWorks community. Get involved in the My developerWorks community. Participate in the AIX and UNIX forums: AIX Forum AIX Forum for developers Cluster Systems Management Performance Tools Forum Virtualization Forum More AIX and UNIX Forums
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 16 of 17
ibm.com/developerWorks/
developerWorks
Ken Milberg Ken Milberg is a technology writer and site expert for techtarget.com and provides Linux technical information and support at searchopensource.com. He is also a writer and technical editor for IBM Systems Magazine, Open Edition. Ken holds a bachelor's degree in computer and information science and a master's degree in technology management from the University of Maryland. He is the founder and group leader of the NY Metro POWER-AIX/Linux Users Group. Through the years, he has worked for both large and small organizations and has held diverse positions from CIO to Senior AIX Engineer. Today, he works for Future Tech, a Long Islandbased IBM Business Partner. Ken is a PMI certified Project Management Professional (PMP), an IBM Certified Advanced Technical Expert (CATE, IBM System p5 2006), and a Solaris Certified Network Administrator (SCNA). You can contact him at kmilberg@gmail.com. Copyright IBM Corporation 2010 (www.ibm.com/legal/copytrade.shtml) Trademarks (www.ibm.com/developerworks/ibm/trademarks/)
Optimizing AIX 7 network performance: Part 1, Network overview - Monitoring the hardware
Page 17 of 17