Ace Ts Guide

Cisco Application Control Engine (ACE) Troubleshooting Guide Welcome to Cisco DocWiki. We encourage registered Cisco.
com users to contribute to this wiki to collaborate on Cisco product documentation. You do not need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments about documentation content. See Terms of Use and About DocWiki for more information about Cisco DocWiki. Click here to go to the Cisco ACE Module documentation on www.cisco.com. Click here to go to the Cisco ACE Appliance documentation on www.cisco.com.
Contents
1 Audience 2 Organization 3 Creating a PDF of the ACE Troubleshooting Wiki 4 Related Documentation 4.1 ACE Module Documentation 4.2 ACE Appliance Documentation
This article provides a systematic approach to identifying and remedying problems that may arise as you use your ACE over a period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that you may encounter.
Audience
This article is intended for all trained network administrators who have experience with the configuration and maintenance of the ACE.
Organization
This article consists of the following major sections: Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting Ethernet Ports (ACE appliance) Troubleshooting Remote Access Contents 1
Cisco Application Control Engine (ACE) Troubleshooting Guide Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing Resources Show Counter Reference
Creating a PDF of the ACE Troubleshooting Wiki

You can create a PDF of one or more articles in this wiki, including the entire ACE Troubleshooting Wiki. For details, see the Creating a PDF article.
Related Documentation
ACE Module Documentation
Customer Documentation for the Cisco Application Control Engine (ACE) Module Cisco Application Control Engine (ACE) Configuration Examples on DocWiki
ACE Appliance Documentation

Hardware Installation Guides for the Cisco ACE 4710 Appliance Release Notes for the Cisco 4700 Series Application Control Engine Appliance Command Reference for the Cisco 4700 Series Application Control Engine Appliance Configuration Guides for the Cisco 4700 Series Application Control Engine Appliance Cisco CSS-to-ACE Conversion Tool User Guide Cisco Application Control Engine (ACE) Configuration Examples on DocWiki This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems that may occur when you configure and use your ACE. Organization 2
Cisco Application Control Engine (ACE) Troubleshooting Guide Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of the ACE Troubleshooting Process 2 Verifying the ACE Image 3 Enabling ACE Logging 4 Gathering ACE Troubleshooting Information 4.1 Rebooting the ACE 4.2 Using show Commands 4.3 Capturing Packets in Real Time 4.4 Copying Core Dumps 4.5 After Gathering Troubleshooting Information 5 Verifying the Physical Connectivity Between the ACE and the End Hosts 6 Verifying the ACE Layer 2 Connectivity 7 Verifying the ACE Layer 3 Connectivity 8 Contacting Cisco Technical Support
ACE Appliance Documentation
Cisco Application Control Engine (ACE) Troubleshooting Guide
Overview of the ACE Troubleshooting Process

To troubleshoot your ACE, follow these general guidelines: 1. Maintain a consistent and recommended software version across all your ACEs. See the "Verifying the ACE Image" section. 2. See the ACE module release notes for your software version for the latest features, operating considerations, caveats, and CLI command changes. 3. Before you introduce configuration changes, use the ACE checkpoint feature to bookmark a known good configuration and save your configuration. If you run into problems with the new configuration, you can roll back the new configuration to the known good configuration. See the Cisco Application Control Engine Module Administration Guide. Troubleshoot new configuration changes immediately after adding them. 4. Verify that your configuration is correct for your network application. Make any required changes to the running-config file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy running-config startup-config command for a particular virtual context or the write memory command from the Admin context to copy all running-config files in every virtual context to their respective startup-config files. 5. Enable system message logging. See the "Enabling ACE Logging" section. 6. Gather information that defines the specific symptoms. See the "Gathering ACE Troubleshooting Information" section. 7. Verify the physical connectivity between your device and end devices. See the "Verifying the Physical Connectivity Between the ACE and the End Hosts" section. 8. Verify the ACE Layer 2 connectivity. See the "Verifying the ACE Layer 2 Connectivity" section. 9. Verify the ACE end-to-end connectivity and the routing configuration. See the "Verifying the ACE Layer 3 Connectivity" section. 10. After you have determined that your troubleshooting attempts have not resolved the problem, contact the Cisco Technical Assistance Center (TAC) or your technical support representative. See the "Contacting Cisco Technical Support" section.
Verifying the ACE Image

To display the version of the software image and the image filename that is currently running in your ACE, enter the following command:
ACE_module5/Admin# show version Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. Software loader: Version system: Version system image file: installed license:
12.2[121] A2(2.0) [build 3.0(0)A2(2.0)] <-------[LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <-------no feature license is installed
Hardware Cisco ACE (slot: 5) cpu info: number of cpu(s): 2 cpu type: SiByte cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz memory info: total: 955396 kB, free: 289704 kB shared: 0 kB, buffers: 2336 kB, cached 0 kB

cf info: filesystem: /dev/cf total: 1000000 kB, used: 494912 kB, available: 505088 kB last boot reason: NP 1 Failed : NP ME Hung
configuration register: 0x1 ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)
This command provides other useful information, for example: Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5) Available control plane memory Last boot reason Configuration register (confreg) value (0x0 boot to rommon, 0x1 boot using boot string) ACE uptime
Enabling ACE Logging

To enable logging on the ACE module and to send system logging (syslog) messages to the monitor, enter the following commands:
ACE_module5/Admin(config)# logging enable ACE_module5/Admin(config)# logging monitor 7 ACE_module5/Admin(config)# exit ACE_module5/Admin# terminal monitor
Note: Use the terminal no monitor command to stop viewing log messages in your remote session. For more information about logging, see the "Troubleshooting with ACE Logging" section.
Gathering ACE Troubleshooting Information

The following sections recommend ways to gather information that is relevant to the problem that is occurring.
Rebooting the ACE

Do not reboot the ACE unless it is absolutely necessary. Some information that is important to troubleshooting your problem may not survive a reboot. Try to gather as much information as possible before rebooting.
Using show Commands

You can use a number of show commands in Exec mode to gather information specific to the symptoms you are observing in your ACE. In most cases, you can gather the information you need to troubleshoot the ACE by entering the show tech-support command. This command runs many show commands that are useful for troubleshooting the ACE. You can redirect the output of the show tech-support command to one the following destinations:
ACE_module5/Admin# show tech-support > ? <File> Name of file to redirect stdout. disk0: Enter the URI to redirect the output. ftp: Enter the URI to redirect the output. sftp: Enter the URI to redirect the output. tftp: Enter the URI to redirect the output. volatile: Enter the URI to redirect the output.
Capturing Packets in Real Time

Capturing packets (sometimes referred to as a "TCP dump") is a useful aid in troubleshooting connectivity problems with the ACE or for monitoring suspicious activity. The ACE can track packet information for network traffic that passes through the ACE. The attributes of the packet are defined by an ACL. The ACE buffers the captured packets, and you can copy the buffered contents to a file in Flash memory on the ACE or to a remote server. You can also display the captured packet information on your console or terminal. The ACE captures packets subject to the following guidelines: One capture session is used per context Capture is triggered at flow setup Capture is configured on the client interface where the flow is received Note: Probe traffic will not hit a security ACL, so ACLs cannot control the capture of those packets. Therefore, probe traffic cannot be captured by the packet capture utility. If possible, you should capture packets using the ACE packet capturing utility before and after symptoms appear. Save the packet captures to a file. To capture packets in real time, follow these steps: 1. Create an ACL for packet capturing or use an existing ACL if it meets the packet capture requirements by entering the following command:
ACE_module5/Admin(config)# access-list FILTER line 10 extended permit tcp any any eq www ACE_module5/Admin# exit
2. Enter the capture command, for example:

ACE_module5/Admin# capture CAPTURE1 interface vlan 200 access-list FILTER
Note: Ensure that the ACL you specify in the capture command is for an input interface. If you configure the packet capture on the output interface, the ACE will fail to match any packets. 3. Display the capture status to determine the capture status and the buffer size by entering the following command:
ACE_module5/Admin# show capture CAPTURE1 status Capture session Buffer size Circular Buffer usage Status : : : : : TEST 64 K no 0.00% stopped
Notice that the capture has not started yet. The default buffer size is 64 KB. You can specify a maximum of buffer size of 5000 KB and you can specify a circular buffer. 4. Start the packet capture on the ACE by entering the following command:
ACE_module5/Admin# capture CAPTURE1 start 11:56:15.354930 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 62: 209.165.201.10.4144 > 172.16.1.100.80: S [bad tcp cksum 2ae 11:56:15.355257 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 62: 209.165.201.10.4144 > 192.168.1.11.80: S [bad tcp cksum c6 11:56:15.355669 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 58: 192.168.1.11.80 > 209.165.201.10.4144: S [tcp sum ok] 118 11:56:15.355979 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 58: 172.16.1.100.80 > 209.165.201.10.4144: S [bad tcp cksum 641 11:56:15.356442 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 1 11:56:15.356839 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9b 11:56:15.357203 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 494: 209.165.201.10.4144 > 172.16.1.100.80: P [tcp sum ok] 1:44 11:56:15.357918 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 494: 209.165.201.10.4144 > 192.168.1.11.80: P [bad tcp cksum 9

11:56:15.358436 11:56:15.358582 11:56:15.358822 11:56:15.359106 11:56:15.359391 11:56:15.359751 11:56:15.360101 11:56:15.360238 11:56:15.360378 11:56:15.360523 11:56:15.360686 11:56:15.360831 11:56:15.360973 11:56:15.361130 11:56:15.361290 11:56:15.361436
0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 272: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 1: 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 272: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 64 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 407: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 21 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 407: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 64 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: F [tcp sum ok] 572 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: F [bad tcp cksum 641 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 5 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9b 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 5 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9b 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: F [tcp sum ok] 441:4 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: F [bad tcp cksum 9b 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641
ACE_module5/Admin# capture CAPTURE1 stop
5. Copy the packet capture to disk0: by entering the following command:

ACE_module5/Admin# copy capture CAPTURE1 disk0:CAPTURE1
You can also copy the packet capture to an FTP, SFTP, or TFTP server. 6. Display the messages and connections within a packet capture by entering the following command:
ACE_module5/Admin# show capture CAPTURE1 0001: msg_type: ACE_HIT ace_id: 637 0002: msg_type: CON_SETUP con_id: 1308623156 0003: msg_type: PKT_RCV con_id: 1308623156 0004: msg_type: PKT_XMT con_id: 167772463 0005: msg_type: PKT_RCV con_id: 167772463 0006: msg_type: PKT_XMT con_id: 1308623156 <snip> 0025: msg_type: PKT_RCV con_id: 167772463 0026: msg_type: PKT_XMT con_id: 1308623156 0027: msg_type: CON_CLOSE con_id: 167772463 0028: msg_type: CON_CLOSE con_id: 1308623156 action_flag: out_con_id: other_con_id: other_con_id: other_con_id: other_con_id: other_con_id: other_con_id: reason: reason: 0x3 167772463 0 0 0 0 0 0 0 0
7. Display the details of each packet within a capture by entering the following command:
ACE_module5/Admin# show capture CAPTURE1 detail 0001: msg_type: ACE_HIT ace_id: 637 action_flag: 0x3 src_addr: 209.165.201.10 src_port: 4144 dst_addr: 172.16.1.100 dst_port: 80 l3_protocol: 0 l4_protocol: 6 message_hex_dump: 0x0000: 0006 0104 0000 027d 0000 0000 d1a5 c90a .......}........ 0x0010: ac10 0164 0609 0013 1030 0050 0000 0000 ...d.....0.P.... 0x0020: 0052 0000 05b4 0000 0000 027d 0300 0000 .R.........}.... 0x0030: 0000 0040 0000 0000 0000 0000 0000 0000 ...@............ 0x0040: 0000 0000 0000 0001 ........ 0002: msg_type: CON_SETUP con_id: 1308623156 out_con_id: 167772463 src_addr: 209.165.201.10 src_port: 4144 dst_addr: 172.16.1.100 dst_port: 80

l3_protocol: 0 message_hex_dump: 0x0000: 0006 0101 0x0010: d1a5 c90a 0x0020: e5b3 f25e 0x0030: 0000 0000 0x0040: 0000 0030 0x0050: 0000 0000 0x0060: 0000 0000 0x0070: 0000 0000 0x0080: c0a8 010b 0x0090: 1a4c 0da2 0x00a0: 0000 0000 0x00b0: 0000 0000 0x00c0: 4a54 f427 0x00d0: 0000 0000 0x00e0: 0000 0000 l4_protocol: 6 4e00 ac10 0012 0018 faf0 e1ad e1ad 0000 d1a5 0055 0018 0000 e1ad e1ad 0000 0134 0164 0000 0480 0010 6b69 6b69 0000 c90a 0000 0480 0000 6b6b 6b6b 0000 0a00 06e9 05b4 2445 05b4 0000 0000 0000 06a1 05b4 2445 05b4 0000 0000 0000 012f 0013 0100 0000 0000 0000 0000 0000 0018 0100 0022 0000 0000 0000 0000 0000 1030 0a00 0000 3008 0000 0000 0000 0050 4e00 0000 0000 0000 0000 0000 0050 012f 0001 e685 027d 0000 0000 1030 0134 0000 0000 027d 0000 ....N..4.../.... .......d.....0.P ...^.........../ ........$E...... ...0........0... ......ki.......} ......ki........ ................ .............P.0 .L...U......N..4 ........$E.".... ................ JT.'..kk.......} ......kk........ ............
0003: msg_type: PKT_RCV con_id: 1308623156 message_hex_dump: 0x0000: 0500 0050 0050 8034 0x0010: 0020 000b fcfe 1b01 0x0020: 4500 0030 0933 4000 0x0030: ac10 0164 1030 0050 0x0040: 7002 faf0 18fd 0000 0004: msg_type: PKT_XMT con_id: 167772463 message_hex_dump: 0x0000: 4010 0050 0050 8034 0x0010: 0004 000c 29f3 cde6 0x0020: 4500 0030 0933 4000 0x0030: c0a8 010b 1030 0050 0x0040: 7002 faf0 18fd 0000 0005: msg_type: PKT_RCV con_id: 167772463 message_hex_dump: 0x0000: 0500 004c 0050 8034 0x0010: 0020 0018 b9a6 890d 0x0020: 4500 002c 0000 4000 0x0030: d1a5 c90a 0050 1030 0x0040: 6012 16d0 6df5 0000 0006: msg_type: PKT_XMT con_id: 1308623156 message_hex_dump: 0x0000: 4010 004c 0050 8034 0x0010: 0004 001c f909 1800 0x0020: 4500 002c 0000 4000 0x0030: d1a5 c90a 0050 1030 0x0040: 6012 16d0 6df5 0000 0007: msg_type: PKT_RCV con_id: 1308623156 message_hex_dump: 0x0000: 0500 004a 0050 8034 0x0010: 0020 000b fcfe 1b01 0x0020: 4500 0028 0934 4000 0x0030: ac10 0164 1030 0050 0x0040: 5010 faf0 05ad 0000 0008: msg_type: PKT_XMT con_id: 167772463 message_hex_dump:
other_con_id: 0 0008 001c 7f06 3008 0204 0014 f909 aa70 e684 05b4 0010 1800 d1a5 e5b3 0101 1488 0800 c90a f25e 0101 ...P.P.4........ ................ E..0.3@....p.... ...d.0.P0......^ p...............
other_con_id: 0 0000 000b 7f06 4a54 0204 0028 fcfe aa70 f426 05b4 0000 1b01 d1a5 0000 0101 0088 0800 c90a 0000 0101 @..P.P.4...(.... ....)........... E..0.3@....p.... .....0.PJT.&.... p...............
other_con_id: 0 0008 000c 4006 46ca 0204 0028 29f3 de68 2127 05b4 0010 cde6 c0a8 4a54 2888 0800 010b f427 ...L.P.4...(..(. ..........)..... E..,..@.@..h.... .....P.0F.!'JT.' `...m.......
other_con_id: 0 0000 000b 4006 2c7e 0204 0014 fcfe de68 1385 05b4 0000 1b01 ac10 3008 0088 0800 0164 e685 @..L.P.4........ ................ E..,..@.@..h...d .....P.0,~..0... `...m.......
other_con_id: 0 0008 001c 7f06 3008 0000 0014 f909 aa77 e685 0010 1800 d1a5 2c7e 1488 0800 c90a 1386 ...J.P.4........ ................ E..(.4@....w.... ...d.0.P0...,~.. P.........
other_con_id: 0

0x0000: 0x0010: 0x0020: 0x0030: 0x0040: 4010 0004 4500 c0a8 5010 004a 000c 0028 010b faf0 0050 29f3 0934 1030 05ad 8034 cde6 4000 0050 0000 0000 000b 7f06 4a54 0000 0028 fcfe aa77 f427 0000 1b01 d1a5 46ca 0088 0800 c90a 2128 @..J.P.4...(.... ....)........... E..(.4@....w.... .....0.PJT.'F.!( P.........
0009: msg_type: PKT_RCV con_id: 1308623156 message_hex_dump: 0x0000: 0500 0200 0050 8034 0x0010: 0020 000b fcfe 1b01 0x0020: 4500 01e0 0935 4000 0x0030: ac10 0164 1030 0050 0x0040: 5018 faf0 a0bb 0000 0x0050: 6c6c 2e68 746d 6c20 0x0060: 0d0a 486f 7374 3a20 0x0070: 2e31 3030 0d0a 5573 0010: msg_type: PKT_XMT con_id: 167772463 message_hex_dump: 0x0000: 4010 0200 0050 8034 0x0010: 0004 000c 29f3 cde6 0x0020: 4500 01e0 0935 4000 0x0030: c0a8 010b 1030 0050 0x0040: 5018 faf0 a0bb 0000 0x0050: 6c6c 2e68 746d 6c20 0x0060: 0d0a 486f 7374 3a20 0x0070: 2e31 3030 0d0a 5573 <snip>
other_con_id: 0 0008 001c 7f06 3008 4745 4854 3137 6572 0014 f909 a8be e685 5420 5450 322e 2d41 0010 1800 d1a5 2c7e 2f73 2f31 3136 6765 1488 0800 c90a 1386 6d61 2e31 2e31 6e74 .....P.4........ ................ E....5@......... ...d.0.P0...,~.. P.......GET./sma ll.html.HTTP/1.1 ..Host:.172.16.1 .100..User-Agent
other_con_id: 0 0000 000b 7f06 4a54 4745 4854 3137 6572 0028 fcfe a8be f427 5420 5450 322e 2d41 0000 1b01 d1a5 46ca 2f73 2f31 3136 6765 0088 0800 c90a 2128 6d61 2e31 2e31 6e74 @....P.4...(.... ....)........... E....5@......... .....0.PJT.'F.!( P.......GET./sma ll.html.HTTP/1.1 ..Host:.172.16.1 .100..User-Agent
Note: If you view the ACE capture file in a third-party sniffer (for example, Wireshark), you will notice only the messages or type PKT_RCV and PKT_XMT are displayed. This situation is expected because the sniffer is not aware of the ACE's internal messaging.
Copying Core Dumps

If the ACE fails with a core dump, the core dump files may contain useful information. The core dump files reside in the core: directory. To view the contents of the core: directory, enter the following command:
ACE_module5/Admin# dir core: 123589 30361 Feb 22 00:34:20 2009 qnx_1_mecore_log.999.tar.gz Feb 22 00:34:22 2009 ixp1_crash.txt Usage for core: filesystem 153950 bytes total used 202943138 bytes free 203097088 total bytes
You can copy the contents of the core: directory to several locations by using the copy core: command. The syntax of this command is as follows: copy {core:filename | disk0:[path/]filename | running-config | startup-config} {ftp://server/path[/filename] | sftp://[username@]server/path[/filename] | tftp://server[:port]/path[/filename]} The ACE provides core dumps for both the control plane and the data plane. Each core dump file contains the following information: Version Time of failure Copying Core Dumps 9
Cisco Application Control Engine (ACE) Troubleshooting Guide Number of CPUs Current CPU BKL status IRQ lock status Buffers
After Gathering Troubleshooting Information

After you have gathered all the above information, be prepared to send the information to your customer service representative or TAC. You can send the information in the following ways: FTP SFTP TFTP
Verifying the Physical Connectivity Between the ACE and the End Hosts
To verify the physical connectivity of the ACE, follow these steps: 1. Check all cable connections on the Catalyst 6500 series switch or Cisco 7600 series router that may impact the ACE. 2. Use the extended ping command to send an ICMP Echo request to the end devices.
ACE_module5/Admin# ping Target IP address: 10.1.1.2 Repeat count [5]: 4 Datagram size [100]: 200 Timeout in seconds [2]: 10 Extended commands [n]: 4 Pinging 10.1.1.2 with timeout = 10, count = 4, size = 200 .... Response Response Response Response 4 packet from 10.1.1.2 : seq 1 time from 10.1.1.2 : seq 2 time from 10.1.1.2 : seq 3 time from 10.1.1.2 : seq 4 time sent, 4 responses received, 0.494 ms 0.367 ms 0.264 ms 0.237 ms 0% packet loss
If a host is one hop away and you are unable to reach the host, then ping the intermediary gateway. If the gateway is not reachable, enter the show ip route command and check to make sure that the correct route is displayed. For example, enter:
ACE_module5/Admin# show ip route Routing Table for Context Admin (RouteId 0) Codes: H - host, I - interface S - static, N - nat A - need arp resolve,
E - ecmp
Destination Gateway Interface Flags -----------------------------------------------------------------------0.0.0.0 10.2.2.1 vlan130 S [0xc] 10.2.2.0/24 0.0.0.0 vlan130 IA [0x30] 172.27.15.0/24 0.0.0.0 vlan100 IA [0x30] 172.27.16.0/24 0.0.0.0 vlan200 IA [0x30] 172.19.110.0/26 0.0.0.0 vlan55 IA [0x30] 172.27.16.16/29 0.0.0.0 vlan200 N [0x280] 172.27.16.33/32 0.0.0.0 vlan100 N [0x280]

Total route entries = 7
If necessary, enter a static route for the gateway.
Verifying the ACE Layer 2 Connectivity

To verify the Layer 2 connectivity of the ACE, follow these steps: 1. Verify that the ARP table is populated with the IP addresses and corresponding MAC addresses of the ACE, the gateway, the local interface, and other IPs that the ACE has learned.
switch/Admin# show arp Context Admin ================================================================================ IP ADDRESS MAC-ADDRESS Interface Type Encap NextArp(s) Status ================================================================================ 10.86.215.208 00.02.7e.39.51.9c vlan130 LEARNED 5 2350 sec up 10.86.215.228 00.e0.81.22.78.ff vlan130 LEARNED 6 6379 sec up 10.86.215.234 00.1a.a1.48.f3.44 vlan130 LEARNED 4 1114 sec up 10.86.215.1 00.00.0c.07.ac.00 vlan130 GATEWAY 2 153 sec up 10.86.215.2 00.11.5d.e1.2f.fc vlan130 LEARNED 3 12054 sec up 10.86.215.134 00.18.b9.a6.91.15 vlan130 INTERFACE LOCAL _ up ================================================================================ Total arp entries 6
2. Verify that the ACE is connected to the switch fabric of the Catalyst 6500 series switch or the Cisco 7600 series router. The ACE uses a 10-Gigabit Ethernet switch fabric interface (SFI) to connect to the chassis backplane as opposed to the CSM, which uses a port channel. The ACE uses the following format for this interface: Te<slot>/1 For example, if the ACE is in slot 5, you can see the status of the backplane connection by entering the following command on the Catalyst 6500 series switch or the Cisco 7600 series router:
cat6k# show interface te5/1 status Port Te5/1 Name Status connected Vlan trunk Duplex Speed Type full 10G MultiService Module
If there is no output from this command, then either the ACE is not installed properly or the ACE is powered down. 3. Verify the association of the ACE MAC entries with the allocated VLAN interfaces. Enter the following command at the Supervisor CLI:
cat6k# show Legend: * age n/a mac-address-table dynamic primary entry - seconds since last seen - not available
vlan mac address type learn age ports ------+----------------+--------+-----+----------+-------------------------. . . * 130 0018.b9a6.9115 dynamic Yes 40 Te5/1 <------- MAC address should be in the range displayed by . . .
Verifying the Physical Connectivity Between the ACE and the EndHosts
11

cat6k# show module 5 Mod Ports Card Type Model Serial No. --- ----- -------------------------------------- ------------------ ----------5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S Mod MAC addresses Hw Fw Sw Status --- ---------------------------------- ------ ------------ ------------ ------5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- MAC address range assigned to Mod Online Diag Status ---- ------------------5 Pass
4. Check the status of the Te5/1 port to ensure that it is in the forwarding state by entering the following command:
cat6k# show spanning-tree vlan 130 MST0 Spanning tree enabled protocol mstp Root ID Priority 32768 Address 0001.632f.2c17 Cost 200019 Port 642 (GigabitEthernet6/2) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority Address Hello Time Role ---Desg Desg Desg Desg Root Desg Sts --FWD FWD FWD FWD FWD FWD 32768 (priority 32768 sys-id-ext 0) 0011.bc06.f800 2 sec Max Age 20 sec Forward Delay 15 sec Cost --------20000 200000 2000 2000 200000 2000 Prio.Nbr -------128.142 128.165 128.257 128.513 128.642 128.897 Type -------------------------------P2p P2p Edge P2p Edge P2p Shr Bound(STP) Edge P2p
Interface ---------------Gi2/14 Gi2/37 Te3/1 Te5/1 Gi6/2 Te8/1

Use the traceroute command to check the route between the ACE and the end devices.
ACE_module5/Admin# traceroute 10.20.12.153 traceroute to 10.20.12.153 (10.20.12.153), 30 hops max, 40 byte packets 1 10.20.215.2 (10.20.215.2) 0.532 ms 0.436 ms 0.362 ms 2 10.20.239.161 (10.20.239.161) 0.421 ms 0.488 ms 0.404 ms 3 10.20.238.93 (10.20.238.93) 0.471 ms 0.422 ms 0.413 ms 4 172.27.16.177 (172.27.16.177) 0.488 ms 0.435 ms 0.430 ms 5 172.27.16.226 (172.27.16.226) 0.474 ms 0.363 ms 0.368 ms 6 192.168.0.134 (192.168.0.134) 0.624 ms 0.510 ms 0.494 ms 7 10.20.12.153 (10.20.12.153) 23.982 ms 24.702 ms 25.976 ms
Contacting Cisco Technical Support

If you are unable to resolve a problem after using the troubleshooting suggestions in the articles in this wiki, contact the Cisco Technical Assistance Center (TAC) for assistance and further instructions. Before you call, have the following information ready to help your TAC engineer assist you as quickly as possible:
12
Cisco Application Control Engine (ACE) Troubleshooting Guide Date that you received the ACE Chassis serial number (located on a label on the right side of the rear panel of the chassis) Type of software and release number (if possible, enter the show version command) Maintenance agreement or warranty information Brief description of the problem Brief explanation of the steps that you have already taken to isolate and resolve the problem For information on steps to take before calling Technical Support, see the "Gathering ACE Troubleshooting Information" section. You can reach TAC in several ways as follows: Create a service request online Call the TAC at the telephone numbers on this page. Contact the Cisco Small Business Support Center
This article describes the ACE architecture and how data flows into, gets processed, and flows out of the ACE. It provides a basic understanding of these concepts to assist you in troubleshooting the ACE.
Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Understanding the ACE Architecture 1.1 Overview of the ACE Hardware Architecture 1.2 Control Plane 13
Contacting Cisco Technical Support
Cisco Application Control Engine (ACE) Troubleshooting Guide 1.3 Data Plane 1.3.1 Classification and Distribution Engine 1.3.2 Network Processors 1.3.3 SSL Crypto Module 2 Understanding the ACE Traffic Flow 2.1 To-the-ACE Traffic 2.2 Through-the-ACE Traffic
Understanding the ACE Architecture

Having a basic understanding of the ACE architecture and data flow can help to make troubleshooting the ACE easier. This section describes the major functional areas of the ACE and how they work together.
Overview of the ACE Hardware Architecture

The ACE hardware architecture is divided into a series of functional areas or subsystems that are defined by processors or groups of processors and interfaces as shown in Figure 1.
Figure 1. ACE Module Architecture
Contents
14
A console connection allows direct access to the ACE control plane (CP) for initial configuration, management, and troubleshooting. The supervisor engine connection allows you to determine the status of the ACE, to load images into the ACE, to reboot the ACE, and to provide remote access to the ACE from the Catalyst 6500 series switch or Cisco 7600 series router when you use the session command. Because the ACE has no external ports, packets enter the ACE through the Switch Fabric Interface (SFI) connected to the Catalyst 6500 series switch or Cisco 7600 series router back plane. The two major functional areas of the ACE are as follows: Control plane Data plane
Control Plane
The control plane (CP) is used to configure the ACE and for management traffic, syslogs, ARP, DHCP, and so on. You can access the CP directly by using the console port. For remote management, you must configure a management interface and enable remote access using a management policy to permit Telnet or SSH access, for example. The CP is responsible for the following ACE functions: Device management and control Configuration management (CLI or XML interface) Server health monitoring syslogs SNMP Address Resolution Protocol (ARP) DHCP relay Redundancy (also known as high availability or fault tolerance) Access control list (ACL) compilation
Overview of the ACE Hardware Architecture
15
Data Plane
The data plane (DP) is responsible for distributing and processing packets and connections that do not match a management policy. In the ACE, the CP and the DP are separated and run on different processors for maximum performance. See Figure 2.
Figure 2. ACE Data Plane
The DP is responsible for the following ACE functions: Access control lists (ACLs) Connection management TCP termination Network address translation (NAT) SSL processing (termination, initiation, encryption, and decryption) Regular expression matching Load balancing and forwarding Application protocol inspection The DP consists of the following functional areas: Classification distribution engine (CDE) Data Plane 16
Cisco Application Control Engine (ACE) Troubleshooting Guide Network processors (NPs) SSL Crypto Module Daughter card interfaces (for future feature expansion) Classification and Distribution Engine The Classification and Distribution Engine (CDE) is the traffic controller for the ACE. Its main purpose is to forward packets that it receives from the SFI to the two network processors (NPs). It also acts as the central point of contact among all the major subsystems within the ACE. The CDE computes, and if necessary, adjusts the IP, TCP, and UDP checksums of every packet that it receives. The CDE appends a special header known as the IMPH header to each packet before sending it to the fast path. The IMPH header is 18 bytes long and contains information from the DBUS header (the header sent to the ACE Module by the Catalyst 6500 series switch or Cisco 7600 series router) as well as special messaging directly understood by the fast path. Fields in the IMPH header can include notification of a checksum error, Layer 3 or Layer 4 offsets, source and destination ports of the CDE, the VLAN for determining the interface that the fast path will use, and so on. Network Processors The ACE has two network processors (NP1 and NP2) that perform most of the packet processing in the ACE. All traffic entering the ACE must traverse one or both NPs after being forwarded by the CDE. Each NP contains a CPU (XScale) and several components called microengines (MEs). See Figure 3.
Figure 3. Network Processor Micorengines
Each microengine can handle eight simultaneous threads or processes and performs a specific function for the NP as follows: Receive - One ME for receiving incoming packets
Cisco Application Control Engine (ACE) Troubleshooting Guide Fast Path - Four MEs for the hardware accelerated data path that is used for MAC rewrite, NAT, TCP normalization, and so on (essentially all operations performed on a per-packet basis) ICM - One ME for the inbound connection manager OCM - One ME for the outbound connection manager CCM - One ME for the connection close manager TCP - Two MEs for TCP termination with a full TCP stack HTTP - Two MEs for HTTP parsing Unused (future expansion) - One ME SSL Record Layer - One ME for the SSL record layer IP fragmentation timers - One ME for IP fragmentation reassembly and timer management DNS and ICMP Inspection - One ME for DNS and ICMP packet inspection The XScale microprocessor is programmed to handle the following features: Load-balancing algorithms SSL handshake FTP and Real-Time Streaming Protocol (RTSP) inspection HTTP inspection (although a considerable part is performed by the microengines) High-availability heartbeat generation Returned statistics for most connection-related commands Each network processor has RDRAM memory to store ACL entries, routing table entries, ARP entries, and inspection policies. Additional SRAM memory provides faster access times and is used to store regular expressions and statistics on a per?virtual system basis, among other things. SSL Crypto Module The SSL Crypto Module is responsible for SSL record layer processing. This processing includes encrypting and decrypting data for SSL flows.
Understanding the ACE Traffic Flow

Because the ACE has no native ports, it relies on the switch fabric in the Catalyst 6500 series switch or the Cisco 7600 series router back plane to send and receive packets to and from the network. Packets that are marked with a destination VLAN and Layer 2 information enter the ACE through the SFI on the 10 Gbps Ethernet link. Packets entering or leaving the ACE traverse this link using VLAN tagging. The switch fabric interface (SFI) forwards to the CDE all packets that are destined to the ACE. See Figure 1. The CDE classifies the packets based on the configured traffic policies, fills out the IMPH header information, and forwards the traffic to one of the NPs. To determine which NP to forward the traffic to, the CDE hashes incoming packets based on the traffic type as follows: TCP/UDP - Hash of source/destination port Non-TCP/UDP IP - Hash of source/destination IP address Non-IP - Hash of source/destination Layer 2 MAC address The NPs process the traffic and forward it back through the CDE to either the control plane or the SSL Crypto Module for further processing.
To-the-ACE Traffic
To-the-ACE traffic is traffic that is destined to an interface VLAN IP address on the ACE. This traffic must match a class map of type management, which is associated with a policy map and applied as a service policy on an interface VLAN. The management
Cisco Application Control Engine (ACE) Troubleshooting Guide class map supports the following protocols: Hypertext Transfer Protocol (HTTP) Hypertext Transfer Protocol Secure (HTTPS) Internet Control Message Protocol (ICMP) Keepalive Application Protocol (KAL-AP) User Datagram Protocol (UDP) Simple Network Management Protocol (SNMP) Secure Shell (SSH) Protocol Telnet This management traffic is called control plane traffic because it is destined to the CP. Because of the separation of the CP traffic from the data plane traffic on different processors, the control plane traffic will never interfere with data plane traffic, even if the control plane is oversubscribed.
Through-the-ACE Traffic
The CDE sends traffic that requires load balancing, forwarding, routing, or other processing by the ACE to one of the NPs. The NPs comprise two parallel forwarding paths that maintain their own connection state information and forward traffic independently.
This article describes some basic troubleshooting steps that you can perform to rule out some of the simpler issues before delving deeper into the troubleshooting process.
Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
To-the-ACE Traffic
19
Contents
1 Preliminary ACE Troubleshooting Steps 2 Checking the ACE Status from the Supervisor Engine 3 Verifying the MSFC VLAN Configuration 4 Establishing a Session with the ACE from the Supervisor Engine 5 Verifying the ACE is Receiving VLAN Allocations from the MSFC 6 Verifying the ACE Image 7 Verifying Your ACE Licenses 8 Configuring an ACL to Permit Input Traffic to the ACE 9 Verifying that the ACE is Sending and Receiving Traffic 10 Verifying to-the-ACE Traffic
Preliminary ACE Troubleshooting Steps

1. Check the status of the ACE module from the Catalyst 6500 series switch or Cisco 7600 series router. See the "Checking the ACE Status from the Supervisor Engine" section. 2. Verify that you have allocated the correct VLANs to the ACE in the Multilayer Switch Feature Card (MSFC) VLAN configuration. See the "Verifying the MSFC VLAN Configuration" section. 3. Verify that you can establish a session to the ACE from the supervisor engine. See the "Establishing a Session with the ACE from the Supervisor Engine" section. 4. Verify that the ACE is receiving VLAN allocations from the MSFC. See the Verifying the ACE is Receiving VLAN Allocations from the MSFC" section. 5. Verify your ACE bandwidth, SSL, and virtualization licenses. See the "Verifying Your ACE Licenses" section. 6. Verify that you have configured an access control list (ACL) to permit traffic on the interfaces on which you wish the ACE to receive traffic. If you do not configure an ACL to permit traffic on an interface, all traffic destined to that interface will be blocked by the ACE. See the "Configuring an ACL to Permit Input Traffic to the ACE" section. 7. Verify that the ACE is sending and receiving traffic. See the "Verifying that the ACE is Sending and Receiving Traffic" section. 8. Verify the management traffic to the control plane. See the "Verifying to-the-ACE Traffic"section.
Checking the ACE Status from the Supervisor Engine

Before you begin to troubleshoot your ACE, Telnet to the Catalyst 6500 series switch or Cisco 7600 series router supervisor engine, log in, and check the status of the ACE. Contents 20

telnet 10.1.1.2 User Access Verification Password: cat6k> enable Password: cat6k# show module Mod Ports Card Type Model Serial No. --- ----- -------------------------------------- ------------------ ----------. . 5 1 Application Control Engine Module ACE20-6500-K9 SAD1031044S . . Mod MAC addresses Hw Fw Sw Status --- ---------------------------------- ------ ------------ ------------ ------. . 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- ACE status . .
Mod Sub-Module Model Serial Hw Status --- --------------------------- ------------------ ----------- ------- ------6 Policy Feature Card 3 WS-F6K-PFC3A SAL09094NUB 2.5 Ok 6 MSFC3 Daughterboard WS-SUP720 SAL09094N33 2.5 Ok
Mod Online Diag Status --- ------------------. . 5 Pass . .
Verifying the MSFC VLAN Configuration
To verify that the VLANs that you intend to use in your ACE have been configured and allocated to the ACE in the MSFC, follow these steps: 1. Check the VLANs configured and allocated to the ACE by entering the following command from the supervisor engine:
cat6k# show run | include svclc
svclc module 5 vlan-group 123,130,133
2. Ensure that the VLAN groups that you intend to use for your ACE are allocated properly in the MSFC configuration by entering the following commands:
cat6k# show svclc module 5 vlan-group Module Vlan-groups ------ ----------05 123,130,133
cat6k# show svclc vlan-group Display vlan-groups created by both ACE module and FWSM commands
Checking the ACE Status from the Supervisor Engine
21

Group ----123 130 133 Created by ---------ACE ACE ACE vlans ----103,105,107,111-112,119,134,160,171,200,203,205,207,211-212,226,253,260 130 100,194,221,256-257
3. Verify that the VLANs you intend to use in your ACE are configured in the MSFC by entering the following command:
cat6k# show interface te5/1 trunk Port Te5/1 Port Te5/1 Port Te5/1 Port Te5/1 Mode on Encapsulation 802.1q Status trunking Native vlan 1
Vlans allowed on trunk 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257,260 Vlans allowed and active in management domain 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257 Vlans in spanning tree forwarding state and not pruned 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257
4. Ensure that traffic is routed to two ACEs in the same chassis when both client- and server-side VLANs are configured as switched virtual interfaces (SVIs) on the MSFC in routed mode by entering the following command:
cat6k# show svclc multiple-vlan-interfaces Multiple ACE vlan interfaces feature is enabled
Establishing a Session with the ACE from the Supervisor Engine

To verify that you can establish a session with the ACE from the supervisor engine in the Catalyst 65000 series switch or the Cisco 7600 series router, enter the following command:
cat6k# session slot 5 processor 0 The default escape character is Ctrl-^, then x. You can also type 'exit' at the remote prompt to end the session Trying 127.0.0.50 ... Open ACE_module5 login:
Verifying the ACE is Receiving VLAN Allocations from the MSFC

Ensure that the VLAN that you intend to use on ACE are allocated properly in the MSFC configuration by entering the following command:
ACE-1/Admin# show vlans Vlans configured on SUP for this module vlan123 vlan130 vlan133
If interface VLANs are already assigned on the ACE you can use the show interface vlan <num> command to verify the interface is properly assigned on the MSFC and up on the MSFC:
Verifying the MSFC VLAN Configuration
22

ACE-1/Admin# show interface vlan 123 vlan10 is up Hardware type is VLAN MAC address is 00:18:b9:a6:89:0d Mode : routed IP address is 10.10.10.1 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 7101679 unicast packets input, 878043707 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 6387914 unicast packets output, 1541924399 bytes 0 multicast, 22826 broadcast 0 output errors, 0 ignored

To display the version of the software image and the image filename that is currently running in your ACE, enter the following command:
ACE_module5/Admin# show version Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. Software loader: Version system: Version system image file: installed license:
12.2[121] A2(2.0) [build 3.0(0)A2(2.0)] <-------[LCP] disk0:c6ace-t1k9-mzg.A2_2_0.bin <-------no feature license is installed
Hardware Cisco ACE (slot: 5) cpu info: number of cpu(s): 2 cpu type: SiByte cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz memory info: total: 955396 kB, free: 289704 kB shared: 0 kB, buffers: 2336 kB, cached 0 kB cf info: filesystem: /dev/cf total: 1000000 kB, used: 494912 kB, available: 505088 kB last boot reason: NP 1 Failed : NP ME Hung
configuration register: 0x1 ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)
This command provides other useful information, for example: Verifying the ACE is Receiving VLAN Allocations from the MSFC 23
Cisco Application Control Engine (ACE) Troubleshooting Guide Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5) Available memory Last boot reason Configuration register (confreg) value ACE uptime
Verifying Your ACE Licenses

Log in to your ACE and enter the following command to display the SSL, virtualization, and bandwidth licenses that are currently installed and in use in your ACE:
ACE_module5/Admin# show license usage License Ins Lic Status Expiry Date Comments Count -------------------------------------------------------------------------------ACE-08G-LIC No Unused ACE-16G-LIC No Unused ACE-UPG1-LIC No Unused ACE-UPG2-LIC No Unused ACE-VIRT-020 No Unused ACE-VIRT-050 No Unused ACE-VIRT-100 No Unused ACE-VIRT-250 Yes 1 In use never ACE-VIRT-UP1 No Unused ACE-VIRT-UP2 No Unused ACE-VIRT-UP3 No Unused ACE10-16G-LIC No Unused ACE-SEC-LIC-K9 No Unused ACE-SSL-05K-K9 No Unused ACE-SSL-10K-K9 No Unused ACE-SSL-15K-K9 No Unused ACE-SSL-20K-K9 No Unused ACE-SSL-UP1-K9 No Unused ACE-SSL-UP2-K9 No Unused ACE-SSL-UP3-K9 No Unused -
ACE_module5/Admin# show license status Licensed Feature Count ---------------------------------SSL transactions per second 1000 Virtualized contexts 250 Module bandwidth in Gbps 4
You can also see the licenses that reside on the Flash disk by entering the following command:
ACE_module5/Admin# dir disk0: 236 Oct 17 09:18:26 2006 235 Oct 17 09:16:58 2006 1024 Sep 28 19:11:11 2006 1654606 Oct 26 12:56:16 2006 Usage for disk0: 2759552 8405120 11164672 ACE-SSL-05K-K9.lic <-------ACE-VIRT-250.lic <-------cv/ dplug
filesystem bytes total used bytes free total bytes
Cisco Application Control Engine (ACE) Troubleshooting Guide In the above example, there is an SSL 5K TPS license on the Flash disk that has not yet been installed in the ACE. To install the license, enter the following command:
ACE_module5/Admin# license install disk0:ACE-SSL-05K-K9.lic Installing license... done ACE_module5/Admin#
Configuring an ACL to Permit Input Traffic to the ACE

You must configure an ACL to allow the ACE to receive traffic. All traffic to the ACE is blocked until you do so. For example, to configure an ACL that permits all IP trafffic except from the 10.1.1.0 network, enter the following commands:
ACE_module5/Admin(config)# access-list ACL1 extended deny ip 10.1.1.0 255.255.255.0 any ACE_module5/Admin(config)# access-list ACL1 extended permit ip any any ACE_module5/Admin(config)# interface vlan 100 ACE_module5/Admin(config-if)# access-group input ACL1
Verifying that the ACE is Sending and Receiving Traffic

You can tell if traffic is reaching the ACE by using the show svclc module number traffic command on the Catalyst 6500 series switch or Cisco 7600 series router. This command displays counters (packets input and packets output) that increase when the switch or router sends packets to or receives packets from the ACE.
cat6k# show svclc module 5 traffic ACE module 5: Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 4d02h Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 528888 packets input, 41329093 bytes, 0 no buffer <------Received 469945 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 7776 packets output, 746361 bytes, 0 underruns <------0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
A trace of the Te?/1 (10-Gbps switch fabric interface, where ? = the module number) interface will show you whether packets are arriving at the switch fabric interface (SFI). Another useful command is the show cde health command on the ACE. This command shows the current state of the Classification Distribution Engine (CDE). The network processors (NP1 and NP2) are represented by IXP0 and IXP1,
Cisco Application Control Engine (ACE) Troubleshooting Guide respectively. You should not observe any drops, errors, or flow control issues in the output of this command. If the Packets Received or the Packets Transmitted counters of the CDE Hyperion Interface are not increasing, then packets are not coming into or going out of the ACE.
ACE_module5/Admin# show cde health CDE BRCM INTERFACE ====================== Packets received Packets transmitted Broadcom interface CRC error count BRCM VOQ status BRCM pull status
[empty]
4933 2437922 0 [not full] [pulling]
CDE HYPERION INTERFACE ====================== Packets received Packets transmitted Short packets drop count Fifo Full drop count Protocol error drop count FCS error drop count CRC error drop count Num times flow control triggered on hyp interface Num self generated multicast packets filtered HYP IXP0 VOQ status [empty] HYP IXP1 VOQ status [empty] HYP SLOW VOQ status [empty] HYP tx pull status CDE IXP0 INTERFACE ====================== Packets received Packets transmitted Num bad pkts recvd on fast Num bad pkts recvd on slow Num bad pkts recvd on fast Num bad pkts recvd on slow IXP0 Fast VOQ status IXP0 BRCM VOQ status IXP0 pull status IXP0 spi src status IXP0 spi snk status
29913371 <------8034 <------0 0 0 0 0 0 1880 [not full] [not full] [not full] [pulling]
spi spi spi spi
channel0 channel8 channel2 channel4 [empty] [empty]
784985 27827116 0 0 0 0 [not full] [not full] [pulling] [healthy] [healthy]
CDE1 SWITCH1 INTERFACE ====================== Packets received (hyp, ixp0) Packets received (bcm) Packets received (daughter card 0) Packets received (daughter card 1) Packets Errors received (hyp, ixp0) Packets Errors received (bcm) Packets Errors received (daughter card 0) Packets Errors received (daughter card 1) Packets transmitted (ixp1) Packets transmitted (nitrox) Packets Errors transmitted (ixp1) Packets Errors transmitted (nitrox) CDE2 SWITCH2 INTERFACE ====================== Packets received (ixp1) Packets received (nitrox) Packets Errors received (ixp1) Packets Errors received (nitrox)
4415 1656608 0 0 0 0 0 0 2089360 0 0 0
2089360 0 0 0

Packets Packets Packets Packets Packets Packets Packets Packets transmitted (hyp, ixp0) transmitted (broadcom) transmitted (daughter card 0) transmitted (daughter card 1) Errors transmitted (ixp1) Errors transmitted (nitrox) Errors transmitted (daughter card 0) Errors transmitted (daughter card 1) 4415 1656608 0 0 0 0 0 0
CDE IXP1 INTERFACE ====================== Packets received Packets transmitted Num bad pkts recvd on fast Num bad pkts recvd on slow Num bad pkts recvd on fast Num bad pkts recvd on slow IXP1 Fast VOQ status IXP1 BRCM VOQ status IXP1 pull status IXP1 spi src status IXP1 spi snk status
spi spi spi spi
channel0 channel8 channel2 channel4 [empty] [empty]
CDE NITROX INTERFACE ====================== Packets received Packets transmitted Num bad pkts recvd on fast spi channel0 Num bad pkts recvd on slow spi channel8 Num bad pkts recvd on fast spi channel2 Num bad pkts recvd on slow spi channel4 NTX Fast VOQ status NTX BRCM VOQ status NTX pull status NTX spi src status NTX spi snk status == Backplane == ITASCA_SYS_CNTL1 0x300 data 0x61f0000 ITASCA_SYS_CNTL2 0x304 data 0x80c30000
[empty] [empty]
You can also use the show interface command on the ACE to display traffic that is sent and received on the interface for each VLAN that is configured on the ACE.
ACE_module5/Admin# show interface bvi2 is administratively down Hardware type is BVI MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored vlan100 is administratively down
Verifying that the ACE is Sending and Receiving Traffic
27

Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored vlan130 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 10.86.215.134 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 59858 unicast packets input, 41711169 bytes <------193118 multicast, 280789 broadcast <------0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops <------6260 unicast packets output, 785167 bytes <------0 multicast, 1892 broadcast <------0 output errors, 0 ignored <-------
Verifying to-the-ACE Traffic

Traffic that is destined to the ACE itself arrives at the control plane in one of the following ways: Directly from the console connection Directly from the supervisor engine connection Traffic from the SFI that is forwarded by the CDE in the data plane Use the following commands to verify that traffic is going to and coming from the control plane.
ACE_module5/Admin# show netio stats High Priority (Control) ----------------------Net Rx Packets : Net Rx Bytes : Net Rx Unsupported L2 : Net Rx Lock Errors : Net Rx Interface Miss : Net Rx No Arp Client : Net Rx Alias Drops : Net Rx Repl. Errors : Net Rx Repl. If Err : Net Rx Internal Errs : Net Tx Packets Normal Priority (Data) ---------------------Net Rx Packets : Net Rx Bytes : Net Rx Unsupported L2 : Net Rx Lock Errors : Net Rx Interface Miss : Net Rx No Arp Client : Net Rx Alias Drops : Net Rx Repl. Errors : Net Rx Repl. If Errs : Net Rx Internal Errs : Net Tx Packets
224 17528 0 0 0 0 0 0 0 0
2521794 196704169 0 0 2326290 0 0 0 0 0
: 0
: 5213
28

Net Net Net Net Net Net Net Net Net Net Net Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Bytes Lock Errors Bad Context ID No Route Found No Adjacency Invalid If ID If Down No Src Addr No Encap FIFO Errors No VMAC Errors : : : : : : : : : : : 0 0 0 0 0 0 0 0 0 0 0 Net Net Net Net Net Net Net Net Net Net Net Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Bytes Lock Errors Bad Context ID No Route Found No Adjacency Invalid If ID If Down No Src Addr No Encap Fifo Errors No VMAC Errors : : : : : : : : : : : 414073 0 0 0 1 0 0 0 0 0 0
IPC Tx Packets IPC Tx Bytes IPC Tx Fifo Errors Client Rx Queue Full Pseudo Rx Queue Full
: 76 : 17638 : 0 : 0 : 0
: 0 : 0 : 0 : 0 : 0
ACE_module5/Admin# show fifo stats High Priority (Control) ----------------------Rx Packets : 224 Rx Bytes : 17528 Rx DMA Errors : 0 Rx Drop Events : 0 Rx Descr Errors : 0 Rx Bad Descrs : 0 Rx Length Errors : 0 Tx Tx Tx Tx Tx Packets Bytes Drops DMA Errors SOP Errors : : : : : 76 17682 0 0 0 Normal Priority (Data) ---------------------Rx Packets : 2524886 Rx Bytes : 196952927 Rx DMA Errors : 0 Rx Drop Events : 0 Rx Descr Errors : 0 Rx Bad Descrs : 0 Rx Length Errors : 0 Tx Tx Tx Tx Tx Packets Bytes Drops DMA Errors SOP Errors : : : : : 5241 464991 0 0 0
Global Errors ------------Rx Underflows Rx Overflows Tx Underflows Tx Overflows Resets Zbuff alloc fail
: : : : : :
0 0 0 0 0 0
Interrupt Stats --------------Total Interrupt count Rx Interrupt count Tx interrupt count
: 2529603 : 2524302 : 5310
This article describes how to troubleshoot basic ACE boot issues.
Guide Contents
29
Cisco Application Control Engine (ACE) Troubleshooting Guide Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Understanding ROMMON Mode and the ACE Boot Configuration 1.1 Setting the Boot Method from the Configuration Register 1.2 Booting the ACE from the ROMMON Prompt 1.3 Setting the BOOT Environment Variable 1.4 Displaying the ACE Boot Configuration 2 Restarting the ACE 2.1 Restarting the ACE from the ACE CLI 2.2 Restarting the ACE from the Supervisor Engine 3 Establishing a Console Connection to the ACE 4 Troubleshooting ACE Boot Problems
Contents
30
Understanding ROMMON Mode and the ACE Boot Configuration

You can control how the ACE performs its boot process through either the ACE configuration mode or ROM Monitor (ROMMON) mode. ROMMON is the ROM-resident code that starts executing at power up, reset, or when a fatal exception occurs. Two user-configurable parameters determine how the ACE boots: Boot field in the configuration register (confreg) BOOT environment variable The ACE enters ROMMON mode if it does not find a valid system image, if the Flash memory configuration is corrupted, or if the configuration register is set to enter ROMMON mode. Note: You can manually enter ROMMON mode by restarting the ACE and then pressing the Break key during the first 60 seconds of startup. If you are connected to the ACE through a terminal server, you can escape to the Telnet prompt and then enter the send break command to enter the ROMMON mode.
Setting the Boot Method from the Configuration Register

To change the configuration register settings and how the ACE boots from the CLI, use the following configuration mode command:
config-register value
The value argument-supported entries are as follows: 0?ACE boots to the ROMMON prompt. The ACE remains in ROMMON mode at startup. 1?ACE boots from the system image identified in the BOOT environment variable. If the ACE encounters an error or if the image is not valid, it will try the second image (if one is specified). If the second image also fails to boot, the ACE returns to ROMMON mode. For example, to set configuration register to boot the system image identified in the BOOT environment variable, enter the following command:
ACE_module5/Admin(config)# config-register 1
Booting the ACE from the ROMMON Prompt

If you specify a value of 0 for the config-register command, this configuration register setting forces the ACE to enter the ROMMON mode upon a reload or power cycle of the ACE. The ACE remains in ROMMON mode until you identify the location of an image file to boot. The ACE supports two methods of booting the module from the ROMMON prompt: To manually change the configuration register setting in ROMMON mode, use the confreg command followed by a value of 0 or 1. To change the boot characteristics using onscreen prompts, use the confreg command without a value. To instruct the ACE to manually boot from a particular system image, use the confreg command and specify a configuration register value of 1. Identify the name of the system image file that the ACE uses to boot.
Understanding ROMMON Mode and the ACE Boot Configuration
31
Cisco Application Control Engine (ACE) Troubleshooting Guide For example, to use the confreg command at the ROMMON prompt to instruct the ACE to boot from the c6ace-t1k9-mzg.3.0.0_A2_2_0.bin system image, enter the following command:
rommon 1 > confreg 1 rommon 2 > BOOT=disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin rommon 3 > sync
To instruct the ACE to automatically boot from the image specified in the BOOT variable, use the confreg command without specifying a configuration register value to launch the Configuration Summary menu-based utility. You can then instruct the ACE to boot from the system image identified in the BOOT environment variable. See the "Setting the BOOT Environment Variable" section. For example, to use the confreg command to display the onscreen prompts for changing the boot characteristics of the ACE and change the configuration register to boot from an image on disk0:, enter the following command:
rommon 4 > confreg
Configuration Summary (Virtual Configuration Register: 0x2000) enabled are: ignore system config info console baud: 9600 boot: the ROM monitor
do you wish to change the configuration? y/n [n]: disable "ignore system config info"? y/n [n]: change the boot characteristics? y/n [n]: y enter to boot: 0 = ROM Monitor 1 = boot file specified in BOOT variable [0]: 1
Configuration Summary (Virtual Configuration Register: 0x2001) enabled are: ignore system config info console baud: 9600 boot: the file specified in BOOT variable do you wish to change the configuration? y/n [n]: You must reset/power cycle for new config to take effect rommon 7 > dir disk0: Directory of disk0: 23951 31071143 -rwc6ace-t1k9-mzg.A2_2_0.bin 2 74448896 -rwTN-CONFIG 4546 32505856 -rwTN-CERTKEY-STORAGE 6530 11534336 -rwTN-LOGFILE 7234 11534336 -rwTN-HOME 7938 209715200 -rwTN-COREFILE 20738 1048576 -rwlkcddump 22689 250 -rwscripted_hm.txt 24584 30337516 -rwc6ace-t1k9-mz.A2_1_1.bin 29540 1048640 -rwACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121 29605 1048640 -rwACE_BOOT_ROM.img.rel.2008Apr01_ver121 rommon 8 > BOOT=disk0:c6ace-t1k9-mzg.A2_2_0.bin variable name contains illegal (non-printable) characters rommon 9 > sync
Setting the BOOT Environment Variable

The BOOT environment variable specifies a list of image files from which the ACE can boot at startup. To set the BOOT environment variable, use the boot system image: command. The syntax of this command is as follows:
boot system image:image_name
The image_name argument specifies the name of the system image file. If the file does not exist (for example, if you entered the wrong filename), then the filename is appended to the bootstring, and the "Warning: File not found but still added in the bootstring" message appears. If the file does exist, but is not a valid image, the file is not added to the bootstring, and the "Warning: file found but it is not a valid boot image" message appears. For example, to set the BOOT environment variable, enter the following command:
ACE_module5/Admin(config)# boot system image:c6ace-t1k9-mzg.3.0.0_A2_2.0.bin
Displaying the ACE Boot Configuration

To display the current BOOT environment variable and configuration register setting, use the show bootvar command in Exec mode. For example, to display the BOOT environment variable settings, enter the following command:
ACE_module5/Admin# show bootvar BOOT variable = "disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin" Configuration register is 0x1
Restarting the ACE

You can reload the ACE directly from its CLI or reboot it by using the supervisor engine CLI. You may need to reboot the ACE from the supervisor engine if you cannot reach the ACE through an external Telnet session or a console connection (for example, the ACE is remote).
Restarting the ACE from the ACE CLI

To reboot the ACE directly from its CLI and reload the configuration, use the reload command in Exec mode. The reload command reboots the ACE and performs a full power cycle of both the hardware and software. The reset process can take several minutes. Any open connections with the ACE are dropped after you enter the reload command. Caution: Configuration changes that are not written to the Flash partition are lost after a reload. Before rebooting, enter the copy running-config startup-config command in Exec mode to store the current configuration in Flash memory. If you fail to save your configuration changes, the ACE reverts to its previous settings upon restarting. When you enter the reload command, the ACE prompts you for confirmation and performs a cold restart of the ACE:
ACE_module5/Admin# reload This command will reboot the system Save configurations for all the contexts. Save? [yes/no]: [yes] Generating configuration.... running config of context Admin saved Perform system reload. [yes/no]: [yes]
Setting the BOOT Environment Variable
33
Restarting the ACE from the Supervisor Engine

To restart the ACE from the supervisor engine CLI, use the hw-module command. The syntax of this command is as follows:
hw-module module mod_num reset
For example, to use the supervisor engine CLI to reset the ACE located in slot 5 of the chassis, enter the following command:
cat6k# hw-module module 5 reset Proceed with reload of module?[confirm] % reset issued for module 5
Establishing a Console Connection to the ACE

In case the ACE becomes unresponsive or you cannot boot the ACE using the reload command from the Admin context, you can establish a direct serial connection between your terminal (laptop) and the ACE by making a serial connection to the console port on the front of the ACE. The console port is an asynchronous RS-232 serial port with an RJ-45 connector. Any device connected to this port must be capable of asynchronous transmission. Connection requires a terminal configured as 9600 baud, 8 data bits, 1 stop bit, no parity. Note: Only the Admin context is accessible through the console port; all other contexts can be reached through Telnet or SSH sessions. After you connect the terminal to the console port, use any terminal communications application to access the ACE CLI. The following procedure uses HyperTerminal for Windows. To access the ACE by using a direct serial connection, follow these steps: 1. Launch HyperTerminal. The Connection Description window appears. 2. Enter a name for your session in the Name field. 3. Click OK. The Connect To window appears. 4. From the drop-down list, select the COM port to which the device is connected. 5. Click OK. The Port Properties window appears. 6. Set the following port properties: Baud Rate = 9600 Data Bits = 8 Flow Control = none Parity = none Stop Bits = 1 7. Click OK to connect. 8. Press Enter to access the ACE login prompt.
switch login:
9. If the ACE does not find a valid software image on disk0: or if the ACE is configured to enter ROMMON mode upon booting up, the ROMMON prompt appears. Restarting the ACE from the Supervisor Engine 34

rommon 1>
Troubleshooting ACE Boot Problems

The ACE module receives power from the chassis back plane and boots up automatically when you insert the module into the chassis. If your ACE does not boot up when you insert it into the chassis or when you enter the reload Exec mode command from the Admin context, you cannot Telnet to the ACE or establish a session from the supervisor engine. In these cases, use the following steps to troubleshoot the issue and boot the ACE: 1. Log in to the Catalyst 6500 series switch or the Cisco 7600 series router and check the status of the ACE by entering the following command:
User Access Verification
Password: cat6k>enable Password: cat6k# show module 5 Mod Ports Card Type Model Serial No. --- ----- -------------------------------------- ------------------ ----------5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S <------- Module is receiving power
Mod MAC addresses Hw Fw Sw Status --- ---------------------------------- ------ ------------ ------------ ------5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 Unknown Unknown Other <------- Firmware and software image Mod Online Diag Status ---- ------------------5 Unknown <------- Diagnostics status is Unknown
The first row of information is populated, so you know that the ACE is powered up. The firmware and software versions are Unknown and the Status is Other. At this point, you cannot session into the ACE from the supervisor engine. 2. Power cycle the ACE from the supervisor engine to attempt to boot the ACE by entering the following commands:
cat6k# config t Enter configuration commands, one per line. cat6k(config)# no power enable module 5 cat6k(config)# power enable module 5 End with CNTL/Z.
Wait long enough for the ACE to boot up. Try to Telnet or session to the ACE. If you still cannot Telnet or session to the ACE, continue with Step 3. 3. Establish a console connection to the ACE. For details about establishing a console connection to the ACE, see the "Establishing a Console Connection to the ACE" section.
rommon 1>
4. Check the ACE configuration register (confreg) by entering the following command:
rommon 2> confreg Configuration Summary (Virtual Configuration Register: 0x1) enabled are: console baud: 9600 boot: the file specified in BOOT variable
A value of 0x1 instructs the ACE to boot from the image in disk0:. A value 0x0 instructs the ACE to boot to the ROMMON prompt. If the image specified in the BOOT variable is not in disk0:, then the ACE boots to the ROMMON prompt as shown in
Cisco Application Control Engine (ACE) Troubleshooting Guide this example issue. 5. Check the BOOT variable by entering the following command:
rommon 3> set PS1=rommon ! > RELOAD_REASON=reload command by admin BOOT=disk0:c6ace-t1k9-mz.3.0.0_A2_2_0.bin ARGV0=quiet ?=0
6. Ensure that the software image specified in the BOOT variable is present in disk0: by entering the following command:
rommon 4> dir disk0: 31071143 250 30337516 1048640 1048640 Dec Feb Jul Aug Aug 1 17:01:06 2008 c6ace-t1k9-mzg.A2_2_0.bin 8 20:04:44 2008 scripted_hm.txt 31 05:47:42 2008 c6ace-t1k9-mz.A2_1_3.bin 8 11:45:06 2008 ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121 8 13:27:32 2008 ACE_BOOT_ROM.img.rel.2008Apr01_ver121 filesystem bytes total used bytes free total bytes
Usage for image: 506789888 517210112 1024000000
7. If the specified image is not in disk0:, then you can boot from another image in disk0: by entering the following command:
rommon 5> boot system disk0:image_name
8. If there is no image on the ACE disk0: to boot from, you can still boot from the supervisor engine. Copy the image to the supervisor engine's disk0: or disk1:, and then from the supervisor CLI, enter the following command:
cat6k(config)# boot device module slot_number disk[0 | 1]:image_name
The ACE boots and stops at the ROMMON prompt.
9. At the ROMMON prompt on the ACE console, enter the following command to boot the ACE from the Ethernet Out-of-Band Channel (EOBC) between the ACE and the Catalyst 6500 series switch or the Cisco 7600 series router:
rommon 6> boot eobc:
10. If the ACE is not local or you cannot establish a console connection for any other reason, use the following procedure to finish booting the ACE from the supervisor engine with the boot eobc: command:
cat6k# remote login switch Trying Switch ... Entering CONSOLE for Switch
cat6k-sp# svclc console 5 Entering svclc ROMMON of slot 5 ... Type "end" to end the session.
rommon 7> boot eobc:
This article describes the ACE system logging facility, how to enable logging, and how to use system messages as troubleshooting tools. 36
Troubleshooting ACE Boot Problems
Contents
1 Overview of ACE System Logging 2 Enabling ACE Logging 3 Logging Severity Levels 4 Adding Information to syslogs 5 Troubleshooting ACE Logging 5.1 Displaying Logging Statistics 5.2 Displaying the Logging History 5.3 Displaying Logging Messages 5.4 Displaying Logging Persistence 5.5 Displaying the Logging Rate Limit
Contents
37
Overview of ACE System Logging

The ACE provides a system logging (syslog) facility that collects and saves system messages and outputs them to destinations that you specify as follows: Buffer Console Flash memory Host (remote syslog server) Monitor Standby Supervisor Each virtual context generates logs independently from the other virtual contexts. The admin virtual context does not log on behalf of the other contexts in the ACE. The ACE logs the following connection setup and teardown messages in the CP at the connection speed: 106023, 302022, 302023, 302024, and 302025. These setup and teardown syslogs are directly forwarded to a remote server. Because of the potentially large number of these messages or if you require high-rate system logging of connection setup and teardown messages, use the logging fastpath command. This command disables CP syslogs and enables logging of these messages through the DP using a slightly different format and the following syslog IDs: 106028, 302028, 302029, 302030, and 302031, respectively. You can log these messages through the fast path only to an external syslog server. All other enabled logging destinations are disabled by the logging fastpath command. You can limit the rate at which the ACE generates syslog messages by using the logging rate-limit command. This command allows you to rate limit syslogs based on one of the following criteria: Time interval Logging level Message ID Besides logging system messages, the ACE logs access control list (ACL) deny entries. Note: Remember to enable logging on the standby ACE in a redundant configuration by entering the logging standby command. To log failover information on the standby ACE, you need to set the logging level to 4. For details about ACE system message logging, see the Cisco Application Control Engine Module System Message Guide (Software Version A2(1.0)).

To enable logging on the ACE module and send syslogs to the monitor, enter the following commands:
ACE_module5/Admin(config)# logging enable ACE_module5/Admin(config)# logging monitor 7 ACE_module5/Admin(config)# logging trap 7
Overview of ACE System Logging
38

ACE_module5/Admin(config)# ACE_module5/Admin(config)# ACE_module5/Admin(config)# ACE_module5/Admin(config)# no logging message 111008 no logging message 111009 logging timestamp do terminal monitor
Use the logging monitor severity_level command only when you are troubleshooting problems on the ACE or when there is minimal load on the network. Using this command at other times when the ACE is active may degrade performance.
Note: logging trap defines the severity sent to the syslog server.
Note: If you do not see syslog messages on the console after enabling logging with the logging enable and logging monitor 7 commands, log out of the ACE and then log in again. To enable logging to a syslog server, use the following command syntax:
logging host ip_address [tcp | udp [/port#]] | [default-udp] | [format emblem]
Note: If you specify the default-udp option and TCP logging fails, the ACE sends logging messages over UDP. You can verify that the ACE defaults to UDP by entering the following command:
ACE_module5/Admin# show logging Syslog logging: enabled Facility: 20 History logging: disabled Trap logging: enabled (level - debugging) Timestamp logging: disabled Fastpath logging: disabled Persist logging: disabled Standby logging: disabled Rate-limit logging: disabled (min - 0 max 100000 msgs/sec) Console logging: disabled Monitor logging: disabled Logging to 5.1.0.40 tcp/514 default-udp (sending on UDP) Device ID: disabled Message logging: none Buffered logging: enabled (level - debugging) maximum size 681984 Buffer info: current size - 681984 global pool - 1048576 used pool - 1048576 min - 0 max - 681984 cur ptr = 42894 wrapped - yes
Use the logging supervisor command to allow the aggregation of critical syslogs from multiple virtual devices to the Catalyst 6500 series switch or to the Cisco 7600 series router syslog. For example, enter the following command:
ACE_module5/Context(config)# logging supervisor ? <0-7> 0-emerg;1-alert;2-crit;3-err;4-warn; 5-notif;6-inform;7-debug cat6k# show logging . . . cat6k#17w3d: %TRINITY-7-TRINITY_SYSLOG_DEBUG:
%ACE-7-111009: User 'admin' executed cmd: show running.
39
Logging Severity Levels

The severity_level argument specifies the maximum level for system log messages sent to the console. The severity level that you specify indicates that you want syslog messages at that level and messages less than the level. For example, if the specified level is 3, the syslog displays level 3, 2, 1, and 0 messages. We recommend that you use a lower severity level, such as 3, since logging at a high rate may impact the performance of the ACE. Allowable entries are as follows: 0?emergencies (System unusable messages) 1?alerts (Take immediate action) 2?critical (Critical condition) 3?errors (Error message) 4?warnings (Warning message) 5?notifications (Normal but significant condition) 6?informational (Information message) 7?debugging (Debug messages)
Adding Information to syslogs

After you have enabled system message logging and have specified a destination for the system messages, you can add more information to the system messages that may be helpful in troubleshooting issues with your ACE module. For example, you can do the following: Add a timestamp Identify the messages sent to a syslog server Identify the ACE device ID in messages that are sent to a syslog server To add a timestamp to syslog messages, enter the following command:
ACE_module5/Admin(config)# logging timestamp
To identify messages that are sent to a syslog server by severity level, enter the following command:
ACE_module5/Admin(config)# logging trap severity_level
For example, to identify the ACE device ID in messages that are sent to a syslog server, use the following command syntax:
ACE_module5/Admin(config)# logging device-id {context-name | hostname | ipaddress interface_name | string text}
Troubleshooting ACE Logging

The commands in the following sections are useful for troubleshooting the system message logging facility.
Displaying Logging Statistics

ACE_module5/Admin# show logging statistics Syslog statistics: sent 349 discarded 64
Messages sent: console 0 buffer 348 persistent 0

supervisor 1 history 0 monitor 0 host 0 misc 0 Messages discarded: cfg rate-limit 0 hard rate-limit 0 server down 5 queue full 59 errors 0 SNMP-related counters: notifications sent 0 history table flushed 0 messages ignored 0 NP-related counters: to-CP dropped 0 fastpath sent 0 fastpath dropped
ACE_module5/Admin# show logging queue Logging Queue length limit : 80 msg(s), 59 msg(s) discarded. Current 0 msg on queue, 80 msgs most on queue CP messages received: 426 IXP messages received: 82 Xscale messages received: 0 , 59 msg(s) discarded.
System Max Queue size: 20080 System Free Queue size for allocation: 19920
In the above example, the ACE has discarded 59 control plane (CP) messages. By default, the syslog message queue can hold 80 messages. You can increase the size of the syslog message queue by using the logging queue command in configuration mode. Set the queue size before you start collecting syslog messages. When traffic is heavy, messages may be discarded if the queue size is too small. The maximum number of messages that the queue can hold is 8192.
Displaying the Logging History

To display the ACE logging history, enter the following command from the console:
ACE_module5/Admin# show logging history syslog_trinity_show_history for context 0: 1(Mar 24 2009 16:39:36): from "KERN" 2(Mar 24 2009 16:39:48): from "KERN" 3(Mar 24 2009 16:39:56): from "KERN" 4(Mar 24 2009 16:49:50): from "KERN" 5(Mar 24 2009 16:51:35): from "KERN" 6(Mar 24 2009 16:51:36): from "KERN" 7(Mar 24 2009 16:51:40): from "KERN" 8(Mar 24 2009 16:51:41): from "KERN" 9(Mar 24 2009 16:54:46): from "KERN"
ACE-5-111008:User 'admin' ACE-5-111008:User 'admin' ACE-7-111009:User 'admin' ACE-7-111009:User 'admin' ACE-4-405001:Received ARP ACE-4-405001:Received ARP ACE-4-405001:Received ARP ACE-4-405001:Received ARP ACE-7-111009:User 'admin'
executed the 'logging history 7' command. executed the 'logging console 7' command. executed cmd: do sho logging history executed cmd: do show logging message REQUEST collision from 10.1.1.240 00.0c.29. RESPONSE collision from 10.1.1.240 00.0b.fc REQUEST collision from 10.1.1.240 00.0c.29. RESPONSE collision from 10.1.1.240 00.0b.fc executed cmd: telnet 10.1.1.130
Displaying Logging Messages

If a particular system message does not appear in the syslog history, check that the message is enabled and that the logging level is correct for that message. To display the default logging level and the current status of a system message, all system messages, or Displaying Logging Statistics 41
Cisco Application Control Engine (ACE) Troubleshooting Guide disabled system messages, enter the following command:
ACE_module5/Admin# show logging message message_id | all | disabled
For example, to display all disabled system messages in the ACE, enter the following command:
ACE_module5/Admin# show logging message disabled Message logging: message 111008: default-level 5 (disabled) message 111009: default-level 7 (disabled)
Displaying Logging Persistence

The logging persistence command enables logging to disk0: in Flash memory. The messages are stored in a subdirectory of disk0: called /messages. To display logging persistence, enter the following command:
ACE_module5/Admin# show logging persistent Persist info: current size - 6626 global pool - 1048576 used pool - 6626 min - 0 max - 189582 cur ptr = 6638 wrapped - no Mar Mar Mar Mar 24 24 24 24 2009 2009 2009 2009 09:51:31 09:51:32 09:51:36 09:51:37 Admin: Admin: Admin: Admin: %ACE-4-405001: %ACE-4-405001: %ACE-4-405001: %ACE-4-405001: Received Received Received Received ARP ARP ARP ARP
REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on int RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on in REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on int RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on in
Displaying the Logging Rate Limit

To display the logging rate limit, enter the following command:
ACE_module5/Admin# show logging rate-limit Rate-limit logging: (min - 0 max 100000 msgs/sec) 100000 messages 1 seconds level 7
This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with your ACE. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Displaying Logging Messages 42
Cisco Application Control Engine (ACE) Troubleshooting Guide Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Connection Handling 2 Internal Mapping of ACE TCP and UDP Flows 3 ACE Connection Table Entries 4 Tracking Connections Through the ACE 5 Troubleshooting Connections
Overview of ACE Connection Handling

This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections, the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (see Figure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled in the fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." The ACE completes the TCP handshake . This process applies to the following functions: Basic load balancing Source IP sticky TCP/IP normalization
Displaying the Logging Rate Limit
43
Cisco Application Control Engine (ACE) Troubleshooting Guide Figure 1. Layer 4 Flow Setup
For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as a proxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCP connection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. The client responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GET or POST).
Figure 2. Layer 7 Flow Setup -- Client Connection
44
After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to the real server based on the load-balancing method and other criteria. See Figure 3.
Figure 3. Layer 7 Flow Setup -- Server Connection
45
Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection to the server. For the life of the HTTP flow, the client communicates directly with the server through the fast path (hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures 4.
Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together
46
Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flows together.
Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers
47
With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parses subsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server and ACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for each new HTTP 1.1 request over the same TCP connection.
Figure 6. Layer 7 Flow Setup -- Reproxy
48
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fully terminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalf of the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCP connection with the server. See Figure 7.
Figure 7. SSL Handshake
49
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client and server connections are completely independent and flows are handled in the software, not in the fast path. See Figure 8.
Figure 8. Layer 7 Flow Setup -- Full Proxy
50
Internal Mapping of ACE TCP and UDP Flows

The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You can display the current connections in the ACE by entering the show connections command. See Figure 8.
Figure 8. Internal Flow Mapping
Internal Mapping of ACE TCP and UDP Flows
51
ACE Connection Table Entries

Understanding ACE?s Conn Table Entries During: L4 TCP Connection Setup (3 Way Handshake) Normalisation Enabled Normalisation Disabled L7 TCP Connection Setup (3 Way Handshake) TCP Connection Teardown 3 Way Handshake 4 Way Handshake Reset
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
Tracking Connections Through the ACE

You can display the IDs for the request and response connections in the ACE by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-c 9" Connection ID:seq: 9[0x9].6 <------- Request and response connection ID Other ConnID : 3[0x3].4 Proxy ConnID : 0[0x0].0 Next Q : 1124073484[0x4300000c] 192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 17 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 0 EncapsID:ver : 0:0 TCP ACK delta : 0x0 MSS : 0 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 152 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 0 Byte Count : 0
67

TCP Information: (State = 0) Window size : 0 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 0 Last seq : 0 timestamp_delta: 0 Last ack : 1ce862 No Trigger : 0 Trigger Status : 0 Timestamp : 26ebc96d TCP options negotiated: Sack:Allow TS:Allow Windowscale: Allow Reserved: Allow Exceed MSS: Allow Window var: Allow
You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option of the show np command as follows:
ACE_module5/Admin# show np 1 me-stats "-c 9 -v" Connection ID:seq: 9[0x9].2 Other ConnID : 7[0x7].14 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0] 10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1260 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 4 Packet Count : 347 Byte Count : 24476 TCP Information: (State = 3) Window size : 5840 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 53768a51 timestamp_delta: 0 Last ack : 658c1f72 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000 Raw Connection Entry 0000 0x00000000 0x0a56d786 0010 0x001712e5 0x00000000 0020 0x4e000007 0x00000000 0030 0x0000015b 0x00005f9c 0040 0x53768a51 0x658c1f72 0050 0x00000094 0x00000000 0060 0x00000000 0x00000000
0xa12c438f 0x00030000 0x00080480 0x16d00030 0x459f781e 0x45729985 0x00000000
0x06210007 0x04ec1004 0x24450000 0x05b40000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0
68

No valid proxy entry. No valid TCB proxy entry. No valid HTTP proxy entry. No valid SSL proxy entry. No valid AI proxy entry. Connection ID:seq: 7[0x7].14 Other ConnID : 9[0x9].2 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0] 172.27.16.143:4837 -> 10.1.1.5:23 [RX-NextHop: CP] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 1 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1460 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 486 Byte Count : 19810 TCP Information: (State = 3) Window size : 65371 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 658c1f72 timestamp_delta: 0 Last ack : 53768a51 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000 Raw Connection Entry 0000 0x00000000 0xa12c438f 0010 0x12e50017 0x00000000 0020 0x02000009 0x00000000 0030 0x000001e6 0x00004d62 0040 0x658c1f72 0x53768a51 0050 0x00000094 0x00000000 0060 0x00000000 0x00000000
0x0a56d786 0x00030000 0x00080481 0xff5b0030 0x459f781e 0x45729985 0x00000000
0x06e90007 0x05b41000 0x24450000 0x05b40000 0x00000000 0x00000000 0x00000000
Doing verbose output for proxy id: 0 No No No No No valid valid valid valid valid proxy entry. TCB proxy entry. HTTP proxy entry. SSL proxy entry. AI proxy entry.
Troubleshooting Connections
To troubleshoot suspected connectivity issues, follow these steps:
Cisco Application Control Engine (ACE) Troubleshooting Guide 1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, go to Step 2. Otherwise, verify that the access list is configured properly to permit traffic.
ACE_module5/Admin# show access-list anyone detail access-list:anyone, elements: 1, status: ACTIVE remark : access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count
2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0, verify that the service policy is active (show service-policy command) and the server farm is up (show server-farm detail command). If the service policy is large, use the show service-policy policy_name summary command for more information as follows:
ACE_module5/Admin# show service-policy VIP summary service-policy: VIP Class State Curr Conns VIP IN-SRVC 0 IN-SRVC IN-SRVC IN-SRVC VIP2 IN-SRVC IN-SRVC 0 0 0 0 0
Hit Count 0 0 0 0 0 0
VIP Conns Drop 192.168.12.192 0 192.168.12.192 0 192.168.12.193 0 192.168.12.193 0 192.168.12.194 0 192.168.12.194 0
Prot tcp tcp tcp tcp tcp tcp
Port eq 443 eq 80 eq 80 eq 443 eq 80 eq 443
VLAN 100 100 100 100 100 100
3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7 rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for any misconfiguration.
ACE_module5/Admin# show stats loadbalance
+------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 3 <-------| Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the conf Total Layer7 rejections : 7 <-------| Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 10 <------- Failed connections due to no real server' Total ACL denied : 0 Total IDMap Lookup Failures : 0
To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalance command. 4. If none of the error statistics is increasing, check the connection record by entering the show conn detail command and checking the connections for the affected VIP.
ACE_module5/Admin# show conn detail total current connections : 6 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB [ idle time : 00:00:00, byte count : 60055 ]

9 [ elapsed time: 04:15:29, packet count: 1473 ] 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 [ conn in reuse pool : FALSE] [ idle time : 00:00:00, byte count : 64880 ] [ elapsed time: 04:15:29, packet count: 1086 ] ESTAB
5. Display existing ACE connection statistics by entering the following command:

ACE_module5/Admin# show stats connection
+------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 628950 Total Connections Current : 7 Total Connections Destroyed: 389 Total Connections Timed-out: 3958 Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or i
The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with the server. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command. 6. Display service policy statistics by entering the following command:
ACE/Context# show service-policy client-vips detail Status : ACTIVE Description: ----------------------------------------Interface: vlan 211 service-policy: client-vips class: VIP-HTTPS VIP Address: Protocol: Port: 172.16.11.190 tcp eq 443 <------- Shows the VIP address, port, and protocol loadbalance: L7 loadbalance policy: HTTPS-POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED-WHEN-ACTIVE VIP State: INSERVICE <------- Service is INSERVICE curr conns : 22 , hit count : 22 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 max-conn-limit : 0 , drop-count : 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : HTTPS-POLICY class/match : class-default LB action : primary serverfarm: backend-ssl backup serverfarm : hit count : 22 <------- Shows the hit count dropped conns : 0
7. Display server farm connection statistics by entering the following command:

ACE/Context# show serverfarm HTTPS-FARM detail serverfarm : HTTPS-FARM, type: HOST total rservers : 4 active rservers: 4
71
description : state : ACTIVE predictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used failaction : back-inservice : 0 partial-threshold : 0 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 ------------------------------------------connections----------real weight state current total failures ---+---------------------+--------+---------------------+-----------+----------+--------rserver: linux-1 192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection s each real server max-conns : , out-of-rotation count : min-conns : conn-rate-limit : , out-of-rotation count : bandwidth-rate-limit : , out-of-rotation count : retcode out-of-rotation count : -
The Connections Failures counter for a real server in a server farm may increment for one of the following reasons: SYN timeout (the three-way handshake fails to complete) RST received (a client sends an RST to the server) Internal exception (internal software issue) 8. Display the statistics for a connection parameter map by entering the following command:
ACE_module5/Admin# show parameter-map CONN_PARAMMAP Number of parameter-maps : 1
Parameter-map : CONN_PARAMMAP Type : connection nagle : slow start : buffer-share size : inactivity timeout (seconds) : embryonic timeout (seconds) : ack-delay (milliseconds) : WAN Optimization RTT (milliseconds): half-closed timeout (seconds) : TOS rewrite : syn retry count : TCP MSS min : TCP MSS max : tcp-options drop range : tcp-options allow range : tcp-options clear range : selective-ack : timestamp : window-scale : window-scale factor : reserved-bits : random-seq-num : SYN data : exceed-mss : urgent-flag : conn-rate-limit : bandwidth-rate-limit :
disabled disabled 32768 TCP: 3600, UDP: 120, ICMP: 2 5 200 65535 3600 disabled 4 0 1460 0-0 0-0 1-255 clear clear clear 0 allow enabled drop drop allow disabled disabled
72
Cisco Application Control Engine (ACE) Troubleshooting Guide 9. Reset the ACE connection statistics by entering the following commands: clear conn [all | flow {icmp | tcp | udp} | rserver server_name] clear stats conn clear tcp statistics clear udp statistics
This article describes how to troubleshoot issues involving ACE remote access. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Remote Access 2 Configuring a Management Policy for Remote Access 3 Troubleshooting Remote Access 3.1 Troubleshooting Telnet 3.2 Troubleshooting SSH 3.3 Troubleshooting KAL-AP
Contents
73
Overview of ACE Remote Access

You can access the ACE remotely using several different protocols as follows: HTTP HTTPS ICMP KALAP-UDP SSH SNMP Telnet These protocols require that you configure a management traffic policy on the ACE and associate that policy with the interface that you intend to use for management traffic. A management policy allows the classification and distribution engine (CDE) to classify (match) incoming management traffic to the management policy and to forward that traffic to the control plane (CP). For complete details about remote access, see the Cisco Application Control Engine Module Administration Guide (Software Version A2(1.0)).
Configuring a Management Policy for Remote Access

To configure a management policy that allows remote access to the ACE using ICMP, SSH, or Telnet, enter the following commands:
ACE_module5/Admin(config)# access-list ACL1 extended permit ip any any ACE_module5/Admin(config)# class-map type management match-any MGMT_CLASS ACE_module5/Admin(config-cmap-mgmt)# 2 match protocol icmp any ACE_module5/Admin(config-cmap-mgmt)# 3 match protocol ssh any ACE_module5/Admin(config-cmap-mgmt)# 4 match protocol telnet any ACE_module5/Admin(config-cmap-mgmt)# exit ACE_module5/Admin(config)# policy-map type management first-match MGMT_POLICY ACE_module5/Admin(config-pmap-mgmt)# class MGMT_CLASS ACE_module5/Admin(config-pmap-mgmt-c)# permit ACE_module5/Admin(config-pmap-mgmt-c)# exit ACE_module5/Admin(config-pmap-mgmt)# exit ACE_module5/Admin(config)# interface vlan 100 ACE_module5/Admin(config-if)# ip address 192.168.12.15 255.255.255.0 ACE_module5/Admin(config-if)# service-policy input MGMT_POLICY ACE_module5/Admin(config-if)# access-group input ACL1
Troubleshooting Remote Access

If you cannot access the ACE module remotely, follow these steps: Overview of ACE Remote Access 74
Cisco Application Control Engine (ACE) Troubleshooting Guide 1. Beginning with ACE software release A2(1.1), by default, the ACE CLI is only locally accessible either using the ACE console port or through the supervisor by entering the session command. Remote access to the ACE (for example, Telnet, SSH, and so on) is disabled until you change the admin user account password from the default. Access to the XML API is also disabled until you change the www user account password from the default. The ACE will display these warnings each time you access the CLI using the the console port or the supervisor until you change these passwords.
cat6k#session slot 5 processor 0 The default escape character is Ctrl-^, then x. You can also type 'exit' at the remote prompt to end the session Trying 127.0.0.20 ... Open ACE_module5 login: admin Password: Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. Please change the password for admin user. Admin user is allowed to login only from supervisor until the password is changed. User 'www' is disabled. Please change the password to enable the user.
Use the following commands to change the passwords of the admin and www user accounts:
ACE_module5/Admin# config Enter configuration commands, one per line. End with CNTL/Z. ACE_module5/Admin(config)# username admin password 0 cisco123 role Admin domain default-domain ACE_module5/Admin(config)# username www password 0 cisco123 role Admin domain default-domain ACE_module5/Admin(config)# exit
Note that, although the passwords were entered in clear text above, they will be stored in the ACE configuration in an encrypted format:
ACE_module5/Admin# show run | i username Generating configuration.... username admin password 5 $1$M7gtcvBC$9ca78Q.ZH5jZpqDVuLnkN0 role Admin domain default-domain username www password 5 $1$ulc7KHL5$2HlgNTEez03.ElmbiWKyY/ role Admin domain default-domain
2. Ensure that the remote access method protocol (for example, Telnet or SSH) that you are trying to use is configured in the management class map and that the management class has been permitted in the management policy. If necessary, correct your ACE configuration. To display your management policy configuration elements, enter the following Exec mode commands:
ACE_module5/Admin# show running-config class-map Generating configuration.... class-map 2 match 3 match 4 match type management match-any MGMT_CLASS protocol icmp any protocol ssh any protocol telnet any
ACE_module5/Admin# show running-config policy-map MGMT_POLICY Generating configuration.... policy-map type management first-match MGMT_POLICY class MGMT_CLASS permit class class-default
75
Cisco Application Control Engine (ACE) Troubleshooting Guide 3. Ensure that the management policy is applied to the correct interface and that you are using the correct IP address for that interface. If necessary, correct your configuration. Enter the following command:
ACE_module5/Admin# show running-config interface interface vlan 100 ip address 192.168.12.15 255.255.255.0 access-group input ACL1 access-group output ACL1 service-policy input MGMT_POLICY no shutdown
4. Check the status of the management interface by entering the following command:
ACE_module5/Admin# show interface vlan 100 vlan100 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 192.168.12.15 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 115303 unicast packets input, 74570169 bytes 273637 multicast, 521226 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 12591 unicast packets output, 2120271 bytes 0 multicast, 4604 broadcast 0 output errors, 0 ignored
5. If the interface is down, ensure that the no shutdown command is configured on the interface to enable it. If necessary, correct your configuration. Enter the following command:
ACE_module5/Admin# show running-config interface
6. Ensure that you have not exceeded the allocated resources for management connections or maximum management bandwidth by entering the following commands:
ACE_module5/Admin# show resource usage resource mgmt-connections Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------Context: Admin mgmt-connections 2 10 0 100000 0 Context: C1 mgmt-connections 0 0 0 100000 0 ACE_module5/Admin# show resource usage resource rate mgmt-traffic Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------Context: Admin mgmt-traffic rate 78 3588 0 125000000 0 Context: C1 mgmt-traffic rate 0 0 0 125000000 0
7. If necessary, allocate more resources to management connections by entering the following command: Troubleshooting Remote Access 76

ACE_module5/Admin(config-resource)# limit-resource mgmt-connections minimum number maximum number
8. Ensure that traffic is reaching the CP by entering the following command:

ACE_module5/Admin# show netio stats High Priority (Control) ----------------------Net Rx Packets : Net Rx Bytes : Net Rx Unsupported L2 : Net Rx Lock Errors : Net Rx Interface Miss : Net Rx No Arp Client : Net Rx Alias Drops : Net Rx Repl. Errors : Net Rx Repl. If Err : Net Rx Internal Errs : Net Net Net Net Net Net Net Net Net Net Net Net Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Packets Bytes Lock Errors Bad Context ID No Route Found No Adjacency Invalid If ID If Down No Src Addr No Encap FIFO Errors No VMAC Errors : : : : : : : : : : : : Normal Priority (Data) ---------------------Net Rx Packets : Net Rx Bytes : Net Rx Unsupported L2 : Net Rx Lock Errors : Net Rx Interface Miss : Net Rx No Arp Client : Net Rx Alias Drops : Net Rx Repl. Errors : Net Rx Repl. If Errs : Net Rx Internal Errs : Net Net Net Net Net Net Net Net Net Net Net Net Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Tx Packets Bytes Lock Errors Bad Context ID No Route Found No Adjacency Invalid If ID If Down No Src Addr No Encap Fifo Errors No VMAC Errors : : : : : : : : : : : :
680 52902 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10799640 <------842376636 <------0 0 10470926 0 0 0 0 0 10469 <------1194879 <------0 0 0 0 0 0 0 0 0 0
: 78 : 17766 : 0 : 0 : 0
: 0 : 0 : 0 : 0 : 0
Management traffic is considered to-the-ACE traffic or CP traffic. If traffic is reaching the CP, the Normal Priority (Data) Net Rx Packets, Net Rx Bytes, Net TX packets, and Net TX bytes counters should be increasing. If not, contact TAC. 9. If traffic is not arriving at the CP, ensure that traffic is reaching the classification and distribution engine (CDE) from the SFI by entering the following command:
ACE_module5/Admin# show cde health CDE BRCM INTERFACE ====================== Packets received Packets transmitted Broadcom interface CRC error count BRCM VOQ status BRCM pull status CDE HYPERION INTERFACE ====================== Packets received Packets transmitted Short packets drop count Fifo Full drop count Protocol error drop count
[empty]
10580 10802845 0 [not full] [pulling]
12312951 <------17361 <------0 0 0
77

FCS CRC Num Num HYP HYP HYP HYP error drop count error drop count times flow control triggered on hyp interface self generated multicast packets filtered IXP0 VOQ status [empty] IXP1 VOQ status [empty] SLOW VOQ status [empty] tx pull status 0 0 0 4618 [not full] [not full] [not full] [pulling]
<snip>
If traffic is reaching the CDE, the Packets received and the CDE Hyperion Interface Packets transmitted counters should be increasing. If not, contact TAC. 10. If packets are not reaching the CDE, ensure that the MSFC in the Catalyst 6500 series switch or the Cisco 7600 series router is sending packets to the switch fabric interface (SFI) by entering the following command on the supervisor engine:
cat6k# show svclc module 5 traffic ACE module 5: Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 1w2d Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 912150 packets input, 74727962 bytes, 0 no buffer Received 796374 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 17390 packets output, 2145844 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
If the MSFC is sending traffic to the SFI, the packets input and the packets output counters should be increasing. If not, contact TAC.
Troubleshooting Telnet
If you cannot Telnet to the ACE, ensure that you have not reached the maximum connection limit for Telnet by entering the following commands:
ACE_module5/Admin# show telnet Session ID 4254 Remote Host 127.0.0.51 :41985 Active Time 0: 8:13
The show telnet command output shows only one Telnet session. A maximum of 15 more users can potentially Telnet to the Admin context.
Troubleshooting Telnet
78
Cisco Application Control Engine (ACE) Troubleshooting Guide To display the maximum number of users allowed to Telnet to a particular context, enter the following command:
ACE_module5/Admin# show telnet maxsessions Maximum Sessions Allowed is 16
Troubleshooting SSH
If you attempt to connect to the ACE using SSH and receive the following error, follow these steps:
[linux]$ ssh admin@192.168.0.210 ssh_exchange_identification: Connection closed by remote host [linux]$
1. Ensure that SSH is enabled in the management policy by entering the following command:
ACE_module5/Admin# show running-config class-map class-map 2 match 3 match 4 match 6 match 7 match 8 match type management match-any MGMT_CLASS protocol http any protocol https any protocol icmp any protocol ssh any <------- SSH is enabled protocol telnet any protocol snmp any
switch/Admin# show running-config policy-map MGMT_POLICY Generating configuration.... policy-map type management first-match MGMT_POLICY class MGMT_CLASS permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH
2. Ensure that the SSH key has been generated by entering the following command:
switch/Admin# show ssh key ************************************** could not retrieve rsa1 key information ************************************** could not retrieve rsa key information ************************************** could not retrieve dsa key information ************************************** no ssh keys present. you will have to generate them **************************************
The show ssh key command output shows that no SSH key has been generated. 3. Generate an SSH key based upon your security requirements by entering the following commands:
ACE_module5/Admin# config Enter configuration commands, one per line. ACE_module5/Admin(config)# ssh key rsa 4096 generating rsa key(4096 bits)..... ......................... generated rsa key ACE_module5/Admin(config)# exit switch/Admin# show ssh key dsa rsa rsa1 ACE_module5/Admin# show ssh key ************************************** could not retrieve rsa1 key information End with CNTL/Z.

************************************** rsa Keys generated:Tue Apr 7 15:55:20 2009
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAr3z0MO0knoS6YwntSxUkWDWjHZfFE7Y6nLd8qQVcPMu0 XpvkabDswLwoEdC9nOWM4v4g4PDpUz+tk+2WHJ4MMgRVeomVbK/2+Zx0Eds1p2XlhiV9KPcV pflpNNt63Mr01oLHoHpjxJ8ubfJJ+gPhMoBmQyGKedQOk5tVlbOpyxS2f7yWGqzF26AzXTFFdS0xYEcN4GrtziduBlh4TYRxk99JR13 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+9C0i ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+uowK ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+gHaa ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ryoo ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+iKsh ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+/cut ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7OvN ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w6ig ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+msEF ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+XvRo ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+w34s ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mbRk ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9p2 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+PEAQ ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+D+Ty ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GsLz ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+EaKa ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+MNQm ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+VVkV ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+4d21 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+x6VH ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+fEGT ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+64zM ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+Hf/a ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+e9Tu ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+W4nD ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+7SUP ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+leL+ ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+Tmwt ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8ymk ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+20L0 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+ym2A ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mfe5 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+A+Py ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+k/Dx ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+NeCF ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+mSK1 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+V747 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+M0xG ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+QiEP ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+CCTj ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+8lkb ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+GXOo ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+C094 ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eDPM ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+F+xp ZAWROUVbRMFz1MjblIVX+C9nygSGeaMJ9KDosbUlQCBnOtWViPblov0lViwk4QEHAAL+eNj zsFzc1023QU8xwlrhgvL0xXKxUmITV4y2VaSmEc/0RH/c4XAinNy955X6w9ejqXG9aYFjljLBtCHKXgyAZbQ48E4UPtPwPTCC3R1pFapfNAJMQ55kf /6x/6XVFPBw4okLcz6tVDLqn7dGiTEjzYgfQBwKXiIPrGg9EmBgwmQKWuJpOde+jbMD9kU1WmUoUWvTvPtXQ0= bitcount:4096 fingerprint: 13:96:fe:f0:f7:d7:5f:9c:d7:2a:da:72:8a:93:53:a6 ************************************** could not retrieve dsa key information ************************************** Now SSH should work.
Cisco Application Control Engine (ACE) Troubleshooting Guide 4. Try connecting to the ACE via SSH again by entering the following command:
[linux]$ ssh admin@192.168.0.210 Warning: Permanently added '192.168.0.210' (RSA) to the list of known hosts.
Password: Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 1985-2009 by Cisco S The copyrights to certain works contained herein are owned by other third parties and are used and distributed und Some parts of this software are covered under the GNU Public License. A copy of the license is available at http:/ ACE_module5/Admin#
5. Confirm the SSH session from the ACE CLI by entering the following command:
ACE_module5/Admin# show ssh session-info Session ID 6986 Remote Host 10.76.248.6 :42116 Active Time 0: 0:46
Troubleshooting KAL-AP
To troubleshoot KAL-AP related issues, follow these steps: 1. Make sure that KAL-AP is enabled under the management policy by entering the following commands:
ACE_module5/Admin# show running-config class-map Generating configuration.... class-map 2 match 3 match 4 match 5 match 6 match 7 match 8 match type management match-any MGMT_CLASS protocol http any protocol https any protocol icmp any protocol kalap-udp any <------- KAL-AP is enabled protocol ssh any protocol telnet any protocol snmp any
ACE_module5/Admin# show running-config policy-map MGMT_POLICY Generating configuration.... policy-map type management first-match MGMT_POLICY class MGMT_CLASS permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH
2. Verify that traffic from the Cisco Global Site Selector (GSS) is reaching the ACE module. KAL-AP statistics should get incremented.
ACE_module5/Admin# sh stats kalap +-----------------------------------------------------+ +---------------- KAL-AP(UDP) statistics -------------+ +-----------------------------------------------------+ Total Total Total Total Total Total Total Total Total bytes received bytes sent requests received responses sent requests successfully received responses successfully sent secure requests received secure responses sent requests with errors : : : : : : : : : 243956 184884 5100 5100 5100 5100 0 0 0
Troubleshooting KAL-AP
81

Total requests with parse errors : 0 Total response transfer errors : 0 -----------------------------------------------------
3. Allow secure KAL-AP requests, and add the GSS IP address and the shared secret to the ACE by entering the following commands:
ACE_module5/Admin# config ACE_module5/Admin(config)# kalap udp ACE_module5/Admin(config-kalap-udp)# ip address 192.168.10.52 encryption md5 cisco (GSS IP)
4. Display information about the load VIP by entering the following command:
ACE_module5/Admin# show kalap udp load vip 10.1.1.1 Error: Vip object not found! ACE_module5/Admin#
If the VIP object is not found while displaying the load value as shown above, check whether the VIP got downloaded in the configuration manager internal table by entering the following command:
ACE_module5/Admin# show cfgmgr internal table vip VIP-Id VIP-Addr Ctx-Id Flags --------------------------------------------------------------------------1 10.1.1.1 1 DATA_VALID, L3Rule_list :-->: 41: 42 Load Value: 255 Load Time stamp: Wed Apr 8 05:10:20 2009
This article describes security access control lists (ACLs) in the ACE, how to configure them, and troubleshooting steps to follow if you encounter problems with ACLs.
Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues Troubleshooting KAL-AP 82
Cisco Application Control Engine (ACE) Troubleshooting Guide ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of Security Access Control Lists 1.1 ACL Types and Uses 1.2 ACL Configuration Guidelines 1.2.1 ACL Entry Order 1.2.2 ACL Implicit Deny 1.2.3 Maximum Number of ACL Entries 2 Configuring ACLs 3 ACL-Related syslogs 4 Troubleshooting ACLs
Overview of Security Access Control Lists

An ACL consists of a series of statements called ACL entries that define the network traffic profile. Each entry permits or denies network traffic (inbound and outbound) from and to the parts of your network specified in the entry. Each entry also contains a filter element that is based on criteria such as the source address, the destination address, the protocol, and protocol-specific parameters such as ports and so on. An implicit deny-all entry exists at the end of each ACL, so you must configure an ACL on each interface that you want to permit connections. Otherwise, the ACE denies all traffic on the interface. ACLs allow you to control network connection setups rather than processing each packet. Such ACLs are commonly referred to as security ACLs. You can configure ACLs as parts of other features (for example, security, Network Address Translation (NAT), Contents 83
Cisco Application Control Engine (ACE) Troubleshooting Guide server load balancing (SLB), and so on). The ACE merges these individual ACLs into one large ACL called a merged ACL. The ACL compiler then parses the merged ACL and generates the ACL lookup mechanisms. A match on this merged ACL can result in multiple actions. Note: You can apply only one extended ACL to each direction (inbound or outbound) of an interface. You can also apply the same ACL on multiple interfaces. You can apply EtherType ACLs only in the inbound direction and only on Layer 2 interfaces.
ACL Types and Uses

You can configure the following two types of ACLs in the ACE: Extended?Control network access for IP traffic (Layer 3 and Layer 4) EtherType?Control network access for non-IP traffic on Layer 2 interfaces The ACE does not explicitly support standard ACLs. To configure a standard ACL, specify the destination address as any and do not specify ports in an extended ACL. For details about configuring an extended ACL, see the ?Configuring an Extended ACL? section.
ACL Configuration Guidelines

This section describes the guidelines to observe when you configure and use ACLs in your network. It contains the following topics: ACL Entry Order ACL Implicit Deny Maximum Number of ACL Entries ACL Entry Order An ACL consists of one or more entries. Depending on the ACL type, you can specify the source and destination addresses, the protocol, the ports (for TCP or UDP), the ICMP type, the ICMP code, or the EtherType as the match criteria. By default, the ACE appends each ACL entry at the end of the ACL. You can also indicate the location of each entry within an ACL by specifying a line number. The order of the entries is important. When the ACE decides whether to accept or refuse a connection, the ACE tests the packet against each ACL entry in the order in which the entries are listed. After it finds a match, the ACE does not check any more entries. For example, if you create an entry at the beginning of an ACL that explicitly permits all traffic, the ACE does not check any other statements in the ACL. Note: If there is a deny statement for packets coming to the Control Plane (CP), then the ACE skips to the next ACL entry. ACL Implicit Deny All ACLs have an implicit deny entry at the end of the ACL, so, unless you explicitly permit it, traffic cannot pass. For example, if you want to allow all users to access a network through the ACE except for those users with particular IP addresses, then you must deny the particular IP addresses in one entry and permit all other IP addresses in another entry. Maximum Number of ACL Entries ACLs are used in ACE as conventional access-groups, and also for use in class maps. The ACLs are converted into trees of different data structures called nodes, which are merged into a single tree instance that consists of nodes of one type called Policy Action Nodes (PANs). This merged tree is then transferred to the dataplane. A tree is created for each instance, and an instance is defined as a VLAN interface in either the input or output direction. Therefore, all ACLs that are applied to a given VLAN and a given direction contribute to the node usage for that instance.
Cisco Application Control Engine (ACE) Troubleshooting Guide The ACE supports a maximum of 256,000 Policy Action Nodes (PANs) entries. Some ACLs use more memory than others, such as an ACL that uses large port number ranges or overlapping networks (for example, one entry specifies 10.0.0.0/8 and another entry specifies 10.1.1.0/24). Depending on the type of ACL, the actual limit that the ACE can support may be less than 256,000 PANs entries. If you use object groups in ACL entries, you enter fewer actual ACL entries, but the same number of expanded ACL entries as you did when you entered entries without object groups. Expanded ACL entries count toward the system limit. To view the number of expanded ACL entries in an ACL, use the show access-list name command. If you exceed the memory limitations of the ACE, it generates a syslog message and increments the Download Failures counter in the output of the show interface vlan number command. The configuration remains in the running-config file and the interface stays enabled. The ACL entries stay the same as they were before the failing configuration was attempted. For example, if you add a new ACL with ten entries, but the addition of the sixth entry fails because the ACE runs out of memory, the ACE removes the five entries that you successfully entered. Note: You must allocate sufficient ACL memory resources for each virtual context in the ACE. The ACE does not generate a syslog if you exceed the maximum number of ACL entries.
Configuring ACLs
You can configure ACLs in one of two ways: Using the access-list command in configuration mode Using the match access-list command in a Layer 3 and Layer 4 class map You can permit or deny network connections based on the IP protocol, source and destination IP addresses, and TCP or UDP ports. To configure a non-ICMP extended ACL, enter the following command: access-list name [line number] extended {deny | permit} {protocol {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name} [operator port1 [port2]] {any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name} [operator port3 [port4]]} | {object-group service_obj_grp_name} {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name} You can also permit or deny network connections based on the ICMP type (for example, echo, echo-reply, unreachable, and so on). To configure an ICMP extended ACL, enter the following command: access-list name [line number] extended {deny | permit} {icmp {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name} [icmp-type code [operator code1 [code2]]]} | {object-group service_obj_grp_name} {any | host src_ip_address | src_ip_address netmask | object-group net_obj_grp_name} {any | host dest_ip_address | dest_ip_address netmask | object-group net_obj_grp_name} You can configure an ACL that controls traffic based on its EtherType. An EtherType is a subprotocol identifier. EtherType ACLs support Ethernet V2 frames; they do not do not support 802.3-formatted frames. To configure an Ethertype ACL, enter the following command: access-list name ethertype {deny | permit} {any | bpdu | ipv6 | mpls} Note: You can configure an EtherType ACL on a Layer 2 interface in the inbound direction only. If you are operating the ACE in bridge mode, be sure to configure an ACL on all interfaces that permit BPDUs. Otherwise, a bridge loop may result.
Cisco Application Control Engine (ACE) Troubleshooting Guide For example, to configure an extended ACL to permit all IP traffic from any source IP address and that is destined to any IP address on interface VLAN 200, enter the following commands:
ACE_module5/Admin(config)# access-list ACL1 extended permit ip any any ACE_module5/Admin(config)# interface vlan 200 ACE_module5/Admin(config-if)# ip address 192.168.1.1 255.255.255.0 ACE_module5/Admin(config)# access-group input ACL1
You can apply an ACL to all interfaces in a context at once, subject to the following conditions: No interface in the context has an ACL applied to it. You can globally apply one Layer 2 and one Layer 3 ACL in the inbound direction only. On Layer 2 bridged-group virtual interfaces (BVIs), you can apply both Layer 3 and Layer 2 ACLs. On Layer 3 virtual LAN (VLAN) interfaces, you can apply only Layer 3 ACLs. In a redundant configuration, the ACE does not apply a global ACL to the FT VLAN. For example, to apply ACL1 to all interfaces in the Admin context, enter the following command in configuration mode:
ACE_module5/Admin(config)# access-group input ACL1
The syntax of the match access-list command is as follows:

match access-list acl_name
To configure an ACL match statement in a class map, enter the following commands:
ACE_module5/Admin(config)# class-map match-any L4_CLASS ACE_module5/Admin(config-cmap)# match access-list ACL1 ACE_module5/Admin(config-cmap)# exit ACE_module5/Admin(config)# policy-map multi-match L4_POLICY ACE_module5/Admin(config-pmap)# class L4_CLASS ACE_module5/Admin(config-pmap-c)#
For more details about ACLs and how to configure them, see the Cisco Application Control Engine Module Security Configuration Guide.
ACL-Related syslogs
When a packet matches an ACL entry, a syslog message is generated based on the following rules: All ACL deny entries generate a syslog message unless logging is explicitly disabled using the no logging enable command in configuration mode. An ACL permit entry generates a syslog message only if logging is enabled using the logging enable command in configuration mode. All implicit deny entries generate the default deny syslog (%ACE-4-106023). To minimize syslog message generation, the ACE uses the flow cache as follows: 1. For the first packet hit on an ACL entry, the ACE generates a syslog and caches the flow (5-tuple) in the connection table. 2. For subsequent hits on the same ACL entry, the ACE checks the cache. If it finds the flow in the cache, the ACE increments a hit counter for this entry in the cache and does not generate a syslog. 3. After some time (the default is 300 seconds, which is configurable in the ACL entry definition in the CLI as the interval_secs option), the ACE generates a syslog and sets the hit count to 0.
Cisco Application Control Engine (ACE) Troubleshooting Guide 4. However, if at the expiry of the above time, the hit count is 0, the ACE deletes the cache entry silently. So by default, a cache entry is aged out 600 seconds after the last hit.
Troubleshooting ACLs
Many ACL issues manifest themselves by all traffic or only certain traffic being denied or permitted access to the ACE or out of the ACE. Remember that, initially, all traffic to the ACE is denied until you permit traffic using an ACL. Every ACL contains an implicit deny at the end of it, so only traffic that you explicitly permit will have access to the ACE. To troubleshoot ACLs, follow these steps: 1. Verify that your ACL configuration is correct for your network application. Make any required changes to the running-config file, and then test the configuration. If it is satisfactory, save it to the startup-config file using the copy runnning-config startup-config command. For example, to display the ACLs that you have configured in your ACE, enter the following command:
ACE_module5/Admin# show running-config access-list Generating configuration....
access-list ACL1 remark This ACL permits any IP traffic from any source going to any destination except for ICMP t 192.168.12.15 255.255.255.192. access-list ACL1 line 8 extended permit ip any any access-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=0) [0x access-list ANYONE line 8 extended permit ip any any
To verify that the configured ACLs are applied to the correct interfaces and in the right directions (input or output), enter the following command:
ACE_module5/Admin# show running-config interface Generating configuration.... interface vlan 100 ip address 10.2.1.1 255.255.255.0 access-group input ANYONE access-group output ANYONE no shutdown interface vlan 200 ip address 192.168.1.1 255.255.255.0 access-group input ACL1 service-policy input MGMT_POLICY no shutdown
2. Verify that you have allocated sufficient resources for ACLs. To display the allocated resources in your ACE, enter the following command:
ACE_module5/Admin# show resource usage Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------Context: Admin conc-connections 10 18 0 8000000 0 mgmt-connections 2 10 0 100000 0 proxy-connections 584 590 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 880 16194 0 625000000 0 throughput 880 12606 0 500000000 0 mgmt-traffic rate 0 3588 0 125000000 0

connection rate ssl-connections rate mac-miss rate inspect-conn rate acl-memory sticky regexp syslog buffer syslog rate Context: C1 conc-connections mgmt-connections proxy-connections xlates bandwidth throughput mgmt-traffic rate connection rate ssl-connections rate mac-miss rate inspect-conn rate acl-memory sticky regexp syslog buffer syslog rate 1 0 0 0 33448 0 0 188416 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21 0 16 0 33448 0 0 188416 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7858944 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7858944 0 0 0 0 1000000 5000 2000 6000 70749384 0 1048576 4194304 100000 8000000 100000 1048574 1048574 625000000 500000000 125000000 1000000 5000 2000 6000 70749384 0 1048576 4194304 100000
0 0 0 0 0 <------- ACL memory resource allocat 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
For example, to allocate a 10 percent minimum and a maximum of unlimited resources for ACL memory in the Admin virtual context, enter the following commands:
ACE_module5/Admin(config)# resource myclass ACE_module5/Admin(config-resource)# limit-resource acl-memory minimum 10 maximum unlimited ACE_module5/Admin(config-resource)# exit ACE_module5/Admin(config)# context Admin ACE_module5/Admin(config-context)# member myclass
3. Display the details of an individual ACL by using the show access-list acl_name detail command. This command displays every entry in the specified ACL, the hit counts for each entry, and a 32-bit hexadecimal MD5-hash value that the ACE computes from the access-list command immediately when you configure an ACL. The ACE includes this hash value in deny syslog messages (106023) to help you identify the ACL entry that caused the deny syslog. For example to display the details of the ACL1 access control list, enter the following command:
ACE_module5/Admin# show access-list ACL1 detail
access-list:ACL1, elements: 2, status: ACTIVE remark : This ACL permits any IP traffic from any source going to any destination except for ICMP traffic origin access-list ACL1 line 8 extended permit ip any any (hitcount=9) [0x894c1008] <------- 32-bit hexadecimal MD5-hash access-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=15) [0
The format of the deny syslog message is as follows:
%ACE-4-106023: Deny protocol number | name src incoming-interface:src-ip dst outgoing-interface:dst-ip by access-g An IP packet was denied by the ACL.
Explanation: This message displays even if you do not have the log option enabled for an ACL. If a packet hits an input ACL, the outgoing interface will not be known. In this case, the ACE prints the outgoing interface as undetermined. The source IP and destination IP addresses are the unmapped and mapped addresses for the input and output ACLs, respectively, when used with NAT.
88
Cisco Application Control Engine (ACE) Troubleshooting Guide Recommended Action: If messages persist from the same source address, messages may indicate a foot-printing or port-scanning attempt. Contact the remote host administrators. An ACL merged list is a large ACL that the CP compiles from multiple security ACL entries and policies. When the ACE executes an ACL merged list, it performs multiple actions on a flow that matches the merged list. 4. Display the actions that the ACE will perform on a flow by entering the show acl-merge merged-list command. For example, to display the merged list for VLAN 100, enter the following command:
ACE_module5/Admin# show acl-merge merged-list vlan 100 in non-redundant All ACEs in merged list 2 Total:18 Non-redundant:12 Priority:1000, Lineno:0, ACE-id:211 Action:PERMIT, Path-id:0x81/0x0/0x0:6/0[6/0] [6/0] Pmap:0x5, Log:FALSE/FALSE[FALSE][FALSE], Interval:0/0[0][0] Hash1:0x0 Hash2:0x0 Generated:TRUE, need-to-add-in-comp:NO_ACT_NEEDED, redundant:FALSE Parent:: feature:SECURITY ace-lineno:5 ACL priority:0[G:0,P:0,C:0,ACL:0] Parent:: feature:TO CP ace-lineno:2 ACL priority:16779265[G:0,P:1,C:8,ACL:1] Feature:SECURITY Policy:1[1][1] sec-level:0x0 Intratype:SKIP Feature:TO CP Policy:1[1][1] sec-level:0x0 Intratype:TERMINATE Intertype:TERMINATE IP address SRC:161.44.0.0/255.255.0.0 DST:10.86.215.134/255.255.255.255 Ports SRC:RANGE 0 65535 DST:RANGE 22 22 Protocol:6 Hit Count:0 Active:TRUE Timerange:0 . . . Feature:SECURITY Policy:0[0] sec-level:0x0 Intratype:TERMINATE . . . Feature:SLB Policy:14[14] sec-level:0x0 Intratype:TERMINATE . . . Feature:SRC NAT Policy:2[2] sec-level:0x0 Intratype:TERMINATE . . .
5. If the acl-memory Denied counter in the output of the show resource usage command is incrementing and the Peak (ACL) memory counter has not exceeded the Max Allocated ACL memory counter, the problem may lie with one of the nodes in the ACL merge tree. The ACL merge tree contains several different kinds of nodes (see the example output below), each of a different size and each with a maximum limit. If you allocate a minimum of 10 percent of the ACE resources to ACL memory, the ACE will guarantee 10% of the maximum number of each node. If your configuration causes the ACE to exceed the maximum value of one of these nodes, the ACL resource allocation will fail and the acl-memory Denied counter will increment. To monitor the ACL merge tree node usage in the ACE, enter the following command:
ACE_module5/Admin# show np 1 access-list resource ACL Tree Statistics for Context ID: Admin ========================================= ACL memory max-limit: None ACL memory guarantee: 10.00 % MTrie nodes(used/guaranteed/max-limit): 43 / 26214 / 262143 (compressed) <-------| 3 / 2199 / 21999 (uncompressed) <--------|

Leaf Head nodes (used/guaranteed/max-limit): | 39 / 26214 / 262143 <--------------------|---- Maximum number of available nodes in the ACE Leaf Parameter nodes (used/guaranteed/max-limit):| 594 / 52428 / 524288 <-------------------| Policy action nodes used: 153 memory consumed: 23776 bytes resource-limited 4896 bytes other 28672 bytes total . min-guarantee: 7861043 bytes total, 0 % consumed. max-limit: 78610432 bytes total, 0 % consumed. ACL Tree Statistics for the linecard ==================================== MTrie nodes(used): 43 (compressed) 3 (uncompressed) <--------------| (shared): 235929 (compressed) 19800 (uncompressed) | Leaf Head nodes (used/shared): 39 / 235929 <-----------------------|---- Number of used nodes in the ACE Leaf Parameter nodes (used/shared): 594 / 471860 <-----------------| Policy action nodes (used/shared): 153 / 261990 <------------------|
You can calculate the percentage of use for each node type by dividing the used nodes value by the maximum number of nodes and multiplying the result by 100. If any of these percentages exceeds the maximum value of allocated ACL memory for the context, increase the max value of allocated ACL memory using the limit-resource acl-memory command in resource class configuration mode so that that value is greater than or equal to the highest used nodes percentage that you calculated. Alternatively, if you are approaching the limits of ACL resource capacity, you may consider consolidating your ACL configuration. If the ACL nodes are depleted while the ACE is downloading ACL configurations for an interface, the complete ACL merged list for that interface is deleted and no traffic flows through that interface. The ACE increments the download failure counter in the output of the show interface command and the ACE logs a system message from the configuration manager. 6. To trace a packet through a specific ACL, enter the following command:
ACE_module5/Admin# show np 1 access-list trace vlan 130 in protocol 1 source 172.27.16.23 2000 destination 192.168 Root 0x2c01b00 Src Mtrie (0) offset 1 curr 0x2c01b00 child 0x0 leaf 0x10a840 Dst Mtrie (0) offset 2 curr 0x10a840 child 0x0 leaf 0x3c01330 proto ICMP head node 0x4004880 proto node 0x4004880 src op range port 0/65535 dst op range port 0/65535 lineno 112000 inner match line#:112000 inner match line#: 112000 packet matched priority 112000 action node 0x4c02460 Action Leaf-node version+aceid 0x99 (version 0 ace_id 153 dirty no) action_flag 0x10 (permit no log no punt_to_cp no capture no bridge yes) path ID 0x0 src nat 0x0 dst nat 0x0 vserver 0x0 fixup 0x0 TCP conn 0x0 AAA 0x0 Websense 0x0 QOS Policer 0x0 Syslog Info 0 Hitcount 130426 Syslog info: idx:[153:0] name_idx:[0:0] hash1:0x0 hash2:0x0 name_len:0 invalid Number of DRAM access: 6 (2 mtrie 4 non-mtrie)
90
Cisco Application Control Engine (ACE) Troubleshooting Guide This article describes ACE network address translation (NAT), how to configure it, and how to troubleshoot issues with NAT that you may encounter. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Network Address Translation 2 NAT Configuration Guidelines and Restrictions 3 Configuring Dynamic NAT and PAT 4 Configuring Server-Farm Based Dynamic NAT 5 Configuring Static NAT and Port Redirection 6 Configuring SNAT with Cookie and Load Balancing 7 Troubleshooting ACE NAT and PAT
Contents
91
Overview of ACE Network Address Translation

You can configure the ACE to translate a client source IP address to a routable address in the server's network. This process is called source NAT (SNAT). If you want to preserve the client source IP address, do not configure SNAT. You can also configure the ACE to translate the private address of a server to a global IP address that is accessible to clients. This process is called destination NAT (DNAT) and protects the server by hiding its real IP address from the Internet. Besides translating IP addresses, you can configure the ACE to translate TCP and UDP ports. This process is called port address translation (PAT). The ACE provides the following types of NAT and PAT: Interface-based dynamic NAT Interface-based dynamic PAT Server farm-based dynamic NAT Static NAT Static port redirection
NAT Configuration Guidelines and Restrictions

When you configure NAT and PAT on your ACE, keep in mind the following NAT and PAT guidelines and restrictions: If a packet egresses an interface that you have not configured for NAT, the ACE transmits the packet untranslated. You can configure dynamic NAT or static NAT as an input service policy only; you cannot configure it as an output service policy. When you remove a traffic policy from the last VLAN interface on which you applied the service policy, the ACE automatically resets the associated service-policy statistics. The ACE performs this action to provide a new starting point for the service-policy statistics the next time that you attach a traffic policy to a specific VLAN interface.
Configuring Dynamic NAT and PAT

Dynamic NAT is typically used for SNAT. When you configure dynamic NAT and PAT, be sure to configure an interface for the client-side VLAN and an interface for the server-side VLAN. The following SNAT configuration example shows the commands that you use to configure dynamic NAT and PAT on your ACE. In this SNAT example, packets that ingress the ACE from the 192.168.12.0 network are translated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command. The pat keyword indicates that ports higher than 1024 are also translated. Note: If you are operating the ACE in one-arm mode, omit the client-side interface VLAN 100 and configure the service policy on interface VLAN 200.
access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http
Overview of ACE Network Address Translation
92

class-map match-any NAT_CLASS match access-list NAT_ACCESS policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200 interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown interface vlan 200 mtu 1500 ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown
Configuring Server-Farm Based Dynamic NAT

The following SNAT configuration example shows the commands that you use to configure server farm-based dynamic NAT on your ACE. In this SNAT example, real server addresses on the 172.27.16.0 network are translated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command. Note: If you are operating the ACE in one-arm mode, omit interface VLAN 100 and configure the service policy on interface VLAN 200.
access-list NAT_ACCESS line 10 extended permit tcp 192.168.12.0 255.255.255.0 172.27.16.0 255.255.255.0 eq http rserver SERVER1 ip address 172.27.16.3 inservice rserver SERVER2 ip address 172.27.16.4 inservice serverfarm SFARM1 rserver SERVER1 inservice rserver SERVER2 inservice class-map type http loadbalance match-any L7_CLASS match http content .*cisco.com class-map match-any NAT_CLASS match access-list NAT_ACCESS policy-map type loadbalance http first-match L7_POLICY class L7_CLASS serverfarm SFARM1 nat dynamic 1 vlan 200 serverfarm primary policy-map multi-match NAT_POLICY class NAT_CLASS loadbalance policy L7_POLICY loadbalance vip inservice interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown interface vlan 200 mtu 1500
Configuring Dynamic NAT and PAT
93

ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 no shutdown
Configuring Static NAT and Port Redirection

The following DNAT configuration example shows those sections of the running configuration that are related to the commands necessary to configure static NAT and port redirection on your ACE. Typically, this configuration is used for DNAT, where HTTP packets that are destined to 192.0.0.0/8 and ingress the ACE on VLAN 101 are translated to 10.0.0.0/8 and port 8080. In this example, the servers are hosting HTTP on custom port 8080.
access-list acl1 line 10 extended permit tcp 10.0.0.0 255.0.0.0 eq 8080 any class-map match-any NAT_CLASS match access-list acl1 policy-map multi-match NAT_POLICY class NAT_CLASS nat static 192.0.0.0 255.0.0.0 80 vlan 101 interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown interface vlan 101 mtu 1500 ip address 172.27.16.100 255.255.255.0 no shutdown
Configuring SNAT with Cookie and Load Balancing

The following configuration example shows those commands necessary to configure SNAT (dynamic NAT) with cookie load balancing. Any source host that sends traffic to the VIP 20.11.0.100 is translated to one of the free addresses in the NAT pool in the range 30.11.100.1 to 30.11.200.1, inclusive. If you want to use PAT instead of NAT, replace nat dynamic 1 vlan 2021 with nat dynamic 2 vlan 2021 in the L7SLBCookie policy map.
server host http ip address 30.11.0.10 inservice serverfarm host httpsf rserver http inservice class-map 2 match class-map 3 match match-any vip4 virtual-address 20.11.0.100 tcp eq www type http loadbalance match-any L7SLB_Cookie http cookie JG cookie-value ?.*?
policy-map type loadbalance first-match L7SLB_Cookie class L7SLB_Cookie serverfarm httpsf policy-map multi-match L7SLBCookie class vip4 loadbalance vip inservice loadbalance L7SLB_Cookie nat dynamic 1 vlan 2021 interface vlan 2020 ip address 20.11.0.2 255.255.0.0

alias 20.11.0.1 255.255.0.0 peer ip address 20.11.0.3 255.255.0.0 service-policy input L7SLBCookie no shutdown interface vlan 2021 ip address 30.11.0.2 255.255.0.0 alias 30.11.0.1 255.255.0.0 peer ip address 30.11.0.3 255.255.0.0 fragment min-mtu 68 nat-pool 2 30.11.201.1 30.11.201.1 netmask 255.255.255.255 pat nat-pool 3 30.11.202.1 30.11.202.3 netmask 255.255.255.255 nat-pool 1 30.11.100.1 30.11.200.1 netmask 255.255.255.255 no shutdown
Troubleshooting ACE NAT and PAT

To verify your NAT and PAT configurations and make any necessary corrections, follow these steps: 1. Display your NAT and PAT configurations by entering the following commands:
ACE_module5/Admin# show running-config class-map class-map match-any L4_CLASS 2 match access-list ACL1 ACE_module5/Admin# show running-config policy-map policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200 ACE_module5/Admin# show service-policy NAT_POLICY Status : ACTIVE ----------------------------------------Interface: vlan 100 service-policy: NAT_POLICY class: NAT_CLASS nat: nat dynamic 1 vlan 200 curr conns : 0 , hit count : 0 dropped conns : 0 client pkt count : 0 , client byte count: 0 server pkt count : 0 , server byte count: 0 conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 ACE_module5/Admin# show running-config interface interface vlan 100 ip Address 192.168.12.2 mtu 1500 service-policy input NAT_POLICY no shutdown interface vlan 200 ip address 172.27.16.2 255.255.255.0 mtu 1500 access-group input acl1 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown
Configuring SNAT with Cookie and Load Balancing
95
Cisco Application Control Engine (ACE) Troubleshooting Guide 2. Use the show xlate command to verify that dynamic NAT and PAT, and static NAT and port redirection, are taking place properly. Dynamic NAT Example The following example output of the show xlate command shows dynamic NAT (SNAT in this example). When you use Telnet from IP address 172.27.16.5 in VLAN 2020, the ACE translates it to IP address 192.168.100.1 in VLAN 2021.
host1/Admin# show xlate global 192.168.100.1 192.168.100.10 NAT from vlan2020:172.27.16.5 to vlan2021:192.168.100.1 count:1
Dynamic PAT Example The following example shows dynamic PAT. When you use Telnet from IP address 172.27.16.5 port 38097 in VLAN 2020, the ACE translates it to IP address 192.168.201.1 port 1025 in VLAN 2021.
host1/Admin# show xlate TCP PAT from vlan2020:172.27.16.5/38097 to vlan2021:192.168.201.1/1025
Static NAT Example The following example shows static NAT. The ACE maps real IP address 172.27.16.5 to IP address 192.168.210.1.
host1/Admin# show xlate NAT from vlan2020:172.27.16.5 to vlan2021:192.168.210.1 count:1 host1/Admin# show conn total current connections : 2 conn-id dir prot vlan source destination state ----------+---+----+----+----------------+----------------+----------+ 7 in TCP 2020 172.27.16.5 192.168.100.1 ESTAB 6 out TCP 2021 192.168.100.1 192.168.210.1 ESTAB
Static Port Redirection (Static PAT) Example The following example shows static port redirection (DNAT in this example). A host at IP address 192.168.0.10:37766 uses Telnet to connect to IP address 192.168.211.1:3030 on VLAN 2021 on the ACE. The ACE maps IP address 172.27.0.5:23 on VLAN 2020 to IP address 192.168.211.1:3030 on VLAN 2021.
host1/Admin# show xlate TCP PAT from vlan2020:172.27.0.5/23 to vlan2021:192.168.211.1/3030 Mar 24 2006 20:05:41 : %ACE-7-111009: User 'admin' executed cmd: show xlate host1/Admin# show conn

total current connections : 2 conn-id dir prot vlan source destination state ----------+---+----+----+------------------+------------------+------+ 6 in TCP 2021 192.168.0.10:37766 192.168.211.1:3030 ESTAB 7 out TCP 2020 172.27.0.5:23 192.168.0.10:1025 ESTAB
3. To display the NAT policy and pool information for the current context, enter the show nat-fabric command. The syntax of this command is as follows: show nat-fabric {policies | src-nat policy_id mapped_if | dst-nat static_xlate_id | nat-pools | implicit-pat| global-static} policies -- Displays the NAT policies. src-nat policy_id mapped_if -- Displays the specified source NAT policy information. To obtain the values for the policy_id and mapped_if arguments, view the policy_id and mapped_if fields displayed by the show nat-fabric policies command. dst-nat static_xlate_id -- Displays the static address translation for the specified static XLATE ID. To obtain the value for the static_xlate_id argument, view the static_xlate_id field displayed by the show nat-fabric policies command. nat-pools -- Displays NAT pool information for a dynamic NAT policy. implicit-pat -- Displays the implicit PAT policies. global-static -- Displays global static NAT information when the static command in global configuration mode is configured.
ACE_module5/Admin# show nat-fabric policies Nat objects: NAT object Hash Bucket: 9 NAT object ID:2 mapped_if:8 policy_id:1 type:DYNAMIC nat_pool_id:4 Pool ID:4 PAT:1 pool_id:1 mapped_if:8 Ref_count:1 ixp_binding:in all IXPs lower:172.27.16.15 upper:172.27.16.24 Bitmap-ID:40 List of NAT object IDs: 2
This article describes ACE health monitoring (probes), how to configure it, and how to troubleshoot issues with probes that you may encounter. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE NAT and PAT 97
Cisco Application Control Engine (ACE) Troubleshooting Guide Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Health Monitoring 1.1 Configuring Probes 1.2 Example of a Probe Configuration 2 Troubleshooting ACE Health Monitoring 2.1 Troubleshooting an HTTP Probe Error 2.2 Troubleshooting an HTTPS Probe Error 2.3 Troubleshooting an SNMP Probe Issue 2.4 Using the Last Status Code Field
Overview of ACE Health Monitoring

This section describes health monitoring on the ACE. The ACE uses probes that you configure to track the state of a server. By default, no probes are configured in the ACE. Also referred to as out-of-band (OOB) health monitoring, the ACE verifies the server response to a probe or checks for any network problems that can prevent a client from reaching a server. Based on the Contents 98
Cisco Application Control Engine (ACE) Troubleshooting Guide server response, the ACE can place the server in or out of service and can make reliable load-balancing decisions. You can also use health monitoring to detect failures for a gateway or a host in high-availability (redundant) configurations. For more information, see the Cisco Application Control Engine Module Administration Guide. The ACE evaluates the health of a server by marking the probes as follows: Passed?The server returns a valid response. Failed?The server fails to provide a valid response to the ACE and the ACE is unable to reach a server for a specified number of retries. By configuring the ACE for health monitoring, the ACE sends active probes periodically to determine the server state. The ACE supports 4096 unique probe configurations, which includes ICMP, TCP, HTTP, and other predefined health probes. The ACE can execute only up to 200 concurrent scripted probes at a time. The ACE also allows the opening of 2048 sockets simultaneously. You can associate the same probe with multiple real servers or server farms. Each time that you use the same probe again, the ACE counts it as another probe instance. You can allocate a maximum of 16,000 probe instances.
Configuring Probes
You can configure health probes on the ACE to actively make connections and explicitly send traffic to servers. The probes determine whether the health status of a server passes or fails by the server's response. Configuring active probes is a three-step process: 1. Configure the health probe with a name, type, and attributes. 2. Associate the probe with one of the following: A real server. A real server and then associate the real server with a server farm. You can associate a single probe or multiple probes with real servers within a server farm. A server farm. All servers in the server farm receive probes of the associated probe types. 3. Place the real server or server farm in service.
Example of a Probe Configuration

The following example shows a running configuration that load balances DNS traffic across multiple real servers and transmits and receives UDP data that spans multiple packets. The configuration uses a UDP health probe.
access-list ACL1 line 10 extended permit ip any any probe udp UDP interval 5 passdetect interval 10 description THIS PROBE IS INTENDED FOR LOAD BALANCING DNS TRAFFIC port 53 send-data UDP_TEST rserver host SERVER1 ip address 192.168.10.45
Overview of ACE Health Monitoring
99

inservice rserver host ip address inservice rserver host ip address inservice SERVER2 192.168.10.46 SERVER3 192.168.10.47
serverfarm host SFARM1 probe UDP rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 inservice class-map match-all L4UDP-VIP_114:UDP_CLASS 2 match virtual-address 192.168.120.114 udp eq 53 policy-map type loadbalance first-match L7PLBSF_UDP_POLICY class class-default serverfarm SFARM1 policy-map multi-match L4SH-Gold-VIPs_POLICY class L4UDP-VIP_114:UDP_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_UDP_POLICY loadbalance vip icmp-reply nat dynamic 1 vlan 120 connection advanced-options 1SECOND-IDLE interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting ACE Health Monitoring

This section describes how to troubleshoot two common probe configuration issues.
Troubleshooting an HTTP Probe Error

In this first scenario, you have configured an HTTP probe, but the real server's health status is displayed as FAILED and the Last disconnect err field indicates that an invalid status code was received as displayed in the show probe detail command output. You have checked your server and it is up and running. A packet capture on the server also shows that everything is fine. Where is the issue? 1. Display the probe status details by entering the following command:
ACE_module5/Admin# show probe detail probe type state : HTTP_PROBE : HTTP : ACTIVE

description : ---------------------------------------------port : 80 address : 0.0.0.0 addr type : interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : send data : ------------------ probe results -----------------associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+-----rserver : SERVER1 192.168.10.45 80 -2 2 0 FAILED Socket state No. Passed states No. Probes skipped No. Out of Sockets Last disconnect err Last probe time Last fail time Last active time : : : : : : : : CLOSED 0 No. Failed states : 1 0 Last status code : 200 <------- Last status code from server 0 No. Internal error: 0 Received invalid status code <------Tue Apr 7 16:17:26 2009 Tue Apr 7 16:17:16 2009 Never
The Last disconnect err field indicates that the ACE received an invalid status code. This error means that you have has not configured the expect status command for the probe. 2. Confirm this finding by entering the following command:
ACE_module5/Admin# show running-config probe Generating configuration.... probe http HTTP_PROBE interval 10 passdetect interval 10 open 1
3. Correct the problem by entering the following commands:

ACE_module5/Admin# config Enter configuration commands, one per ACE_module5/Admin(config)# probe http ACE_module5/Admin(config-probe-http)# ACE_module5/Admin(config-probe-http)#
line. End with CNTL/Z. HTTP_PROBE expect status 200 200 <------- 200 indicates the 200 OK message from the ser end
4. Confirm the configuration by entering the following command:

ACE_module5/Admin# show running-config probe Generating configuration.... probe http HTTP_PROBE interval 10 passdetect interval 10 expect status 200 200 open 1
5. Display the probe status details again and observe that the server health status value is SUCCESS by entering the following command: Troubleshooting an HTTP Probe Error 101

ACE_module5/Admin# show probe HTTP_PROBE detail probe : HTTP_PROBE type : HTTP state : ACTIVE description : ---------------------------------------------port : 80 address : 0.0.0.0 addr type : interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : send data : ------------------ probe results -----------------associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+-----rserver : SERVER1 192.168.10.45 80 -24 15 9 SUCCESS Socket state No. Passed states No. Probes skipped No. Out of Sockets Last disconnect err Last probe time Last fail time Last active time : : : : : : : : CLOSED 1 No. Failed states : 1 0 Last status code : 200 0 No. Internal error: 0 - <------- No error indicated now. The probe is successful. Tue Apr 7 16:21:05 2009 Tue Apr 7 16:17:16 2009 Tue Apr 7 16:20:05 2009
Troubleshooting an HTTPS Probe Error

In addition to the methods for Troubleshooting an HTTP Probe Error, use SSL statistics to troubleshoot HTTPS probe failures. HTTPS probe traffic runs in the Admin virtual context so view the output of the show stats crypto client command in that context.
Troubleshooting an SNMP Probe Issue

In this scenario, you have configured an SNMP probe, but the Last disconnect err field indicates that the sum of the weights does not add up to the maximum weight value as displayed in the output of the show probe detail command. 1. Display the probe status details by entering the following command:
ACE_module5/test# show probe detail probe : SNMP_PROBE type : SNMP state : ACTIVE description : snmp probe ---------------------------------------------port : 161 address : 0.0.0.0 addr type : interval : 15 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 version : 2c community : test_comm oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000 ------------------ probe results -----------------associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+-----serverfarm : least-loaded, predictor least-loaded real : SERVER1[0]
Troubleshooting an HTTPS Probe Error
102

192.168.10.45 Socket state No. Passed states No. Probes skipped No. Out of Sockets Last disconnect err Last probe time Last fail time Last active time Server load : : : : : : : : : 161 -0 0 0 INIT
CLOSED 0 No. Failed states : 0 0 Last status code : 0 0 No. Internal error: 30 Sum of weights don't add up to max weight value <------- Error condition Never Never Never 16000 <------- Note the server load value
The reason for this error is that the weight command needs to be configured when you have multiple OIDs configured for a single probe and from those OIDs if you want to give priority to a specific OID. The sum of the weights should equal 16000 (see the Server Load field). For a single OID, the weight command does not have any significance. 2. Display the probe configuration by entering the following command:
ACE_module5/Admin# show running-config probe probe snmp SNMP_PROBE description snmp probe port 161 interval 15 passdetect interval 10 version 2c community TEST_COMM oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 <-------
In the above configuration, the weight is configured as 10000 for a single OID. The ACE is expecting another OID to be configured in the probe and the sum of both weights should equal 16000. The configuration is not complete and the ACE is expecting additional parameters in the probe configuration. Because there is not another OID in the configuration, the ACE is not able to calculate the load and that is why the "Sum of weights don't add up to max weight value" error message appears. 3. Resolve the issue by modifying the probe configuration as follows:
probe snmp SNMP_PROBE description test port 161 interval 15 passdetect interval 60 version 2c community test_comm oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 oid .1.3.6.1.2.1.4.10.0 type absolute max 1000000000 weight 6000 <------- 10000 + 6000 = 16000
4. Display the probe status details again by entering the following command:
ACE_module5/test# show probe SNMP_PROBE detail probe : snmp1
Troubleshooting an SNMP Probe Issue
103

type : SNMP state : ACTIVE description : snmp probe ---------------------------------------------port : 161 address : 0.0.0.0 interval : 15 pass intvl : 10 fail count: 3 recv timeout: 10 version : 2c community : test_comm oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000 oid string #2 : .1.3.6.1.2.1.4.10.0 type : ABSOLUTE max value : 1000000000 weight : 6000 threshold : 1000000000 ------------------ probe results -----------------associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+-----serverfarm : least-loaded, predictor least-loaded real : SERVER1[0] 192.168.10.45 161 -4143 0 4143 SUCCESS Socket state No. Passed states No. Probes skipped No. Out of Sockets Last disconnect err Last probe time Last fail time Last active time Server load : : : : : : : : : CLOSED 1 No. Failed states : 0 0 Last status code : 0 0 No. Internal error: 0 - <------- No error indicated now. The probe is successful. Mon Apr 6 09:12:54 2009 Never Sun Apr 5 15:57:28 2009 0
addr type : pass count : 3
Using the Last Status Code Field

Details regarding the last status code field can be provided for nontrivial probes. For example, in the case of scripted probe PROBENOTICE_PROBE, the status code 30001 means that the probe is successful and the value 30002 indicates an error in probe arguments. The last disconnect error for status code 30002 displays "Did not receive correct response from the server," but the actual issue is related to arguments in the probe configuration, which can be checked by looking at the script for the probe.
ACE_module5/Admin# show probe TEST detail probe : TEST type : SCRIPTED state : ACTIVE description : ---------------------------------------------port : 0 address : 0.0.0.0 addr type : interval : 15 pass intvl : 20 pass count : 3 fail count: 3 recv timeout: 10 script filename : PROBENOTICE_PROBE --------------------- probe results -------------------probe association probed-address probes failed passed health ------------------- ---------------+----------+----------+----------+------serverfarm : sf1 real : rs2[0] 23.0.0.5 4082 54 4028 SUCCESS Socket state No. Passed states No. Probes skipped No. Out of Sockets Last disconnect err : : : : : RESET 6 8 0 -
No. Failed states : 5 Last status code : 30001 <------- Indicates success No. Internal error: 0
Using the Last Status Code Field
104

Last probe time Last fail time Last active time : Wed Apr : Tue Apr : Tue Apr 8 04:44:41 2009 7 12:02:10 2009 7 12:03:45 2009
This article describes how to troubleshoot Layer 4 (L4) load balancing on the ACE. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE L4 Load Balancing 1.1 Classifying L4 Traffic for Server Load Balancing 1.2 Example of a Layer 4 Load-Balancing Configuration 2 Troubleshooting L4 Load Balancing on the ACE
Contents
105
Overview of ACE L4 Load Balancing

Load balancing at L4 involves selecting a server in a server farm to service a client request based on the VIP address and protocol in the request. You configure a class map to classify (match) interesting traffic arriving at the ACE and associate the class map with a policy map to perform an action on the traffic based on the classification. With L4 load balancing, the ACE selects a server based on the first packet it receives in a particular flow. See the "Overview of ACE Connection Handling" section in the Troubleshooting Connectivity article. For detailed information about ACE load balancing, see the Cisco Application Control Engine Module Server Load Balancing Configuration Guide.
Classifying L4 Traffic for Server Load Balancing

You classify inbound network traffic destined to or passing through the ACE based on a series of flow match criteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is of interest to you. A policy map defines a series of actions (functions) that you want applied to a set of classified inbound or outbound traffic. ACE L3 and L4 traffic policies support the following server load-balancing (SLB) traffic attributes: Source or destination IP address Source or destination port Virtual IP (VIP) address IP protocol The three major steps in the traffic classification process are as follows: 1. Create a class map using the class-map command and the associated match commands, which comprise a set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocol classifications. 2. Create a policy map using the policy-map command, which refers to the class maps and identifies a series of actions to perform based on the traffic match criteria. 3. Activate the policy map by associating it with a specific VLAN interface or globally with all VLAN interfaces using the service-policy command to filter the traffic received by the ACE. Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7 policies that the ACE uses for SLB. The figure also shows how you associate the various components of the SLB policy configuration with each other.
Overview of ACE L4 Load Balancing
106
Cisco Application Control Engine (ACE) Troubleshooting Guide Figure 1. SLB Flow Diagram
Example of a Layer 4 Load-Balancing Configuration

The following example shows a L4 load-balancing configuration:
access-list ACL1 line 10 extended permit ip any any rserver host SERVER1 ip address 192.168.252.245 inservice rserver host SERVER2 ip address 192.168.252.246 inservice rserver host SERVER3 ip address 192.168.252.247 inservice
107

rserver host SERVER4 ip address 192.168.252.248 inservice rserver host SERVER5 ip address 192.168.252.249 inservice rserver host SERVER6 ip address 192.168.252.250 inservice serverfarm host SFARM1 probe TCP_PROBE predictor roundrobin rserver SERVER1 weight 10 inservice rserver SERVER2 weight 20 inservice rserver SERVER3 weight 30 inservice serverfarm host SFARM2 probe TCP_PROBE predictor roundrobin rserver SERVER4 weight 10 inservice rserver SERVER5 weight 20 inservice rserver SERVER6 weight 30 inservice class-map match-all L4WEB_CLASS 2 match virtual-address 192.168.120.112 tcp eq www policy-map type loadbalance first-match LB_WEB_POLICY class class-default serverfarm SFARM1 backup SFARM2 policy-map multi-match L4WEB_POLICY class L4WEB_CLASS loadbalance vip inservice loadbalance policy LB_WEB_POLICY loadbalance vip icmp-reply active nat dynamic 1 VLAN 120 interface vlan 100 description Upstream VLAN_100 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4WEB_POLICY no shutdown ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of a Layer 4 Load-Balancing Configuration
108
Troubleshooting L4 Load Balancing on the ACE

To troubleshoot L4 load-balancing issues, follow these steps: 1. Ensure that your load-balancing configuration is correct and that the following conditions exist: Real servers have valid IP addresses and are in service. Servers are associated with server farms of the same type. A load-balancing policy exists with an associated server farm and is associated with a L4 multimatch policy. An L4 class map contains a valid match virtual-address command and is associated with the L4 multimatch policy map. The L4 policy is applied to the appropriate active interface using a service policy. A static route is configured for the server network. Use the following show commands: show running-config rserver show running-config serverfarm show running-config policy-map show running-config class-map show running-config interface show ip route 2. Check the ACE connectivity. See the "Troubleshooting Connectivity" section. 3. Verify that the L4 VIP class map is referenced in a L4 policy by entering the following command. Also, check the following fields: VIP address and port VIP state Hit count Dropped connections
ACE_module5/Admin# show service-policy L4WEB_POLICY detail Status : ACTIVE Description: ----------------------------------------Interface: vlan 100 service-policy: L4WEB_POLICY <------- L4 multimatch policy map class: L4WEB_CLASS <------- L4 VIP class map VIP Address: Protocol: Port: 192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port loadbalance: L7 loadbalance policy: LB_WEB_POLICY VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56 dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495 L7 Loadbalance policy : LB_WEB_POLICY class/match : class-default LB action : serverfarm: SFARM1 hit count : 0 <-------|-- Check these counters to see if they are increasing
Troubleshooting L4 Load Balancing on the ACE
109

dropped conns : 0 <-------|
The dropped conns counter under a VIP in the output of the show service policy detail command is incremented whenever the ACE discards a connection request destined to that VIP. There are several reasons why the ACE discards such connection requests. For example: If all the real servers in the server farm associated with the VIP go down, then the VIP will go down. So, all the incoming connections to that VIP are discarded. If the URL in a connection request to the VIP is unknown, then the connection request is discarded. If the server to which the ACE load balances the connection does not respond to the request, then, after the maximum number of retries, the ACE discards the connection. The dropped conns counter is cumulative and the value may comprise entries from any of the following show command counters: show stats loadbalance - Total Layer4 rejections - Total Layer7 rejections - Total Layer4 LB policy misses - Total Layer7 LB policy misses - Total times rserver was unavailable show stats connection - Total Connections Timed-out - Total Connections Failed The failures counter of the show serverfarm serverfarm_name command The Total drop decisions counter of the show stats inspect command
4. Verify that the L4 policy is applied as a service policy to an active interface by entering the following command:
ACE_module5/Admin# show running-config interface Generating configuration.... interface vlan 100 ip address 192.168.120.1 255.255.255.0 access-group input ACL1 access-group output anyone service-policy input L4WEB_POLICY no shutdown . . .
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also, check the IP address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail serverfarm : total rservers : active rservers: description : SFARM1, type: HOST 3 3 -
state : ACTIVE <------- Current state of the server farm predictor : ROUNDROBIN <------- Load-balancing method weight : autoadjust : MAXLOAD failaction : back-inservice : 40 partial-threshold : 40 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded ------------------------------------------connections----------real weight state current total failures ---+---------------------+------+------------+----------+----------+--------rserver: SERVER1 192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, an max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : , out-of-rotation count : bandwidth-rate-limit : , out-of-rotation count : retcode out-of-rotation count : load value : 0 rserver: SERVER2 192.168.252.246:0 20 INSERVICE 0 max-conns : 4000000 , out-of-rotation min-conns : 4000000 conn-rate-limit : , out-of-rotation bandwidth-rate-limit : , out-of-rotation retcode out-of-rotation count : load value : 0 rserver: SERVER3 192.168.252.247:0 30 INSERVICE 0 max-conns : 4000000 , out-of-rotation min-conns : 4000000 conn-rate-limit : , out-of-rotation bandwidth-rate-limit : , out-of-rotation retcode out-of-rotation count : load value : 0
0 count : 0 count : count : -
6. Check the L4 load-balance statistics by entering the following command:

ACE_module5/Admin# show stats loadbalance +------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0 Total Layer7 decisions : 0 Total Layer7 rejections : 0 Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in a redundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails to find the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a proper remote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the ID Map table was not created. Troubleshooting L4 Load Balancing on the ACE 111
This article describes how to diagnose and troubleshoot ACE L7 load-balancing issues. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Layer 7 Load Balancing 1.1 Load-Balancing Predictors 1.2 Classifying L7 Traffic for Server Load Balancing 2 Example of a L7 Load-Balancing Configuration 3 Troubleshooting Layer 7 Load Balancing
Contents
112
Overview of ACE Layer 7 Load Balancing

Layer 7 server load balancing (SLB) is the process that the load-balancing device uses to decide which server should fulfill a client request for an L7 service. For example, a client request may consist of a HyperText Transport Protocol (HTTP) GET for a web page or a File Transfer Protocol (FTP) GET to download a file. The job of the load balancer is to select the server that can successfully fulfill the client request and do so in the shortest amount of time without overloading either the server or the server farm as a whole. The ACE supports the load balancing of the following protocols: Generic protocols HTTP Remote Authentication Dial-In User Service (RADIUS) Reliable Datagram Protocol (RDP) Real-Time Streaming Protocol (RTSP) Session Initiation Protocol (SIP) Depending on the load-balancing algorithm?or predictor?that you configure, the ACE performs a series of checks and calculations to determine which server can best service each client request. The ACE bases server selection on several factors including the source or destination address, cookies, URLs, HTTP headers, or the server with the fewest connections with respect to load. For detailed information about ACE load balancing, see the Cisco Application Control Engine Module Server Load Balancing Configuration Guide.
Load-Balancing Predictors
The ACE uses the following predictors to select the best server to fulfill a client request: Application response?Selects the server with the lowest average response time for the specified response-time measurement based on the current connection count and server weight (if configured). Hash address?Selects the server using a hash value based on either the source or destination IP address or both. Use these predictors for firewall load balancing (FWLB). For more information about FWLB, see Configuring Firewall Load Balancing in the Cisco Application Control Engine Module Server Load-Balancing Configuration Guide (Software Version A2(1.0)). Hash content?Selects the server using a hash value based on a content string in the Trusted Third Parties (TTP) packet body. Hash cookie?Selects the server using a hash value based on a cookie name. Hash header?Selects the server using a hash value based on the HTTP header name. Hash URL?Selects the server using a hash value based on the requested URL. You can specify a beginning pattern and an ending pattern to match in the URL. Use this predictor method to load balance cache servers.
Overview of ACE Layer 7 Load Balancing
113
Cisco Application Control Engine (ACE) Troubleshooting Guide Least bandwidth?Selects the server that processed the least amount of network traffic based on the average bandwidth that the server used over a specified number of samples. Least connections?Selects the server with the fewest number of active connections based on the server weight. For the least-connections predictor, you can configure a slow-start mechanism to avoid sending a high rate of new connections to servers that you have just put into service. Least loaded?Selects the server with the lowest load based on information obtained from Simple Network Management Protocol (SNMP) probes. To use this predictor, you must associate an SNMP probe with it. Round-robin?Selects the next server in the list of real servers based on the server weight (weighted round-robin). Servers with a higher weight value receive a higher percentage of the connections. This is the default predictor. Note: The hash predictor methods do not recognize the weight value that you configure for real servers. The ACE uses the weight that you assign to real servers only in the least-connections, application-response, and round-robin predictor methods.

You classify inbound network traffic destined to or passing through the ACE based on a series of flow match criteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is of interest to you. A policy map defines a series of actions (functions) that you want applied to a set of classified inbound or outbound traffic. ACE traffic policies support the following server load-balancing (SLB) traffic attributes: Layer 3 and Layer 4 connection information?Source or destination IP address, source or destination port, virtual IP address, and IP protocol Layer 7 protocol information?Hypertext Transfer Protocol (HTTP) cookie, HTTP URL, HTTP header, Remote Authentication Dial-In User Service (RADIUS), Remote Desktop Protocol (RDP), Real-Time Streaming Protocol (RTSP), Session Initiation Protocol (SIP), and Secure Sockets Layer (SSL) The three major steps in the traffic classification process are as follows: 1. Create a class map using the class-map command and the associated match commands, which comprise a set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocol classifications. 2. Create a policy map using the policy-map command, which refers to the class maps and identifies a series of actions to perform based on the traffic match criteria. 3. Activate the policy map by associating it with a specific VLAN interface or globally with all VLAN interfaces using the service-policy command to filter the traffic received by the ACE. Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7 policies that the ACE uses for SLB. The figure also shows how to associate the various components of the SLB policy configuration with each other.
Figure 1. SLB Flow Diagram
Load-Balancing Predictors
114
Example of a L7 Load-Balancing Configuration

The following example shows a running configuration that includes multiple class maps and policy maps that define a traffic policy for SLB. In this configuration, when a server farm is chosen for a connection, the connection is sent to a real server based on one of several load-balancing predictors. The leastconns predictor method load balances connections to the server that has the lowest number of open connections.
access-list ACL1 line 10 extended permit ip any any probe tcp TCP interval 5 faildetect 2 passdetect interval 10 open 3 parameter-map type http PERSIST-REBALANCE persistence-rebalance <---------------------------- Enabled by default in the ACE appliance. Enabled by parameter-map type connection PRED-CONNS-UDP_CONN set timeout inactivity 300 rserver SERVER1
115

ip address 10.1.0.2 inservice rserver SERVER2 ip address 10.1.0.3 inservice rserver SERVER3 ip address 10.1.0.4 inservice rserver SERVER4 ip address 10.1.0.5 inservice rserver SERVER5 ip address 10.1.0.6 inservice rserver SERVER6 ip address 10.1.0.7 inservice rserver SERVER7 ip address 10.1.0.8 inservice rserver SERVER8 ip address 10.1.0.9 inservice serverfarm host PRED-CONNS predictor leastconns rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 inservice rserver SERVER4 inservice rserver SERVER5 inservice rserver SERVER6 inservice rserver SERVER7 inservice rserver SERVER8 inservice serverfarm host PRED-CONNS-UDP failaction purge predictor leastconns rserver SERVER1 inservice rserver SERVER2 inservice rserver SERVER3 probe ICMP inservice rserver SERVER5 inservice rserver SERVER6 inservice rserver SERVER7 inservice serverfarm host PREDICTOR probe TCP rserver SERVER1 inservice rserver SERVER2 inservice
Example of a L7 Load-Balancing Configuration
116

rserver SERVER6 inservice rserver SERVER7 inservice sticky http-cookie COOKIE_TEST STKY-GRP-43 cookie offset 1 length 999 timeout 30 replicate sticky serverfarm PREDICTOR class-map 2 match class-map 2 match class-map 2 match match-all L4PRED-CONNS-UDP-VIP_128:2222_CLASS virtual-address 192.168.120.128 udp eq 0 match-all L4PRED-CONNS-VIP_128:80_CLASS virtual-address 192.168.120.128 tcp eq www match-all L4PREDICTOR_117:80_CLASS virtual-address 192.168.120.117 tcp eq www
policy-map type loadbalance first-match L7PLBSF_PRED-CONNS_POLICY class class-default serverfarm PRED-CONNS policy-map type loadbalance first-match L7PLBSF_PRED-CONNS-UDP_POLICY class class-default serverfarm PRED-CONNS-UDP policy-map type loadbalance first-match L7PLBSF_PREDICTOR_POLICY class class-default sticky-serverfarm STKY-GRP-43 policy-map multi-match L4SH-Gold-VIPs_POLICY class L4PREDICTOR_117:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PREDICTOR_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-VIP_128:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-UDP-VIP_128:2222_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS-UDP_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE connection advanced-options PRED-CONNS-UDP_CONN interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting Layer 7 Load Balancing

To troubleshoot L7 load-balancing issues, use the following steps:
117
Cisco Application Control Engine (ACE) Troubleshooting Guide 1. Ensure that your load-balancing configuration is correct and that the following conditions exist: Real servers have valid IP addresses and are in service Servers are associated with server farms of the same type L7 load-balancing policy exists with an associated server farm and that the L7 load-balancing policy is associated with a L4 multimatch policy An L4 class map contains a valid match virtual-address command and is associated with the L4 multimatch policy map The L4 policy is applied to the appropriate active interface using a service policy A static route is configured for the server network Use the following show commands to verify your load-balancing configuration: show running-config rserver show running-config serverfarm show running-config policy-map show running-config class-map show running-config interface show ip route 2. Check the ACE connectivity. See the Troubleshooting Connectivity section. 3. Verify that the L7 load-balancing policy is referenced in the L4 policy by entering the following command. Also, check the following fields: VIP address, protocol, and port VIP state Hit count Dropped connections
ACE_module5/Admin# show service-policy L4WEB_POLICY detail
Status : ACTIVE Description: ----------------------------------------Interface: vlan 100 service-policy: L4WEB_POLICY class: L4WEB_CLASS VIP Address: Protocol: Port: 192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port loadbalance: L7 loadbalance policy: LB_WEB_POLICY <-------L7 load-balancing policy referenced in the L4 multimatch poli VIP Route Metric : 77 VIP Route Advertise : DISABLED VIP ICMP Reply : ENABLED VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56 dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495 L7 Loadbalance policy : LB_WEB_POLICY <------- L7 policy statistics class/match : class-default LB action : serverfarm: SFARM1 hit count : 0 <-------|-- Check these counters to see if they are increasing dropped conns : 0 <-------|
4. Verify that the L4 policy is applied as a service policy to an active interface by entering the following command:

ACE_module5/Admin# show running-config interface Generating configuration.... interface vlan 100 ip address 192.168.120.1 255.255.255.0 access-group input ACL1 access-group output anyone service-policy input L4WEB_POLICY no shutdown . . .
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also, check the IP address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail
serverfarm : SFARM1, type: HOST total rservers : 3 active rservers: 3 description : state : ACTIVE <------- Current state of the server farm predictor : ROUNDROBIN <------- Load-balancing method weight : autoadjust : MAXLOAD failaction : back-inservice : 40 partial-threshold : 40 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded ------------------------------------------connections----------real weight state current total failures ---+---------------------+------+------------+----------+----------+--------rserver: SERVER1 192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, an max-conns : 4000000 , out-of-rotation count : 0 min-conns : 4000000 conn-rate-limit : , out-of-rotation count : bandwidth-rate-limit : , out-of-rotation count : retcode out-of-rotation count : load value : 0 rserver: SERVER2 192.168.252.246:0 20 INSERVICE 0 max-conns : 4000000 , out-of-rotation min-conns : 4000000 conn-rate-limit : , out-of-rotation bandwidth-rate-limit : , out-of-rotation retcode out-of-rotation count : load value : 0 rserver: SERVER3 192.168.252.247:0 30 INSERVICE 0 max-conns : 4000000 , out-of-rotation min-conns : 4000000 conn-rate-limit : , out-of-rotation bandwidth-rate-limit : , out-of-rotation retcode out-of-rotation count : load value : 0
Note: Troubleshooting Layer 7 Load Balancing 119
Cisco Application Control Engine (ACE) Troubleshooting Guide The connection failures counter increments only if the ACE attempts to load balance a connection and the ACE does not receive a SYN-ACK from the real server in response to a SYN or if the real server responds to the SYN with a RST. 6. Check the L7 load-balance statistics by entering the following command:
ACE_module5/Admin# show stats loadbalance +------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0 Total Layer7 decisions : 0 Total Layer7 rejections : 0 Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in a redundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails to find the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a proper remote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the ID Map table was not created. 7. If you are having problems with HTTP, check the HTTP statistics and error counters by entering the following command:
ACE_module5/Admin# show stats http +------------------------------------------+ +-------------- HTTP statistics -----------+ +------------------------------------------+ LB parse result msgs sent : 0 , TCP data msgs sent : Inspect parse result msgs : 0 , SSL data msgs sent : sent TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: SSL fin/rst msgs sent : 0 , Unproxy msgs sent : Drain msgs sent : 0 , Particles read : Reuse msgs sent : 0 , HTTP requests : Reproxied requests : 0 , Headers removed : Headers inserted : 0 , HTTP redirects : HTTP chunks : 0 , Pipelined requests : HTTP unproxy conns : 0 , Pipeline flushes : Whitespace appends : 0 , Second pass parsing : Response entries recycled : 0 , Analysis errors : Header insert errors : 0 , Max parselen errors : Static parse errors : 0 , Resource errors : Invalid path errors : 0 , Bad HTTP version errors : Headers rewritten : 0 , Header rewrite errors :
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8. If you suspect a probe issue, for example, a TCP probe, check the probe statistics and error counters by entering the following command:
ACE_module5/Admin# show stats probe type tcp +------------------------------------------+ +----------- Probe statistics -------------+ +------------------------------------------+ ----- tcp probe ---Total probes sent : 0 Total Total probes passed : 0 Total Total connect errors : 0 Total Total RST received : 0 Total
send failures probes failed conns refused open timeouts
: : : :
0 0 0 0
120

Total receive timeout : 0 Total active sockets : 0
9. Check the parameter map statistics for an HTTP parameter map by entering the following command:
ACE_module5/Admin# show parameter-map HTTP_PMAP Number of parameter-maps : 1
Parameter-map : HTTP_PMAP Type : http server-side connection reuse case-insensitive parsing persistence-rebalance header modify per-request header-maxparse-length content-maxparse-length parse length-exceed action urlcookie-delimiters
: : : : : : : :
disabled disabled disabled disabled 4096 4096 drop /&#+
<---------------------------- Enabled by default in the ACE
10. Clear the L7 load-balancing statistics by entering the following commands: clear stats loadbalance [radius | rdp] clear service-policy policy_name clear stats http clear rserver server_name clear serverfarm serverfarm_name
This article describes the procedures for troubleshooting redundancy issues with your ACE. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
121
Contents
1 Overview of ACE Redundancy 1.1 Redundancy Protocol 1.2 FT VLAN 1.3 Configuration Requirements and Restrictions 1.4 Example of a Redundancy Configuration 2 Troubleshooting ACE Redundancy 3 FT Peer and Group Status Details 3.1 FT Group Status Conditions 3.1.1 STANDBY_COLD 3.1.2 STANDBY_CONFIG 3.2 FT Peer Status Conditions 3.2.1 PEER_DOWN 3.2.2 TL_ERROR 3.2.3 FT_VLAN_DOWN 3.2.4 FSM_PEER_STATE_ERROR 3.3 About WARM_COMPATIBLE and STANDBY_WARM
Overview of ACE Redundancy

Redundancy (or fault tolerance) allows your network to remain operational even if one of the ACEs becomes unresponsive. Redundancy ensures that your network services and applications are always available. Redundancy provides seamless switchover of flows if an ACE becomes unresponsive or a critical host, interface, or HSRP group (ACE module only) fails. Redundancy supports the following network applications that require fault tolerance: Mission-critical enterprise applications Banking and financial services E-commerce Long-lived flows such as FTP and HTTP file transfers
Redundancy Protocol
You can configure a maximum of two ACE modules (peers) in the same Catalyst 6500 series switch or in different chassis for redundancy. You can also configure a maximum of two ACE 4710 appliances for redundancy. Each peer ACE can contain one or more fault-tolerant (FT) groups. Each FT group consists of two members: one active context and one standby context. For more information about contexts, see the Cisco Application Control Engine Module Virtualization Configuration Guide or the Cisco 4700 Series Application Control Engine Appliance Administration Guide (Software Version A3(2.4)). An FT group has a unique group ID that you assign. Both ACEs can be active at the same time, processing traffic for distinct virtual devices and backing up each other (stateful redundancy). An Active-Active configuration requires two FT groups and two virtual contexts on each ACE. See Figure 1. Figure 1. Example of an Active-Active Configuration Contents 122
The ACE uses the redundancy protocol to communicate between the redundant peers. The election of the active member within each FT group is based on a priority scheme. The member configured with the higher priority is elected as the active member. If a member with a higher priority is found after the other member becomes active, the new member becomes active because it has a higher priority. This behavior is known as preemption and is enabled by default. One virtual MAC address (VMAC) is associated with each FT group. The format of the VMAC is: 00-0b-fc-fe-1b-groupID. Because a VMAC does not change upon a switchover, the client and server ARP tables does not require updating. The ACE selects a VMAC from a pool of virtual MACs available to it. You can specify the pool of MAC addresses that the local ACE and the peer ACE use by configuring the shared-vlan-hostid command and the peer shared-vlan-hostid command, respectively. To avoid MAC address conflicts, be sure that the two pools are different on the two ACEs. For more information about VMACs and MAC address pools, see the Cisco Application Control Engine Module Routing and Bridging Configuration Guide. Each FT group acts as an independent redundancy instance. When a switchover occurs, the active member in the FT group becomes the standby member and the original standby member becomes the active member. A switchover can occur for the following reasons: The active member becomes unresponsive. A tracked host, interface, or HSRP group fails. You enter the ft switchover command to force a switchover.
FT VLAN
Redundancy uses a dedicated FT VLAN between redundant ACEs to transmit flow-state information and the redundancy heartbeat. You must configure this same VLAN on both peer ACEs. You also must configure a different IP address within the same subnet on each ACE for the FT VLAN. Cisco recommends two port-channeled 1-Gigabit Ethernet links for the FT VLAN. For the appliance, when you configure the ft-port-vlan command, the ACE modifies the associated Ethernet port or port-channel interface to a trunk port. Note: Do not use the FT VLAN for any other network traffic, including HSRP traffic and data. The two redundant ACEs constantly communicate over the FT VLAN to determine the operating status of each ACE. The standby member uses the heartbeat packet to monitor the health of the active member. The active member uses the heartbeat packet to monitor the health of the standby member. Communications over the switchover link include the following data: Redundancy protocol packets State information replication data Configuration synchronization information Redundancy Protocol 123
Cisco Application Control Engine (ACE) Troubleshooting Guide Heartbeat packets For multiple contexts, the FT VLAN resides in the system configuration file. Each FT VLAN on the ACE has one unique MAC address associated with it. The ACE uses these device MAC addresses as the source or destination MACs for sending or receiving redundancy protocol state and configuration replication packets. Note: The IP address and the MAC address of the FT VLAN do not change at switchover.
Configuration Requirements and Restrictions

Follow these requirements and restrictions when configuring the redundancy feature: Redundancy is not supported between an ACE module and an ACE appliance operating as peers. Redundancy must be of the same ACE device type and software release. In bridged mode (Layer 2), two contexts cannot share the same VLAN. To achieve active-active redundancy, a minimum of two contexts and two FT groups are required on each ACE. When you configure redundancy, the ACE keeps all interfaces that do not have an IP address in the Down state. The IP address and the peer IP address that you assign to a VLAN interface should be in the same subnet but should be different IP addresses. For more information about configuring VLAN interfaces, see the Cisco Application Control Engine Module Routing and Bridging Configuration Guide or the Cisco ACE 4700 Series Appliance Routing and Bridging Configuration Guide FT Interfaces are put into an automatic trunk status and, for the module, the Catalyst 6500 series switch needs to be set to trunk the specific VLAN you are using for the FT interface.
Example of a Redundancy Configuration

The following example shows a running-configuration file that defines fault tolerance (FT) for a single ACE operating in a redundancy configuration. You must configure a maximum of two ACEs (peers) for redundancy to fail over from the active ACE to the standby ACE. Note: All FT parameters are configured in the Admin context. This configuration addresses the following redundancy components: A dedicated FT VLAN for communication between the members of an FT group. You must configure this same VLAN on both peers. An FT peer definition. An FT group that is associated with the Admin context. A critical tracking and failure detection process for an interface.
access-list ACL1 line 10 extended permit ip any any class-map 2 match 3 match 4 match 5 match 7 match 8 match type management match-any L4_REMOTE-MGT_CLASS protocol telnet any protocol ssh any protocol icmp any protocol http any protocol snmp any protocol https any
policy-map type management first-match L4_REMOTE-MGT_POLICY class L4_REMOTE-MGT_CLASS permit interface vlan 100 ip address 192.168.83.219 255.255.255.0 peer ip address 192.168.83.230 255.255.255.0 alias 192.168.83.200 255.255.255.0
FT VLAN
124

access-group input ACL1 service-policy input L4_REMOTE-MGT_POLICY no shutdown ft interface vlan 200 ip address 192.168.1.1 255.255.255.0 peer ip address 192.168.1.2 255.255.255.0 no shutdown ft peer 1 ft-interface vlan 200 heartbeat interval 300 heartbeat count 10 ft group 1 peer 1 priority 200 associate-context Admin inservice ft track interface TRACK_VLAN100 track-interface vlan 100 peer track-interface vlan 200 priority 50 peer priority 5 ip route 0.0.0.0 0.0.0.0 192.168.83.1
Troubleshooting ACE Redundancy

This section describes the methods and CLI commands that you can use to troubleshoot redundancy issues in your ACE. 1. Ensure that the software versions and licenses installed in the two ACEs are identical. A software or license mismatch may generate the following syslog message:
%ACE-1-727006: HA: Peer is incompatible due to error str. Cannot be Redundant.
To verify the software (SRG) and license compatibility of the FT peer, enter the following command:
ACE_5/Admin# show ft peer status Peer Id State Maintenance mode SRG Compatibility License Compatibility FT Groups : : : : : : 1 FSM_PEER_STATE_MY_IPADDR MAINT_MODE_OFF COMPATIBLE COMPATIBLE 1
If the software or license is incompatible, install the appropriate software image or license in the peer to correct the problem. 2. Ensure that any SSL certificates (certs) and keys that exist in the active ACE are also configured in the standby ACE. SSL certs and keys are not synchronized automatically from the active to the standby. Use the crypto export and crypto import commands to accomplish this task. This requirement also applies to scripts and scripted probes. Failure to keep the active and standby configurations identical will cause configuration synchronization to fail and may cause the standby ACE to enter the STANDBY_COLD state. 3. The ACE sends heartbeat packets via UDP over the FT VLAN between peers. When heartbeats are not received during the specified interval (the interval and count are configurable), the ACE notifies the HA processor on the CP by sending a Peer_Down interprocess communication protocol (IPCP) message. If a peer is down or unreachable, you may receive one of the following Example of a Redundancy Configuration 125
Cisco Application Control Engine (ACE) Troubleshooting Guide syslog messages:

%ACE-1-727001: HA: Peer IP address is not reachable. Error: error str %ACE-1-727002: HA: FT interface interface name to reach peer IP address is down. Error: error str
4. Verify connectivity between the peers over the FT VLAN. If a peer device is physically up but connectivity is the problem, you may end up with two active devices. If connectivity is lost due to the peer going down, reboot the peer to restore redundancy between the two devices. 5. Display heartbeat statistics, including missed heartbeats, by entering the following command:
ACE_5/Admin# show ft stats HA Heartbeat Statistics -----------------------Number of Heartbeats Sent Number of Heartbeats Received Number of Heartbeats Missed Number of Unidirectional HB's Received Number of HB Timeout Mismatches Num of Peer Up Events Sent Num of Peer Down Events Sent Successive HB's miss Intervals counter Successive Uni HB's recv counter : : : : : : : : : 0 0 0 0 0 0 0 0 0
6. Provide an alternate path for the ACE to check the peer's status in case of missed heartbeats and configure a query interface using the followng commands:
ACE_5/Admin# config Enter configuration commands, one per line. End with CNTL/Z. ACE_5/Admin(config)# ft peer 1 ACE_5/Admin(config-ft-peer)# query-interface vlan 100
If the query interface is configured, upon receiving a PEER_DOWN message from the heartbeat process, the ACE data plane attempts to ping the peer using the Query VLAN. If the ping fails, the standby transitions to the ACTIVE state. If the ping is successful, the standby transitions to the STANDBY_COLD state. To recover from the STANDBY_COLD state, reboot the standby. 7. Each peer uses a VMAC that is dependent on the FT group number. If you are using multiple ACE modules in the same chassis, be careful when you configure the same FT groups in more than one module. Display the VMAC for an FT group by entering the following command:
ACE_5/Admin# show interface internal iftable vlan100 vlan100 -------ifid: 6 Context: 0 ifIndex: 16777316 physid: 100 rmode: 0 (unknown) iftype: 0 (vlan) bvi_bgid: 0 MTU: 1500 MAC: 00:18:b9:a6:91:15 VMAC: 00:00:00:00:00:00 <------- Virtual MAC Address Flags: 0x8a000800 (valid, down, admin-down, Non-redundant, tracked) ACL In: 0 ACL Out: 0
126

Route ID: FTgroupID: Zone ID: Sec Level: L2 ACL: 0 0 6 0 bpdu DENY, ipv6 DENY, mpls DENY, all DENY
LastChange: 0 (Thu Jan 1 00:00:00 1970) iflookup index: 100 vlan-vmac index:0 Next Shared IF: 0 Lock: Unlocked, seq 5 Lock errors: 0 Unlock errors: 0 No. of times locked: 5 No. of times unlocked: 5 Current/last owner: 0x40a7fc
8. If the members of an FT group are unable to reach the ACTIVE or the STANDBY_HOT state, there may be a context name mismatch for the same FT group. You may receive the following syslog message:
%ACE-1-727003: HA: Mismatch in context names detected for FT group FTgroupID. Cannot be redundant.
Be sure that the context names within the same FT group are identical on both ACEs. 9. Check the FT group configuration on both devices. Make sure that both devices are associated with the same context. Enter the following command:
ACE_5/Admin# show running-config ft
10. Verify the FT peer status and configuration by entering the following command:
ACE_5/Admin# show ft peer detail Peer Id State Maintenance mode FT Vlan FT Vlan IF State My IP Addr Peer IP Addr Query Vlan Query Vlan IF State Peer Query IP Addr Heartbeat Interval Heartbeat Count Tx Packets Tx Bytes Rx Packets Rx Bytes Rx Error Bytes Tx Keepalive Packets Rx Keepalive Packets TL_CLOSE count FT_VLAN_DOWN count PEER_DOWN count SRG Compatibility License Compatibility FT Groups : : : : : : : : : : : : : : : : : : : : : : : : : 1 FSM_PEER_STATE_COMPATIBLE MAINT_MODE_OFF 100 DOWN 10.1.1.1 10.1.1.2 110 DOWN 172.25.91.202 300 20 318573 66301061 318540 66272840 0 318480 318480 0 0 0 COMPATIBLE COMPATIBLE 3
11. Verify the FT group status and configuration by entering the following command:
ACE_5/Admin# show ft group detail
127

FT Group No. of Contexts Configured Status Maintenance mode My State My Config Priority My Net Priority My Preempt Peer State Peer Config Priority Peer Net Priority Peer Preempt Peer Id Last State Change time Running cfg sync enabled Running cfg sync status Startup cfg sync enabled Startup cfg sync status Bulk sync done for ARP: 0 Bulk sync done for LB: 0 Bulk sync done for ICM: 0 : : : : : : : : : : : : : : : : : : 1 1 in-service MAINT_MODE_OFF FSM_FT_STATE_ACTIVE 110 110 Enabled FSM_FT_STATE_STANDBY 100 100 Enabled 1 Thu Apr 2 00:00:00 2009 Enabled Running configuration sync has completed Enabled Running configuration sync has completed
For information on troubleshooting the FT group status, see the "FT Group Status Conditions"
FT Peer and Group Status Details

This section describes how to diagnose unexpected status conditions for the FT group and FT peer. This information may enable you to troubleshoot an issue directly or help you to provide additional information to your Cisco support representative.
FT Group Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT group status. STANDBY_COLD An FT group status of STANDBY_COLD may appear when: Config sync fails (including, incr-sync and bulk-sync), or FT VLAN is down while the query interface is up Config Sync Failure In configuration synchronization fails, the peers are not correctly exchanging configuration information. This failure can be identified as follows: 1. Output of the show ft peer detail command shows that the peer state is "Compatible". 2. Entering show ft group detail shows that the FT group is in "Standby Cold" mode and entering cfg sync status shows the reason for the failure. For incr-sync failure, the output shows exactly which command resulted in an execution error on the standby. For a bulk-sync failure, the reason is "Error on Standby device when applying configuration file replicated from active". 3. To further investigate a bulk-sync failure, perform these steps on the standby device: For software version A2(2.0) and earlier and version A2(1.3) and earlier releases, from the Admin context, enter show ft history cfg_cntlr and grep for "error:" to find any CLI commands that caused execution errors. For later releases, enter show ft config-error ctx_name to view the failed CLI commands.
FT Peer and Group Status Details
128
Cisco Application Control Engine (ACE) Troubleshooting Guide To work around a bulk sync failure, perform these steps to remove the CLI commands that triggered the error (as identified from the preceding analysis) and then retrigger the bulk sync operation, as follows: 1. Retrigger bulk sync by disabling config sync with the no ft auto-sync running command. 2. Re-enable config sync with ft auto-sync running. If the problem persists, repeat the above sequence until you eliminate the CLI command that triggered the problem. FT VLAN Down with Query Interface Up This condition can be identified by: 1. Entering show ft peer detail, which shows a peer state of FT_VLAN_DOWN. 2. Entering show ft stats, which shows that heartbeats are being missed. In this case, check the physical connectivity of the device. It might be a physical port or cable issue. STANDBY_CONFIG If a device appears to be stuck in the STANDBY_CONFIG state: 1. Run show ft history cfg_cntlr to determine whether the peer devices successfully exchanged notifications regarding configuration synchronization. 2. Grep for the keywords MTS_OPC_REQ_CFG_DNLD_STATUS and MTS_OPC_CFG_DNLD_STATUS. If one or both of the messages are missing, an error occurred in the synchronization exchange process. Note that once it is stuck in the STANDBY_CONFIG state, configuration mode will be disabled on both the active and standby devices. It can be stuck in this state for up to 4 hours, after which a timeout period expires.
FT Peer Status Conditions

This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT peer. PEER_DOWN If the peer status shows PEER_DOWN: 1. Check whether IP addresses for the local and peer device are configured correctly on both. 2. Verify that pinging or telneting to the peer IP address works. If ping fails, check whether the interface is up (show interface). If so, the interface VLAN is probably not allocated to the ACE module on the supervisor (which suggests a configuration issue on the supervisor). 3. Enter show arp to see if the FT peer IP address is resolved. (If ARP is not resolved and ping/telnet also failed, it might be an encapsulation issue requiring support). 4. Enter show conn on both sides to see if HA connections have been set up. If connections have not been set up, check the HA DP manager log (show ft history ha_dp_mgr). Setup may have failed for various reasons. If this is the case, contact Cisco technical support. 5. Enter show ft stats on both devices to see if heartbeats are being sent or received. If the Number of Heartbeats Missed counter is incrementing, the heartbeat packets could be getting dropped. Enter show np 1 me-stats -sfp to see if heartbeat packets are being received and forwarded to X-Scale, as indicated by the counter Packets forward to XScale. If this counter is not incrementing, provide the information to Cisco technical support.
STANDBY_COLD
129
Cisco Application Control Engine (ACE) Troubleshooting Guide TL_ERROR This state may occur when the telnet connection used to exchange configuration information between the peers cannot be established but heartbeat packets are exchanged successfully. To identify this issue: 1. Verify that heartbeats are flowing by checking the statistics, show ft stats. 2. Attempt to connect by telnet or to ping the FT peer. The telnet connection attempt will likely fail. 3. Run show arp to see if the FT peer IP address can be resolved. If show arp indicates that the address is not resolvable and the ping or telnet connect attempts fail, it is likely an encapsulation issue on the ACE. FT_VLAN_DOWN This state typically occurs when the FT VLAN goes down while the query interface is up. If the heartbeat exchange fails and the query interface is determined to be up based on an ICMP message check, the status is FT_VLAN_DOWN. To verify, attempt to connect to the FT VLAN Peer IP address by ping or telnet. If running show ft stats indicates that heartbeats are being missed, it is likely a physical connectivity issue, such as the physical port or cable failure. FSM_PEER_STATE_ERROR This indicates a Software Relationship Graph (SRG) version inconsistency between the peers. See the relationship graph table in the following section.
About WARM_COMPATIBLE and STANDBY_WARM

While peers in a redundant configuration are designed to operate with identical versions of the software, when you are upgrading or downgrading the software in the ACEs, it is possible for the peers to temporarily employ different software versions. The WARM_COMPATIBLE and STANDBY_WARM redundancy states help minimize the operational impact of CLI compatibility issues between the peers, and allow failovers to occur on a best-effort basis during such transitions. When you upgrade or downgrade the ACE software in a redundant configuration with different software versions, the STANDBY_WARM and WARM_COMPATIBLE states allow the configuration and state synchronization process between the peers to continue on a best-effort basis. This basis allows the active ACE to synchronize configuration and state information with the standby even though the standby may not recognize or understand the CLI commands or state information. In the STANDBY_WARM state, as with the STANDBY_HOT state, configuration mode is disabled on the standby ACE and configuration and state synchronization continues. A failover from the active to the standby based on priorities and preempt can still occur while the standby is in the STANDBY_WARM state. However, while stateful failover is possible for a WARM standby, it is not guaranteed. In general, modules should be allowed to remain in this state only for a short period of time. When redundancy peers run different software versions, the SRG compatibility field shown by the show ft peer status command output displays WARM_COMPATIBLE instead of COMPATIBLE. When the peer is in the WARM_COMPATIBLE state, the FT groups in the standby transition to the STANDBY_WARM state instead of the STANDBY_HOT state. The following software version combinations tables indicate whether the SRG compatibility field will display WARM_COMPATIBLE (WC) or COMPATIBLE (C): ACE Module: C = COMPATIBLE / WC = WARM_COMPATIBLE Active(Column)/Standby(Row) A2(1.5) A2(1.6) A2(2.0) TL_ERROR
A2(2.1)
A2(2.2)
A2(3.0)
A2(2.3) 130
Cisco Application Control Engine (ACE) Troubleshooting Guide A2(1.5) A2(1.6) A2(2.0) A2(2.1) A2(2.2) A2(3.0) C WC C C WC WC WC C C WC WC WC C C C C C C C WC C C WC WC WC WC C WC C WC WC A3(2.2) C C C C WC WC WC WC C WC WC C WC A3(2.3) WC WC WC WC C WC WC WC C WC WC WC C A3(2.4) WC WC WC WC WC C
A2(2.3) WC WC C WC ACE Appliance: C = COMPATIBLE / WC = WARM_COMPATIBLE Active(Column)/Standby(Row) A3(1.0) A3(2.0) A3(2.1) A3(1.0) A3(2.0) A3(2.1) A3(2.2) A3(2.3) A3(2.4) C C C C WC WC C C C C WC WC C C C C WC WC
This article describes the process and CLI commands for troubleshooting SSL in the ACE. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE SSL Troubleshooting 1.1 Example of an SSL Termination Configuration 1.2 Example of an SSL Initiation Configuration 2 Troubleshooting ACE SSL 131
About WARM_COMPATIBLE and STANDBY_WARM
Overview of ACE SSL Troubleshooting

Secure Sockets Layer (SSL) runs over TCP. After the TCP three-way handshake completes and the ACE has proxied the connection, the SSL handshake takes place. For information about proxied connections, see the Troubleshooting Connectivity article. See Figure 1 for an illustration of the SSL handshake.
Figure 1. SSL Handshake
132
The ACE supports the following SSL configurations (see Figure 2): SSL termination (ACE acts as an SSL server) SSL initiation (ACE acts as a client) End-to-end SSL (SSL termination plus SSL initiation)
Figure 2. SSL Configurations
133
Before you begin to troubleshoot potential SSL issues, be sure that the following conditions exist: You have configured basic SLB and SSL on your ACE. For details about configuring SLB, see the Cisco Application Control Engine Module Server Load-Balancing Configuration Guide or the Cisco ACE 4700 Series Appliance Server Load-Balancing Configuration Guide. For details about configuring SSL, see the Cisco Application Control Engine Module SSL Configuration Guide or the Cisco ACE 4700 Series Appliance SSL Configuration Guide. If you are running multiple ACEs in a redundant configuration, be sure that you have copied the SSL certificates (certs) and keys to the standby ACE. Certs and keys are not replicated in a redundant configuration from the active ACE to the standby ACE. Also, ensure that the configurations on the active and the standby are identical, including the same licenses and software versions. Be sure that the certs and keys are no larger than 4096 bits and that they are of an RSA type supported by the ACE. For details about configuring SSL, see the Cisco Application Control Engine Module SSL Configuration Guide or the Cisco ACE 4700 Series Appliance SSL Configuration Guide. The ACE supports the following RSA key pair sizes: 512 (least security) 768 (normal security) 1024 (high security, level 1) 1536 (high security, level 2) 2048 (high security, level 3) 4096 (high security, level 4) - For software release A2(2.4) and later in the ACE module and software release A3(2.6) and later in the ACE appliance, you can use 4096-bit SSL certificates in chaingroups and authgroups. You can also import public certificates and keys that are 4096 bits in length. Server certs are valid, installed, and have not expired
134
Example of an SSL Termination Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy server; terminating SSL or TLS connections from a client and then establishing a TCP connection to an HTTP server. When the ACE terminates the SSL or TLS connection, it decrypts the cipher text from the client and transmits the data as clear text to the HTTP server.
access-list ACL1 line 10 extended permit ip any any probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200 rserver SERVER1 ip address 10.1.0.11 inservice rserver SERVER2 ip address 10.1.0.12 inservice rserver SERVER3 ip address 10.1.0.13 inservice rserver SERVER4 ip address 10.1.0.14 inservice rserver SERVER5 ip address 10.1.0.15 inservice rserver SERVER6 ip address 10.1.0.16 inservice rserver SERVER7 ip address 10.1.0.17 inservice rserver SERVER8 ip address 10.1.0.18 inservice serverfarm host SFARM1 description SERVER FARM 1 FOR SSL TERMINATION probe GEN_HTTP rserver SERVER1 80 inservice rserver SERVER2 80 inservice rserver SERVER3 80 inservice rserver SERVER4 80 inservice serverfarm host SFARM2 description SERVER FARM 2 FOR SSL TERMINATION probe GEN_HTTP rserver SERVER5 80 inservice rserver SERVER6 80 inservice rserver SERVER7 80 inservice rserver SERVER8 80 inservice parameter-map type ssl PARAMMAP_SSL_TERMINATION

cipher RSA_WITH_3DES_EDE_CBC_SHA cipher RSA_WITH_AES_128_CBC_SHA priority 2 cipher RSA_WITH_AES_256_CBC_SHA priority 3 version all parameter-map type connection TCP_PARAM syn-data drop exceed-mss allow ssl-proxy service SSL_PSERVICE_SERVER ssl advanced-options PARAMMAP_SSL_TERMINATION key MYKEY.PEM cert MYCERT.PEM class-map type http loadbalance match-all L7_SERVER_CLASS description Sticky for SSL Testing 2 match http url .*.jpg 3 match source-address 192.168.130.0 255.255.255.0 class-map type http loadbalance match-all L7_SLB-HTTP_CLASS 2 match http url .* 3 match source-address 192.168.130.0 255.255.255.0 class-map match-all L4_SSL-TERM_CLASS description SSL Termination VIP 2 match virtual-address 192.168.130.11 tcp eq https policy-map type loadbalance first-match L7_SSL-TERM_POLICY class L7_SERVER_CLASS serverfarm SFARM1 insert-http I_AM header-value "SSL_TERM" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" insert-http SRC_IP header-value "is" class L7_SLB-HTTP_CLASS serverfarm SFARM1 insert-http I_AM header-value "SSL_TERM" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" insert-http SRC_IP header-value "is" policy-map multi-match L4_SSL-VIP_POLICY class L4_SSL-TERM_CLASS loadbalance vip inservice loadbalance policy L7_SSL-TERM_POLICY loadbalance vip icmp-reply ssl-proxy server SSL_PSERVICE_SERVER connection advanced-options TCP_PARAM interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of an SSL Initiation Configuration

The following example shows a running-configuration file of the ACE acting as an SSL proxy client, initiating and maintaining an SSL connection between itself and an SSL server. The ACE receives clear text from an HTTP client, and then encrypts and
Cisco Application Control Engine (ACE) Troubleshooting Guide transmits the data as cipher text to the SSL server. On the reverse side, the ACE decrypts the cipher text that it receives from the SSL server and sends the data to the client as clear text.
access-list ACL1 line 10 extended permit ip any any probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200 rserver SERVER1 ip address 10.1.0.11 inservice rserver SERVER2 ip address 10.1.0.12 inservice rserver SERVER3 ip address 10.1.0.13 inservice rserver SERVER4 ip address 10.1.0.14 inservice rserver SERVER5 ip address 10.1.0.15 inservice rserver SERVER6 ip address 10.1.0.16 inservice rserver SERVER7 ip address 10.1.0.17 inservice rserver SERVER8 ip address 10.1.0.18 inservice serverfarm host SFARM1 description SERVER FARM 1 FOR SSL INITIATION probe GEN_HTTP rserver SERVER1 443 inservice rserver SERVER2 443 inservice rserver SERVER3 443 inservice rserver SERVER4 443 inservice serverfarm host SFARM2 description SERVER FARM 2 FOR SSL TERMINATION probe GEN_HTTP rserver SERVER5 443 inservice rserver SERVER6 443 inservice rserver SERVER7 443 inservice rserver SERVER8 443 inservice parameter-map type http PARAMMAP_HTTP server-conn reuse case-insensitive persistence-rebalance parameter-map type ssl PARAMMAP_SSL_INITIATION
137

cipher RSA_WITH_RC4_128_MD5 cipher RSA_WITH_RC4_128_SHA cipher RSA_WITH_DES_CBC_SHA cipher RSA_WITH_3DES_EDE_CBC_SHA cipher RSA_WITH_AES_128_CBC_SHA cipher RSA_WITH_AES_256_CBC_SHA cipher RSA_EXPORT_WITH_RC4_40_MD5 cipher RSA_EXPORT1024_WITH_RC4_56_MD5 cipher RSA_EXPORT_WITH_DES40_CBC_SHA cipher RSA_EXPORT1024_WITH_DES_CBC_SHA cipher RSA_EXPORT1024_WITH_RC4_56_SHA version all parameter-map type connection TCP_PARAM syn-data drop exceed-mss allow ssl-proxy service SSL_PSRVICE_CLIENT ssl advanced-options PARAMMAP_SSL_INITIATION class-map type http loadbalance match-all L7_SERVER_CLASS description Sticky for SSL Testing 2 match http url .*.jpg 3 match source-address 192.168.130.0 255.255.255.0 class-map type http loadbalance match-all L7_SLB-HTTP_CLASS 2 match http url .* 3 match source-address 192.168.130.0 255.255.255.0 class-map match-all L4_SSL-INIT_CLASS description SSL Initiation VIP 2 match virtual-address 192.168.130.12 tcp eq www policy-map type loadbalance first-match L7_SSL-INIT_POLICY class L7_SERVER_CLASS serverfarm SFARM1 insert-http SRC_IP header-value "%is" insert-http I_AM header-value "SSL_INIT" insert-http SRC_Port header-value "%ps" insert-http DEST_IP header-value "%id" insert-http DEST_Port header-value "%pd" ssl-proxy client SSL_PSERVICE_CLIENT class L7_SLB-HTTP_CLASS serverfarm SFARM2 insert-http SRC_IP header-value "%is" insert-http I_AM header-value "SSL_INIT" insert-http DEST_Port header-value "%pd" insert-http DEST_IP header-value "%id" insert-http SRC_Port header-value "%ps" ssl-proxy client SSL_PSERVICE_CLIENT policy-map multi-match L4_SSL-VIP_POLICY class L4_SSL-INIT_CLASS loadbalance vip inservice loadbalance policy L7_SSL-INIT_POLICY loadbalance vip icmp-reply active appl-parameter http advanced-options PARAMMAP_HTTP connection advanced-options TCP_PARAM interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown ip route 10.1.0.0 255.255.255.0 192.168.120.254
138
Troubleshooting ACE SSL

To troubleshoot SSL issues, follow these steps: 1. For the ACE module, check the health of the Nitrox-II (crypto module) and ensure that it has not become unresponsive. Stop all traffic, and then enter the following command: ACE_module5/Admin# show crypto hardware Figure 3. Example of the Show Crypto Hardware Command Output for an Unresponsive Crypto Module
139
STX1 is a count of the number of packets transmitted by the Nitrox-II and IMX1 is the number of packets received by the Nitrox-II. On a normal system, these values should be the same once traffic has stopped. If the values are not the same, the Nitrox-II has become unresponsive. The Nitrox-II uses 0x500 TX buffers to transmit packets and 0x200 RX buffers to receive packets. If the [TR]X Buffers used count ever exceeds the amount available, the Nitrox-II has become unresponsive. The available cores field shows which of the 22 cores of the Nitrox-II are active. When no traffic is flowing, there should be no numbers following the Using: statement. If there are, as in the sample output above, then that core (0 in this case) is hung, and the Nitrox-II has become unresponsive.
Cisco Application Control Engine (ACE) Troubleshooting Guide For the POM count, there are two numbers, A(B). The "A" value is the number of outstanding packets to the Packet Order Manager, while the "B" value, counts the number of packets that have been processed in the last second. When no traffic is flowing, both of these values should be 0. If no traffic is flowing, and the value of "A" is nonzero as shown above, then there are outstanding requests to the POM that are not being processed, because the Nitrox-II has become unresponsive. 2. Ensure that appropriate ports are designated for PAT in an SSL termination configuration. By default, connections to the real server from the ACE will inherit the destination port from the client to VIP connection so that a connection to port 443 on the VIP will go to port 443 on the real server, unless otherwise specified in the server farm configuration. This will cause problems if you are using ACE to offload SSL between the client and the VIP and send clear-text traffic to the real servers. The following example demonstrates a port definition in a server farm configuration:
serverfarm host sf1 probe HTTP_PROBE rserver rs1 80 inservice rserver rs2 80 inservice
3. Verify that the SSL certificate and key are correct by entering the following command:
ACE_module5/Admin# crypto verify key cert
4. Verify that a certificate revocation list (CRL) has been downloaded, enter the following command:
ACE_module5/Admin# show crypto crl test1 test1: URL: http://192.168.12.23/test.crl Last Downloaded: not downloaded yet Total Number Of Download Attempts: 0 Failed Download Attempts: 0
5. Verify the contents of an authgroup by entering the following command:

ACE_module5/Admin# show crypto authgroup authgroup_name
6. Display client SSL statistics by entering the the following command:

ACE_module5/Admin# show stats crypto client +----------------------------------------------+ +---- Crypto client termination statistics ----+ +----------------------------------------------+ SSLv3 negotiated protocol: TLSv1 negotiated protocol: SSLv3 full handshakes: SSLv3 resumed handshakes: SSLv3 rehandshakes: TLSv1 full handshakes: TLSv1 resumed handshakes: TLSv1 rehandshakes: SSLv3 handshake failures: SSLv3 failures during data phase: TLSv1 handshake failures: TLSv1 failures during data phase: Handshake Timeouts: total transactions: SSLv3 active connections: SSLv3 connections in handshake phase: SSLv3 conns in renegotiation phase:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

SSLv3 TLSv1 TLSv1 TLSv1 TLSv1 connections in data phase: active connections: connections in handshake phase: conns in renegotiation phase: connections in data phase: 0 0 0 0 0
+----------------------------------------------+ +------- Crypto client alert statistics -------+ +----------------------------------------------+ SSL alert CLOSE_NOTIFY rcvd: SSL alert UNEXPECTED_MSG rcvd: SSL alert BAD_RECORD_MAC rcvd: SSL alert DECRYPTION_FAILED rcvd: SSL alert RECORD_OVERFLOW rcvd: SSL alert DECOMPRESSION_FAILED rcvd: SSL alert HANDSHAKE_FAILED rcvd: SSL alert NO_CERTIFICATE rcvd: SSL alert BAD_CERTIFICATE rcvd: SSL alert UNSUPPORTED_CERTIFICATE rcvd: SSL alert CERTIFICATE_REVOKED rcvd: SSL alert CERTIFICATE_EXPIRED rcvd: SSL alert CERTIFICATE_UNKNOWN rcvd: SSL alert ILLEGAL_PARAMETER rcvd: SSL alert UNKNOWN_CA rcvd: SSL alert ACCESS_DENIED rcvd: SSL alert DECODE_ERROR rcvd: SSL alert DECRYPT_ERROR rcvd: SSL alert EXPORT_RESTRICTION rcvd: SSL alert PROTOCOL_VERSION rcvd: SSL alert INSUFFICIENT_SECURITY rcvd: SSL alert INTERNAL_ERROR rcvd: SSL alert USER_CANCELED rcvd: SSL alert NO_RENEGOTIATION rcvd: SSL alert CLOSE_NOTIFY sent: SSL alert UNEXPECTED_MSG sent: SSL alert BAD_RECORD_MAC sent: SSL alert DECRYPTION_FAILED sent: SSL alert RECORD_OVERFLOW sent: SSL alert DECOMPRESSION_FAILED sent: SSL alert HANDSHAKE_FAILED sent: SSL alert NO_CERTIFICATE sent: SSL alert BAD_CERTIFICATE sent: SSL alert UNSUPPORTED_CERTIFICATE sent: SSL alert CERTIFICATE_REVOKED sent: SSL alert CERTIFICATE_EXPIRED sent: SSL alert CERTIFICATE_UNKNOWN sent: SSL alert ILLEGAL_PARAMETER sent: SSL alert UNKNOWN_CA sent: SSL alert ACCESS_DENIED sent: SSL alert DECODE_ERROR sent: SSL alert DECRYPT_ERROR sent: SSL alert EXPORT_RESTRICTION sent: SSL alert PROTOCOL_VERSION sent: SSL alert INSUFFICIENT_SECURITY sent: SSL alert INTERNAL_ERROR sent: SSL alert USER_CANCELED sent: SSL alert NO_RENEGOTIATION sent:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
+-----------------------------------------------+ +--- Crypto client authentication statistics ---+ +-----------------------------------------------+ Total SSL client authentications: 0 Failed SSL client authentications: 0 SSL client authentication cache hits: 0
142

SSL static CRL lookups: SSL best effort CRL lookups: SSL CRL lookup cache hits: SSL revoked certificates: Total SSL server authentications: Failed SSL server authentications: 0 0 0 0 0 0
+-----------------------------------------------+ +------- Crypto client cipher statistics -------+ +-----------------------------------------------+ Cipher sslv3_rsa_rc4_128_md5: Cipher sslv3_rsa_rc4_128_sha: Cipher sslv3_rsa_des_cbc_sha: Cipher sslv3_rsa_3des_ede_cbc_sha: Cipher sslv3_rsa_exp_rc4_40_md5: Cipher sslv3_rsa_exp_des40_cbc_sha: Cipher sslv3_rsa_exp1024_rc4_56_md5: Cipher sslv3_rsa_exp1024_des_cbc_sha: Cipher sslv3_rsa_exp1024_rc4_56_sha: Cipher sslv3_rsa_aes_128_cbc_sha: Cipher sslv3_rsa_aes_256_cbc_sha: Cipher tlsv1_rsa_rc4_128_md5: Cipher tlsv1_rsa_rc4_128_sha: Cipher tlsv1_rsa_des_cbc_sha: Cipher tlsv1_rsa_3des_ede_cbc_sha: Cipher tlsv1_rsa_exp_rc4_40_md5: Cipher tlsv1_rsa_exp_des40_cbc_sha: Cipher tlsv1_rsa_exp1024_rc4_56_md5: Cipher tlsv1_rsa_exp1024_des_cbc_sha: Cipher tlsv1_rsa_exp1024_rc4_56_sha: Cipher tlsv1_rsa_aes_128_cbc_sha: Cipher tlsv1_rsa_aes_256_cbc_sha:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7. Display SSL server statistics by entering the following command:

ACE_module5/Admin# show stats crypto server +----------------------------------------------+ +---- Crypto server termination statistics ----+ +----------------------------------------------+ SSLv3 negotiated protocol: TLSv1 negotiated protocol: SSLv3 full handshakes: SSLv3 resumed handshakes: SSLv3 rehandshakes: TLSv1 full handshakes: TLSv1 resumed handshakes: TLSv1 rehandshakes: SSLv3 handshake failures: SSLv3 failures during data phase: TLSv1 handshake failures: TLSv1 failures during data phase: Handshake Timeouts: total transactions: SSLv3 active connections: SSLv3 connections in handshake phase: SSLv3 conns in renegotiation phase: SSLv3 connections in data phase: TLSv1 active connections: TLSv1 connections in handshake phase: TLSv1 conns in renegotiation phase: TLSv1 connections in data phase:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
143

+----------------------------------------------+ +------- Crypto server alert statistics -------+ +----------------------------------------------+ SSL alert CLOSE_NOTIFY rcvd: SSL alert UNEXPECTED_MSG rcvd: SSL alert BAD_RECORD_MAC rcvd: SSL alert DECRYPTION_FAILED rcvd: SSL alert RECORD_OVERFLOW rcvd: SSL alert DECOMPRESSION_FAILED rcvd: SSL alert HANDSHAKE_FAILED rcvd: SSL alert NO_CERTIFICATE rcvd: SSL alert BAD_CERTIFICATE rcvd: SSL alert UNSUPPORTED_CERTIFICATE rcvd: SSL alert CERTIFICATE_REVOKED rcvd: SSL alert CERTIFICATE_EXPIRED rcvd: SSL alert CERTIFICATE_UNKNOWN rcvd: SSL alert ILLEGAL_PARAMETER rcvd: SSL alert UNKNOWN_CA rcvd: SSL alert ACCESS_DENIED rcvd: SSL alert DECODE_ERROR rcvd: SSL alert DECRYPT_ERROR rcvd: SSL alert EXPORT_RESTRICTION rcvd: SSL alert PROTOCOL_VERSION rcvd: SSL alert INSUFFICIENT_SECURITY rcvd: SSL alert INTERNAL_ERROR rcvd: SSL alert USER_CANCELED rcvd: SSL alert NO_RENEGOTIATION rcvd: SSL alert CLOSE_NOTIFY sent: SSL alert UNEXPECTED_MSG sent: SSL alert BAD_RECORD_MAC sent: SSL alert DECRYPTION_FAILED sent: SSL alert RECORD_OVERFLOW sent: SSL alert DECOMPRESSION_FAILED sent: SSL alert HANDSHAKE_FAILED sent: SSL alert NO_CERTIFICATE sent: SSL alert BAD_CERTIFICATE sent: SSL alert UNSUPPORTED_CERTIFICATE sent: SSL alert CERTIFICATE_REVOKED sent: SSL alert CERTIFICATE_EXPIRED sent: SSL alert CERTIFICATE_UNKNOWN sent: SSL alert ILLEGAL_PARAMETER sent: SSL alert UNKNOWN_CA sent: SSL alert ACCESS_DENIED sent: SSL alert DECODE_ERROR sent: SSL alert DECRYPT_ERROR sent: SSL alert EXPORT_RESTRICTION sent: SSL alert PROTOCOL_VERSION sent: SSL alert INSUFFICIENT_SECURITY sent: SSL alert INTERNAL_ERROR sent: SSL alert USER_CANCELED sent: SSL alert NO_RENEGOTIATION sent:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
+-----------------------------------------------+ +--- Crypto server authentication statistics ---+ +-----------------------------------------------+ Total SSL client authentications: Failed SSL client authentications: SSL client authentication cache hits: SSL static CRL lookups: SSL best effort CRL lookups: SSL CRL lookup cache hits: SSL revoked certificates: Total SSL server authentications: Failed SSL server authentications:
0 0 0 0 0 0 0 0 0
144

+-----------------------------------------------+ +------- Crypto server cipher statistics -------+ +-----------------------------------------------+ Cipher sslv3_rsa_rc4_128_md5: Cipher sslv3_rsa_rc4_128_sha: Cipher sslv3_rsa_des_cbc_sha: Cipher sslv3_rsa_3des_ede_cbc_sha: Cipher sslv3_rsa_exp_rc4_40_md5: Cipher sslv3_rsa_exp_des40_cbc_sha: Cipher sslv3_rsa_exp1024_rc4_56_md5: Cipher sslv3_rsa_exp1024_des_cbc_sha: Cipher sslv3_rsa_exp1024_rc4_56_sha: Cipher sslv3_rsa_aes_128_cbc_sha: Cipher sslv3_rsa_aes_256_cbc_sha: Cipher tlsv1_rsa_rc4_128_md5: Cipher tlsv1_rsa_rc4_128_sha: Cipher tlsv1_rsa_des_cbc_sha: Cipher tlsv1_rsa_3des_ede_cbc_sha: Cipher tlsv1_rsa_exp_rc4_40_md5: Cipher tlsv1_rsa_exp_des40_cbc_sha: Cipher tlsv1_rsa_exp1024_rc4_56_md5: Cipher tlsv1_rsa_exp1024_des_cbc_sha: Cipher tlsv1_rsa_exp1024_rc4_56_sha: Cipher tlsv1_rsa_aes_128_cbc_sha: Cipher tlsv1_rsa_aes_256_cbc_sha:
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
8. Display the number of SSL data messages sent and SSL FIN/RST messages sent by entering the following command:
ACE_module5/Admin# show stats http +------------------------------------------+ +-------------- HTTP statistics -----------+ +------------------------------------------+ LB parse result msgs sent : 0 , TCP data msgs sent : Inspect parse result msgs : 0 , SSL data msgs sent : sent TCP fin/rst msgs sent : 0 , Bounced fin/rst msgs sent: SSL fin/rst msgs sent : 0 , Unproxy msgs sent : Drain msgs sent : 0 , Particles read : Reuse msgs sent : 0 , HTTP requests : Reproxied requests : 0 , Headers removed : Headers inserted : 0 , HTTP redirects : HTTP chunks : 0 , Pipelined requests : HTTP unproxy conns : 0 , Pipeline flushes : Whitespace appends : 0 , Second pass parsing : Response entries recycled : 0 , Analysis errors : Header insert errors : 0 , Max parselen errors : Static parse errors : 0 , Resource errors : Invalid path errors : 0 , Bad HTTP version errors : Headers rewritten : 0 , Header rewrite errors :
0 0 <------0 0 <------0 0 0 0 0 0 0 0 0 0 0 0
9. Display session cache statistics for the current context by entering the following command:
switch/Admin# show crypto session SSL Session Cache Stats for Context -----------------Number of Client Sessions: Number of Server Sessions:
0 0
This article describes how to troubleshoot performance issues with your ACE. Troubleshooting ACE SSL 145
Contents
1 Overview of Troubleshooting Performance Issues 2 Troubleshooting Performance Issues
Contents
146
Overview of Troubleshooting Performance Issues

Before you begin to troubleshoot ACE performance issues, check and record the following items: 1. Be sure that the correct licenses are installed in your ACE. 2. Record the number of flows that you are sending to the ACE. 3. Record the performance of a single flow. 4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on) 5. Identify the ACE context that is receiving the traffic. 6. Enter the following Exec mode commands and save the output to a file: clear stats all show clock show tech-support show clock 7. Be familiar with your application setup.
Troubleshooting Performance Issues

To troubleshoot performance issues with your ACE, follow these steps: 1. Display the resources allocated to each resource class in the ACE by entering the following command:
ACE_module5/Admin# show resource allocation --------------------------------------------------------------------------Parameter Min Max Class --------------------------------------------------------------------------acl-memory 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% default RC1 default RC1 default RC1 default RC1 default RC1 default RC1 default RC1
syslog buffer
conc-connections
mgmt-connections
proxy-connections
bandwidth
connection rate
Overview of Troubleshooting Performance Issues
147

inspect-conn rate 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 5.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 5.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% default RC1 default RC1 default RC1 default RC1 default RC1 default RC1 default RC1 default RC1 default RC1
syslog rate
regexp
sticky
xlates
ssl-connections rate
mgmt-traffic rate
mac-miss rate
throughput
2. Display the resources allocated to the context in question by entering the following command:
ACE_module5/Admin# show resource usage context C1
Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------Context: C1 conc-connections 0 0 0 8000000 0 mgmt-connections 0 0 0 100000 0 proxy-connections 0 0 0 1048574 0 xlates 0 0 0 1048574 0 bandwidth 0 0 0 625000000 0 throughput 0 0 0 500000000 0 mgmt-traffic rate 0 0 0 125000000 0 <------- 1 GBps bandwidth reserved f connection rate 0 0 0 1000000 0 ssl-connections rate 0 0 0 5000 0 mac-miss rate 0 0 0 2000 0 inspect-conn rate 0 0 0 6000 0 acl-memory 0 0 0 78610432 0 sticky 0 0 209714 0 0 regexp 0 0 0 1048576 0 syslog buffer 0 0 0 4194304 0 syslog rate 0 0 0 100000 0
Note: All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply the displayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management (to-the-ACE) traffic. 3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:
cat6k# show fabric status slot channel speed 2 3 4 5 6 8 0 0 0 0 0 0 8G 8G 8G 8G 20G 8G module status OK OK OK OK OK OK
fabric status OK OK OK OK <-------Shows 8 Gbps connectivity to the chassis back p OK OK
148
Cisco Application Control Engine (ACE) Troubleshooting Guide 4. Check the fabric utilization by entering the following command:
cat6k# show fabric utilization slot channel speed 2 0 8G 3 0 8G 4 0 8G 5 0 8G 6 0 20G 8 0 8G Ingress % 3 0 0 0 0 2 Egress % 2 0 0 0 0 3
5. Display the load of the network processors (NPs) in terms of packets and connection processing for each microengine (ME) by entering the following command:
ACE_module5/Admin# show np 1 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------RECEIVE: FASTPATH: SLOWTX: TCP_RX: HTTP: IH_RX SSL_ME: CM_CLOSE: X_TO_ME: FIXUP: REASSEMBLY: OCM: TCP_TX: ICM: ACE/Admin# show np 2 me-stats -cpu 0 proxies open. ME Utilization Statistics -------------RECEIVE: FASTPATH: SLOWTX: TCP_RX: HTTP: IH_RX SSL_ME: CM_CLOSE: X_TO_ME: FIXUP: REASSEMBLY: OCM: TCP_TX: ICM:
7 44 0 0 0 0 0 36 0 0 0 0 0 39
9 46 2 0 0 0 0 43 0 0 0 0 0 46
Note: All show np commands must be entered for both NP1 and NP2 to obtain the total combined results. NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100 percent, the traffic load is stressing the system close to its architectural limits. Any ME function that reaches 100 percent utilization can cause back pressure and lead to dropped packets or dropped connections. 6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering the following command:
ACE_module5/Admin# show cde health | include Fifo Fifo Full drop count 0
149
Cisco Application Control Engine (ACE) Troubleshooting Guide Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally. Queues that can be affected and create backpressure are as follows: FIFOs for the CDE, NPs, and the Crypto Module Internal queues for each ME It is possible that some packets that are received by the ACE could be dropped internally if backpressure is applied. 7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQ Transmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are not incrementing by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-s fp" | include Backpressure FastQ Transmit Backpressure: 0 SlowQ Transmit Backpressure: 0 Drop: Transmit Backpressure: 0 ACE/Admin# show np 1 me-stats "-s fp" | include queue Drop: Next-Hop queue full: 0
8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpath queue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Drops due to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, it can drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.
ACE/Admin# show np 1 me-stats "-s tcp" | include queue Drop reproxy msg queue full: 0 Drops due to FastTX queue full: 0 Drops due to Fastpath queue full: 0 Drops due to HTTP queue full: 0 Drops due to SSL queue full: 0 Drops due to AI queue full: 0 Drops due to Fixup queue full: 0
The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP, probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforce configuration dependencies and requirements before pushing the configuration to the data plane. 9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by entering the following command:
ACE_module5/Admin# show processes cpu | inc util CPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%
The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections (such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing or inspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectional concurrent connection limit regardless of the features enabled. Table 1. Concurrent Connection Support Connection Type ACE Module Limit Layer 4 Layer 7 4,000,000 512,000
150
The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained with distinct connection objects. 10. Display the connection table by entering the following command:
ACE_module5/Admin# show conn total current connections : 6 conn-id np dir proto vlan source destination state ----------+--+---+-----+----+---------------------+---------------------+------+ 1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB 2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB 4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB 3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB 4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB 3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB
Note: You can add the detail command option to provide the following additional fields: connection idle time, elapsed time of the connection, byte count, and packet count for each connection object. The total current connections counter is also maintained in the output of the following command:
switch/Admin# show stats connection +------------------------------------------+ +------- Connection statistics ------------+ +------------------------------------------+ Total Connections Created : 124 Total Connections Current : 6 Total Connections Destroyed: 62 Total Connections Timed-out: 58 Total Connections Failed : 0
Note: The Total Connections Current counter counts the number of used connection objects, not the number of TCP flows. The number of TCP flows can be roughly determined as half the number of connection objects minus any UDP connections. The Total Connections Current counter is always up to date and the maximum value can be 8,000,000. Because of the Cisco ACE Module?s architecture, with distinct paths for new and established connections, the number of existing concurrent connections does not heavily impact the rate at which new connections can be set up. Nevertheless, a very large number of concurrent connections will eventually affect the performance of the system in setting up new connections. 11. Use the command "tcp wan-optimization rtt 0" for slow connections. The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order to increase performance for a given connection. The LB decision is made in the software (proxy) and then moved to the fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a given connection. It is possible that if a packet enters the system during the transition Between the proxy and unproxy states, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affect performance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibited This can be accomplished by configuring a parameter map with the following:
"tcp wan-optimization rtt 0"
This article describes the ACE system limits and performance numbers for various resources and configuration objects.
151
Contents
1 ACE Performance Numbers and Resource Limits 1.1 ACE Appliance Data Sheet 1.2 ACE Module Data Sheets 1.3 SLB-Related Limits 1.4 Security-Related Limits 1.5 Management-Related Limits
Contents
152
ACE Performance Numbers and Resource Limits

For the most current performance numbers for the ACE products, always refer to the data sheets for the ACE appliance and the ACE module.
ACE Appliance Data Sheet

ACE appliance data sheet
ACE Module Data Sheets

ACE10/ACE20 module data sheet ACE30 module data sheet If you have any questions or concerns related to ACE performance, please contact your Cisco account team for guidance.
SLB-Related Limits
Scalability Numbers The scalability numbers provided here are intended to provide guidelines related to configuration scalability. The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to your deployment, testing with your feature combination is strongly recommended. If there are any questions or concerns related to ACE performance, please contact your Cisco account team for guidance. SLB-Related Object ARP Entries Bridge Table Entries Bridge-Group Virtual Interfaces (BVIs) Concurrent Conns L4 (Unproxied) Concurrent Connections L7 (Proxied) Domains Domain Objects Logical Interfaces Resource Classes Roles Sticky Groups Sticky Table Entries Virtual Contexts VLANs ACE Module System Limit 32,768 32,768 4096 4,000,000 512,000 2,500 None 8,192 100 (99) 4,000 4,096 4,000,000 251 4,000 (2-4094) ACE Module Context Limit 32,768 32,768 2048 4,000,000 512,000 10 (9) None 8,192 1 16 (8) 4,096 4,000,000 N/A 4,000 (2-4094) ACE Appliance Additional Information Limit 32,768 32,768 512 1,000,000 128,000 10 (9 per context) One is used for the default domain. None 8,192 100 (99) 4,096 800,000 21 (1 Admin context) 4,000 (2-4094) 250 user contexts + 1 Admin context One is used for the default class. 16 (8) per context Eight are predefined. Any object within the virtual partition can be added to a domain. A few are reserved for L2 interafces, redundancy, and so on.
ACE Performance Numbers and Resource Limits
153
Security-Related Limits
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related to configuration scalability. The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to a particular customer, testing with that customer?s feature combination is strongly recommended before any commitment on ACE performance is made to the customer. If there are any questions or concerns related to ACE performance, please contact your Cisco account team for guidance. Security Related Object Static NAT Policies Dynamic NAT Policies Maximum of addresses in a NAT pool Maximum of addresses in a PAT pool PAT Entries Total NAT Pools Xlates Concurrent SSL Conns RSA key size SSL Certs/Key files ACE Module System Limit 4096 4096 64 ACE Module Context Limit 4096 4096 64 ACE Appliance Limit Additional Information 4096 4096 32
63k 4,000,000 8,192 1,000,000 100,000 up to 4096 bits 3800/3800 (A2(3.x) and earlier) 4096/4096
63k 4,000,000 8,192 1,000,000 100,000 up to 4096 bits 3800/3800 (A2(3.x) and earlier) 4096/4096
63l 1,000,000 8,192 64,000 100,000 up to 4096 bits Subset of L7 (proxied) connections Supported: 512, 786, 1536, 1024, 2048, and 4096 (imported public keys only) bits
3800/3800 (A3(1.x) and This number is strictly enforced in A220, earlier) A214, and A322 4096/4096 (A3(2.x) and later, incl. A4(1.0)) 154
Security-Related Limits
Cisco Application Control Engine (ACE) Troubleshooting Guide (A4(1.0) and later) (A4(1.0) and later)
Management-Related Limits
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related to configuration scalability. The scalability numbers, however, are based on basic configurations. In order to obtain scalability numbers specific to a particular customer, testing with that customer?s feature combination is strongly recommended before any commitment on ACE performance is made to the customer. If there are any questions or concerns related to ACE performance, please contact your Cisco account team for guidance. Management-Related Object AAA LDAP Servers AAA RADIUS Servers ACE Module System Limit 6,144 2K (256*8) ACE Module Context Limit 8 (24 total) 8 (24 total) 8 (24 total) 64 (63) 30 (Admin context: 28) No limit Not applicable 16 (8) 10 4 4 MB 4 1 MB 100 16 (8) Eight are predefined and cannot be altered, leaving eight for you to customize ACE Appliance 8 8 8 64 (63) 31 (including admin, www, and dm) Any object within the virtual partition can be added to a domain One domain is used for the default-domain and cannot be removed Additional Information
AAA TACACS+ Servers 6K (256*24) Domains Local Users 2500 7500
Objects within a Domain No limit Resource-classes Roles SNMP Hosts SSH Sessions Syslog buffer size Syslog CP rate 252 4000 No Limit 256 4 MB 5,000 per seconds
5,000 per seconds 3,000 per seconds
Management-Related Limits
155
Cisco Application Control Engine (ACE) Troubleshooting Guide Syslog DP rate Syslog history table size Syslog Hosts Syslog persistence size Syslog rate limit table size Telnet Sessions 350,000 per second 256 x 500 256 1M 256 x 100 256 350,000 per second 500 2 10 MB 1M 100 4 10,000 messages per sec 4 2 8,192 messages 100,000 per second
Syslog internal queue size 10 MB
This article describes how to manage and control the ACE system resources. Guide Contents Main Article Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Troubleshooting Connectivity Troubleshooting ACE Appliance Ethernet Ports Troubleshooting Remote Access Troubleshooting Access Control Lists Troubleshooting Network Address Translation Troubleshooting ACE Health Monitoring Troubleshooting Layer 4 Load Balancing Troubleshooting Layer 7 Load Balancing Troubleshooting Redundancy Troubleshooting SSL Troubleshooting Compression Troubleshooting Performance Issues ACE Resource Limits Managing ACE Resources Show Counter Reference
Contents
1 Overview of ACE Resources 2 Managing ACE Resources 2.1 ACE Resource Planning 2.2 Creating a Resource Class for Resource Management 2.3 Allocating Resources Within a Resource Class 2.4 Changing the Resource Allocation of a Resource Class 2.5 Displaying the ACE Resource Allocation and Usage
Contents
156
Overview of ACE Resources

Resource classes allow you to manage context access to ACE resources, such as concurrent connections or bandwidth rate. The ACE is preconfigured with a default resource class that it applies to the Admin context and any user context upon creation. The default resource class is configured to allow a context to operate within a range that can vary from no resource access (0 percent) to complete resource access (100 percent). When you use the default resource class with multiple contexts, you run the risk of oversubscribing ACE resources because the ACE permits all contexts to have full access to all of the resources on a first-come, first-served basis. When a resource is utilized to its maximum limit, the ACE denies additional requests made by any context for that resource. To avoid oversubscribing resources and to help guarantee access to a resource by any context, the ACE allows you to create customized resource classes that you associate with one or more contexts. A context becomes a member of the resource class when you make the association. Creating a resource class allows you to set limits on the minimum and maximum amounts of each ACE resource that a member context is entitled to use. You define the minimum and maximum values as percentages of all resources. For example, you can create a resource class that allows its member contexts access to no less that 25 percent of the total number of SSL connections that the ACE supports. You can limit and manage the allocation of the following ACE resources: ACL memory Buffers for syslog messages and TCP out-of-order (OOO) segments Concurrent connections (through-the-ACE traffic) Management connections (to-the-ACE traffic) Proxy connections Set resource limit as a rate (number per second) Regular expression (regexp) memory SSL connections Sticky entries Static or dynamic network address translations (Xlates) By default, when you create a context, the ACE associates the context with the default resource class. The default resource class provides resources of a minimum of 0 and a maximum of unlimited for all resources except sticky entries. For stickiness to work properly, you must explicitly configure a minimum resource limit for sticky entries by using the limit-resource command.
Overview of ACE Resources
157
Cisco Application Control Engine (ACE) Troubleshooting Guide For more information about managing ACE resources, see the Cisco Application Control Engine Module Virtualization Configuration Guide (Software Version A2(1.0)).
Managing ACE Resources

You can allocate system resources to multiple contexts by creating and defining one or more resource classes and then associating the contexts with a resource class. This section contains the following topics: ACE Resource Planning Creating a Resource Class for Resource Management Allocating Resources within a Resource Class Changing the Resource Allocation of a Resource Class
ACE Resource Planning

When you plan the initial resource allocations for the virtual contexts in your configuration, allocate only the minimum required or estimated resources. The ACE protects resources that are in use, so to decrease a context's resources, those resources must be unused. Although it is possible to decrease the resource allocations in real time, it may require additional management overhead to clear any used resources before reducing them. Therefore, it is considered a best practice to initially keep as many resources in reserve as possible and allocate the unused reserved resources as needed. To address scaling and capacity planning, we recommend that new installations do not exceed 60 to 80 percent of the ACE's total capacity. To accomplish this goal, create a reserved resource class with a guarantee of 20 to 40 percent of all the ACE resources and configure a virtual context dedicated solely to ensuring that these resources are reserved. Then, you can efficiently distribute such reserved resources to contexts as capacity demands for handling client traffic increase over time.
Creating a Resource Class for Resource Management

You can create a resource class to allocate and manage system resources by one or more contexts. The ACE supports a maximum of 100 resource classes. After you create and configure the resource class, use the member command in context configuration mode to assign a resource class to the context (see the "Associating a Context with a Resource Class" section). To create a resource class, use the resource-class command in configuration mode. The syntax of the command is as follows:
resource-class name
For the name argument, enter an unquoted text string with no spaces and a maximum of 64 alphanumeric characters. For example, to create the RC1 resource class, enter the following command:
ACE_module5/Admin(config)# resource-class RC1 ACE_module5/Admin(config-resource)
To remove the resource class from the configuration, enter the following command:
host1/Admin(config)# no resource-class RC1
When you remove a resource class from the ACE, any contexts that were members of that resource class automatically become members of the default resource class. The default resource class allocates a minimum of 0.00 percent to a maximum of 100.00 percent of all ACE resources to each context. You cannot modify the default resource class.
Managing ACE Resources
158
Allocating Resources Within a Resource Class

You can allocate all resources or individual resources to all member contexts of a resource class. For example, you can allocate only concurrent connections or sticky table memory or management traffic. To allocate system resources to all members (contexts) of a resource class, use the limit-resource command in resource-class configuration mode. The syntax of this command is as follows: limit-resource {acl-memory | all | buffer {syslog} | conc-connections | mgmt-connections | proxy-connections | rate {bandwidth | connections | inspect-conn | mac-miss | mgmt-traffic | ssl-bandwidth | syslog} | regexp | sticky | xlates} {minimum number} {maximum {equal-to-min | unlimited}} Note: The limit that you set for individual resources when you use the limit-resource command overrides the limit that you set for all resources when you use the limit-resource all command. If you lower the limits for one context (context A) in order to increase the limits of another context (context B), you may experience a delay in the configuration change because the ACE will not lower the limits of context A until the resources are no longer being used by the context. For example, to allocate 20 percent of all resources (minimum and maximum) to all member contexts of the resource class, enter the following command:
(config-resource)# limit-resource all minimum 20% maximum equal-to-min
To restore resource allocation to the default values of 0 percent minimum and 100 percent maximum for all resources to all member contexts, enter the following command:
(config-resource)# no limit-resource all
Table 1 lists the managed system resources of the ACE. You can limit these resources per context or for all contexts associated with the resource class by using the limit-resource command. See the "Allocating Resources within a Resource Class" section. Table 1. System Resource Maximum Values Resource ACL Memory Buffer Memory (Syslog) Concurrent Connections (Layer 4) Concurrent Connections (SSL) Management Connections SSL Proxy Connections Rate ---Bandwidth 4 gigabits per second (Gbps) You can upgrade the ACE maximum bandwidth to 8 Gbps or 16 Gbps by purchasing a separate license from Cisco. For more information, see the Cisco Application Control Engine Module Administration Guide (Software Version A2(1.0)). ---Connections (any kind) 325,000 connections per second (CPS) 159 Maximum Value 78,610,432 bytes 4,000,000 bytes 4,000,000 connections 200,000 100,000 connections 200,000
Proxy Connections (Layer 7) 524,286 connections
Allocating Resources Within a Resource Class
Cisco Application Control Engine (ACE) Troubleshooting Guide ---MAC miss ---Management traffic ---SSL transactions 2000 packets per second (PPS) 1 Gbps 1000 transactions per second (TPS), upgradeable to 15000 TPS with a separate license. For more information, see the Cisco Application Control Engine Module Administration Guide (Software Version A2(1.0)). For traffic going to the ACE (control plane), 5000 messages per second For traffic going through the ACE (data plane), 350,000 messages per second Regular Expression Memory 1,048,576 bytes Sticky Entries Xlates (network and port address translation entries) 4,194,304 entries 524,286 translations
---Syslog
Changing the Resource Allocation of a Resource Class

If you (as the global Admin) need to change the resource allocation in a resource class of which two or more user contexts are members, you may do so at any time by entering the appropriate CLI commands. (For details about allocating resources, see the "Allocating Resources Within a Resource Class" section.) However, the shift in resources between the contexts does not take place immediately unless the appropriate resources are available to accommodate the change. In most cases, to effect a change in resource allocation, you must inform the context administrators involved to ensure that the new resource allocation is possible. Changing the Resource Allocation of a Resource Class 160
Cisco Application Control Engine (ACE) Troubleshooting Guide For example, suppose that context A is using 100 percent of the available resources of the class and you want to allocate 50 percent of the resources to context A and 50 percent of the resources to context B. Although the CLI accepts your resource allocation commands, context B cannot allocate 50 percent of the resources until context A deallocates 50 percent of its resources. In this case, you must perform the following: Inform the Context A administrator to start deallocating resources Inform the Context B administrator to start allocating resources after the Context A administrator releases the resources Note: As resources are released from other contexts, the ACE assigns the resources to resource-starved contexts (contexts where the resource-class minimum allocations have not been met).
Displaying the ACE Resource Allocation and Usage

To view the current resource allocation in your ACE, enter the following command:
ACE_mdule5/Admin# show resource allocation --------------------------------------------------------------------------Parameter Min Max Class --------------------------------------------------------------------------acl-memory syslog buffer conc-connections mgmt-connections proxy-connections bandwidth connection rate inspect-conn rate syslog rate regexp sticky xlates ssl-connections rate mgmt-traffic rate mac-miss rate throughput 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% default default default default default default default default default default default default default default default default
To view the current resource usage, enter the following command:

ACE_mdule5/Admin# show resource usage Allocation Resource Current Peak Min Max Denied ------------------------------------------------------------------------------Context: Admin conc-connections 0 0 0 8000000 0

mgmt-connections proxy-connections xlates bandwidth throughput mgmt-traffic rate connection rate ssl-connections rate mac-miss rate inspect-conn rate acl-memory sticky regexp syslog buffer syslog rate 2 0 0 1094 938 156 1 0 0 0 23776 0 0 0 0 8 0 0 80192 75902 4290 28 0 0 0 28616 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100000 1048574 1048574 625000000 500000000 125000000 1000000 5000 2000 6000 78610432 0 1048576 4194304 100000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Note: All bandwidth values are in bytes per second. To convert to bits per second (bps), multiply the values by eight. The ACE guarantees 1 Gbps of bandwidth for management traffic. So, the total bandwidth for a 4-Gbps ACE license is actually 5 Gbps. Throughput is still 4 Gbps. To display the data plane resource allocation and usage and to cross-check the output of the above two commands, enter the following command:
ACE_module5/Admin# show np 1 me-stats -L0 Resource limts for context : 0 Rate Configured Counters Policer Name Min Max min-toks bandwidth: 0 ee6b280 0 throughput: 0 ee6b280 0 mgmt-traffic rate: 0 3b9aca0 0 connection rate: 0 7a120 0 ssl-connections rate: 0 9c4 0 mac-miss rate: 0 3e8 0 inspect-conn rate: 0 bb8 0 Resource Configured Policer Name Min Max conc-connections: 0 3d0900 mgmt-connections: 0 c350 proxy-connections: 0 7ffff ip-reassemble buffer: 0 0 tcp-ooo buffer: 0 0 regexp: 0 0 xlates: 0 7ffff
max-toks ee6b0fa ee6b280 3b9aca0 7a120 9c4 3e8 bb8
peak-toks d8a4 d8a4 a0e 11 0 0 0
deny 0 0 0 0 0 0 0
Counters Min Max 0 0 0 0 0 0 0 0 0 0 0 0 0 0
peak 0 4 0 0 0 0 0
deny 0 0 0 0 0 0 0
The Admin context has a context ID of 0. To display the resource allocation and and usage statistics for another context, change the "0" in the "-L<context_id>" parameter to the context ID of another context.
Displaying the ACE Resource Allocation and Usage
162

Ace Ts Guide

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ace Ts Guide

Uploaded by

Copyright:

Available Formats

Cisco Application Control Engine (ACE) Troubleshooting Guide Welcome to Cisco DocWiki. We encourage registered Cisco.

Creating a PDF of the ACE Troubleshooting Wiki

ACE Appliance Documentation

ACE Appliance Documentation

Cisco Application Control Engine (ACE) Troubleshooting Guide

Overview of the ACE Troubleshooting Process

Verifying the ACE Image

Cisco Application Control Engine (ACE) Troubleshooting Guide

Enabling ACE Logging

Gathering ACE Troubleshooting Information

Rebooting the ACE

Using show Commands

Verifying the ACE Image

Cisco Application Control Engine (ACE) Troubleshooting Guide

Capturing Packets in Real Time

2. Enter the capture command, for example:

Cisco Application Control Engine (ACE) Troubleshooting Guide

ACE_module5/Admin# capture CAPTURE1 stop

5. Copy the packet capture to disk0: by entering the following command:

Capturing Packets in Real Time

Cisco Application Control Engine (ACE) Troubleshooting Guide

Capturing Packets in Real Time

Cisco Application Control Engine (ACE) Troubleshooting Guide

Copying Core Dumps

After Gathering Troubleshooting Information

Cisco Application Control Engine (ACE) Troubleshooting Guide

If necessary, enter a static route for the gateway.

Verifying the ACE Layer 2 Connectivity

Cisco Application Control Engine (ACE) Troubleshooting Guide

Interface ---------------Gi2/14 Gi2/37 Te3/1 Te5/1 Gi6/2 Te8/1

Verifying the ACE Layer 3 Connectivity

Contacting Cisco Technical Support

Verifying the ACE Layer 2 Connectivity

Contacting Cisco Technical Support

Understanding the ACE Architecture

Overview of the ACE Hardware Architecture

Figure 1. ACE Module Architecture

Cisco Application Control Engine (ACE) Troubleshooting Guide

Overview of the ACE Hardware Architecture

Cisco Application Control Engine (ACE) Troubleshooting Guide

Figure 2. ACE Data Plane

Figure 3. Network Processor Micorengines

Understanding the ACE Traffic Flow

Cisco Application Control Engine (ACE) Troubleshooting Guide

Preliminary ACE Troubleshooting Steps

Checking the ACE Status from the Supervisor Engine

Cisco Application Control Engine (ACE) Troubleshooting Guide

Mod Online Diag Status --- ------------------. . 5 Pass . .

Verifying the MSFC VLAN Configuration

cat6k# show run | include svclc

svclc module 5 vlan-group 123,130,133

Checking the ACE Status from the Supervisor Engine

Cisco Application Control Engine (ACE) Troubleshooting Guide

Establishing a Session with the ACE from the Supervisor Engine

Verifying the ACE is Receiving VLAN Allocations from the MSFC

Verifying the MSFC VLAN Configuration

Cisco Application Control Engine (ACE) Troubleshooting Guide

Verifying the ACE Image

Verifying Your ACE Licenses

filesystem bytes total used bytes free total bytes

Configuring an ACL to Permit Input Traffic to the ACE

Verifying that the ACE is Sending and Receiving Traffic

4933 2437922 0 [not full] [pulling]