Professional Documents
Culture Documents
14470_04_2008_c3
Cisco Public
Troubleshooting Firewalls
BRKSEC-3020
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
Note: Cisco IOS firewall is covered in SEC-3000 (Troubleshooting Cisco IOS Security Features) and will not be covered in this presentation
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
Packet Flow
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Example Flow
Flow
SRC IP: 10.1.1.9 DST IP: 198.133.219.25 SRC Port: 11030 DST Port: 80 Protocol: TCP
Interfaces
Source: Inside Destination: Outside
Client: 10.1.1.9
In
Servers
Packet Flow
Eng
er rtn Pa
si de
DM
Accounting
Ho st in g
With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Server: 198.133.219.25
Cisco Public
Outside
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
Check first for existing connection If connection exists, flow is matched; bypass ACL check If no existing connection
TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks Established Connection:
ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
10
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
11
First packet in flow must match a translation rule* A quick route lookup is done only to determine egress interface Translation rule can be to NAT, or not to NAT NAT order of operations dictates what happens with overlapping translation rules Once translation rule is matched, connection is created Translation Exists:
ASA-5540# show xlate debug NAT from inside:10.1.1.9 to outside:172.18.124.68 flags - idle 0:00:07 timeout 3:00:00
12
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
1. nat 0 access-list (nat-exempt) 2. Match existing xlates 3. Match static commands (Cisco ASA/PIX first match; FWSM best match)
Static NAT with and without access-list Static PAT with and without access-list
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
First Match
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
13
Inspections are applied to ensure protocol compliance (Optional) Customized AIC inspections NAT embedded IPs in payload Additional security checks are applied to the packet (Optional) Packets passed to Content Security and Control (CSC) Module
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
14
Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
15
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) Egress interface is determined first by translation rules If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface Example:
Inside
172.16.0.0/16
Outside DMZ
172.16.12.0/24 172.16.12.4
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
16
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Once on egress interface, an interface route lookup is performed Only routes pointing out the egress interface are eligible Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
ASA-6-110001: No route to 209.165.202.130 from 10.1.1.9
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
17
Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed Layer 2 rewrite of MAC header If Layer 2 resolution failsno syslog show arp will not display an entry for the L3 next hop debug arp will indicate if we are not receiving an ARP reply
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
18
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
Packet is transmitted on wire Interface counters will increment on interface Output hardware and software queues indicate buffering at driver level, interface is busy
ASA-5540# show interface gb-ethernet0 interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.626c IP address 172.18.124.64, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 3529518 packets input, 337798466 bytes, 0 no buffer Received 32277 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 5585431 packets output, 359059032 bytes, 0 underruns input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (0/2) software (0/0)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
19
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
20
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
10
Cisco ASA platforms have software imposed connection limits; Cisco PIX platforms do not (bound by RAM)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
21
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
22
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
11
Software Hardware
Session Manager NP 3
Control Point ACL Compilation, Fixups, Syslog, AAA in Software Session Manager Session Establishment and Teardown, AAA Cache, ACLs
Fast Path NP 1
Fast Path NP 2
FWSM
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
23
24
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
12
FWSMHardware Limits
FWSM has several hardware limits that should be considered in your network design Limits are hard set, but vary based on single or multimode Some limits include:
2.3 (Multimode) ACEs AAA Rules Global Statements Static NAT Statements Policy NAT ACEs NAT Translations Connections Route table entries Fixup/Inspect Rules 56,627 (9,704) 3,942 (606) 1K (1K) 2K (2K) 3,942 (606) 256K (256K) 999,990 (999,990) 32K (32K) 32 (32 per)
Increase over 2.3 Increase over 3.1
3.1 (Multimode) 72,806 (11,200) 6,451 (992) 4K (4K) 2K (2K) 1,843 (283) 256K (256K) 999,990 (999,990) 32K (32K) 4147 (1,417)
4.0 (Multimode) 100,567 (14,801) 8,744 (1,345) 4K (4K) 2K (2K) 2,498 (384) 256K (256K) 999,990 (999,990) 32K (32K) 5621 (1,537) 3747 (576)
X X
25
Filter Statements 3942 (606) 2764 (425) *Complete list in FWSM docs, Appendix A (Specifications)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
Classifier in Multimode
FWSM has a single MAC address for all interfaces Cisco ASA/PIX has single MAC for shared interfaces (physical interfaces have unique MACs)
Cisco ASA/PIX 7.2 introduces an option to change this
When the firewall receives a packet, it must classify it to determine where to send the packet Packets are classified based on the following
Unique ingress interface/VLAN Packets destination IP matches a global IP
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
26
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
13
Classifier in Multimode
Example Inbound traffic is classified to context CTX3, based on the global IP in the static
FWSM Inside 10.1.1.2 Inside VLAN 5 10.1.2.2 Inside VLAN 6 10.1.3.2 static (inside,outside) 10.14.3.89 10.1.3.2
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
SRC IP 192.168.5.4
VLAN 4
CTX2
.2
CTX3
.3
Shared Interface
27
Classifier in Multimode
If the firewall is unable to classify a packet, the following syslog message is generated in the Admin context*
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 10.14.3.25/80
28
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
14
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
29
Failover Basics
Active/standby vs. primary/secondary Serial vs. LAN failover Stateful failover (optional) A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall Both firewalls swap MAC and IP addresses when a failover occurs Level 1 syslogs will give reason of failover
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Internet
Corp
Cisco Public
30
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
15
Interface Monitoring
rcv 73 73
rerr 0 0
Cisco Public
31
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
32
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
16
The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
33
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
34
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
17
Other host -
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
35
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
36
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
18
Troubleshooting Tools
Syslogs Debug commands Show commands Packet capture Packet tracer
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
37
Uses of Syslogs
Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available
Archival Purposes Debugging Purposes
Console
Syslog Server Internet
SSH Client
Trap SNMP Server Syslog . Buffered
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
38
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
19
Log Level 0 1 2 3 4 5 6 7
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
39
Log Level 0 1 2 3 4 5 6 7
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
40
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
20
Problem
You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level 7 (debug)
%PIX-7-111009: User johndoe executed cmd: show run
The problem is we dont want to log all 1602 other syslogs that are generated at debug level
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
41
Or
ASA(config)# logging message 111009 level error
Or
ASA(config)# logging message 111009 level 7
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
42
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
21
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem 2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution 3. Debugs are not conditional* 4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
43
Valuable tool used to troubleshoot connectivity issues Provides interface and translation information to quickly determine flow Echo-replys must be explicitly permitted through ACL, or ICMP inspection must be enabled
Example debug icmp trace output
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
44
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
22
Solution
Create a logging list with only syslog ID 711001 Enable debug output to syslogs Log on the logging list
ASA(config)# logging list Networkers message 711001 ASA(config)# logging debug-trace ASA(config)# logging trap Networkers
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
45
Use output filters to filter the output of show command to only the information you want to see To use them, at the end of show <Command>, use the pipe character | followed by
begin include exclude grep grep v
Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
46
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
23
Examples
Display the interface stats starting with the inside interface
show interface | begin inside
47
48
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
24
Show Traffic
The show traffic command displays the traffic received and transmitted out each interface of the firewall
ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
49
50
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
25
Connection Flags
awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, outside back connection, F - outside FIN, f - inside FIN, group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, incomplete, J - GTP, j - GTP data, K - GTP t3-response Skinny media, M - SMTP data, m - SIP media, n - GUP outbound data, P - inside back connection, q - SQL*Net data, outside acknowledged FIN, UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, inspected by service module
TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
51
ExampleConnection Build Up
1. Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA 2. The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A 3. The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag) 4. The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection 5. Finally, the inside device has sent a data packet and the connection is updated to include the O flag
1 5 3 SYN+ACK Data SYN ACK Connection Flags 42
UI UIO s A UaA
Inside Client Outside Server
52
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
26
ExampleConnection Teardown
1. Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface 2. The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR 3. The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
3 1
Connection Flags 2
UfFRr UfUfFR
Inside Client Outside Server
53
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
54
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
27
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
55
Description Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout Flow Was Terminated by Inspection Feature Flow Was Terminated by IPS Flow Was Reset by IPS Flow Was Terminated by TCP Intercept SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer Than the Timeout Value Flow Was Terminated Due to IPS Card Down Back Channel Initiation from Wrong Side
Cisco Public
56
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
28
TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Size Variation Tunnel Has Been Torn Down Uauth Deny Unknown Xlate Clear
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Connection Terminated Due to a Variation in the TCP Window Size Flow Terminated Because Tunnel Is Down Connection Denied by URL Filtering Server Catch-All Error User Executed the Clear Xlate Command
Cisco Public
57
show local-host
A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers
ASA# show local-host Interface inside: 1131 active, 2042 maximum active, 0 denied local host: <10.1.1.9>, TCP connection count/limit = 1/unlimited TCP embryonic count = 0 TCP intercept watermark = 50 UDP connection count/limit = 0/unlimited AAA: user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10) absolute timeout: 0:05:00 inactivity timeout: 0:00:00 Xlate(s): Global 172.18.124.69 Local 10.1.1.9 Conn(s): TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
58
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
29
show service-policy
The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0 Inspect: ftp, packet 43, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: http, packet 562, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 349, drop 0, reset-drop 0 Inspect: esmtp, packet 0, drop 0, reset-drop 0 ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
59
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
60
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
30
10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893
61
Packet Capture
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
Capture command first introduced in Cisco PIX 6.2; FWSM 2.3; it deprecates the debug packet command 7.2(3) and 8.0(3) added a real-time option ASDM 6.0 adds a capture wizard Capture sniffs packets on an interface that match an ACL, or match line Key steps
Create an ACL that will match interesting traffic Define the capture and bind it to an access-list and interface View the capture on the firewall, or copy it off in .pcap format
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
62
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
31
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
63
Packets are captured at the first and last points they can be in the flow Ingress packets are captured before any packet processing has been done on them Egress packets are captured after all processing (excluding L2 source MAC rewrite)
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
64
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
32
Internet
198.133.219.25
10.1.3.2
192.168.2.2
Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
65
Step 3: Have inside user access www.cisco.com Step 4: Copy the captures off to a TFTP server
! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap
66
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
33
Inside CAP
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
67
ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
68
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
34
Session Manager NP 3
Capture requires an ACL to be applied Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
FWSM
FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware
69
Packet Tracer
Packet tracer is the future of troubleshooting configuration issues (and many other issues) Introduced in version 7.2 and ASDM 5.2 A packet can be traced by:
Defining the packet characteristics via the CLI Capturing the packets using the trace option
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
70
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
35
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
71
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
72
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
36
73
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
74
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
37
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
75
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
76
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
38
Define Packet
Final Result
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
77
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
78
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
39
Case Study
Intermittent Access to Web Server
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
79
NATed to 10.1.1.50
Clients
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
80
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
40
Traffic Spike
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
81
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
82
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
41
ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
83
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
84
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
42
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
85
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
86
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
43
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
87
per-client-max
TCP Intercept
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
88
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
44
Cisco Public
89
Case Study
Poor Voice Quality
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
90
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
45
100 Mbps
Cable Modem
2 Mbps
WAN
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
91
WAN
2 Mbps
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
92
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
46
To view statistics on the operation of the shaper, use the command show service-policy shape
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
93
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
94
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
47
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
95
Tools
ASDM Output interpreter Online learning modules
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
96
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
48
ASDM
Run as a standalone application using the ADSM Launcher This allows for one-stop access to multiple firewalls ASDM 6.0 adds Upgrade Wizard to upgrade ASA and ASDM software direct from cisco.com ASDM 6.1 works with both ASA 8.1 and 8.0 releases ASDM 6.1F works with FWSM 4.0, 3.2 and 3.1 releases
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
97
Device Information
Real-Time Syslogs
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
98
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
49
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
99
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
100
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
50
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
101
ASDM 6.0
Drag-and-drop and inplace editing for simplified policy editing User interface customization with dockable windows and toolbars New Firewall Dashboard that provides at-a-glance status of firewall services Live ACL hitcount in firewall rule table for easy policy auditing
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
102
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
51
Output Interpreter
Linked Off the Technical Support and Documentation Tools and Resources Section on CCO
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
103
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
104
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
52
Direct link
http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.html
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
105
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Tools Best Practices
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
106
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
53
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
107
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
108
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
54
GD
GD
7.0(2)
7.0(4) 7.1(2)
7.0(5)
7.0(6)
7.0(7)
7.0(8)
Maintenance Trains
7.2(2)
7.2(3)
7.2(4) 8.0(3)
7.0(1)
7.1(1)
7.2(1)
8.0(2)
8.1(1)
8.2(1)
109
Cisco has Announced the End-of-Sale and End-of-Life Dates for Cisco PIX Security Appliances
End of Sale: July 28, 2008 Last day of sale for software, accessories, and licenses: January 28, 2009 End of Software Maintenance Releases: July 28, 2009 End of Support / End of Life: July 27, 2013
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
110
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
55
Q and A
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
111
Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books I recommend: Cisco ASA: All-in-One Firewall, IPS, and VPN Adaptive Security Appliance Available Onsite at the Cisco Company Store
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
112
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
56
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
113
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
114
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
57
Appendix
Lucky You This appendix contains some extra information which you may find useful, but I just didnt have enough time to cover in the lecture Enjoy :-)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
115
Appendix
Release trains Gotchas upgrading Cisco PIX to 7.0 Cisco PIX password recovery Case study
Out of Order Packet Buffering TCP MSS issue Out of memory High CPU
Online Tools (Network Professionals Connection, Bug Toolkit) Information to include when opening a TAC case
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
116
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
58
6.0(2)
6.0(3)
Maintenance Trains
6.3(3)
6.3(4)
6.0(1)
6.1(1)
6.2(1)
6.3(1)
117
Cisco PIX
6.0(1)
Cisco PIX/ASA
7.1(1) 7.2(1)
7.0(1)
FWSM
1.1(1)
Feature Releases
2.2(1)
2.3(1)
3.1(1)
3.2(1)
4.0(1)
SafeHarbor
1.1(2)
1.1(3)
2.3(2)
3.2(2)
SafeHarbor
3.2(4)
GD
3.1(6)
3.1(10)
Time
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
118
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
59
The upgrade process automatically converts your pre-7.0 config to the new 7.0 CLI If there were any errors during the config conversion process, view them by issuing
show startup-config errors
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
119
http://www.cisco.com/warp/public/110/34.shtml
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
120
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
60
121
Case Study
Out-of-order packet buffering
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
122
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
61
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
123
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
124
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
62
How to fix?
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
125
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
126
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
63
Case Study
TCP MSS (Maximum Segment Size)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
127
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
128
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
64
SYN+ACK MSS=1400
DATA=1390
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
129
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
How to fix?
access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
130
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
65
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
131
Case Study
Out of Memory
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
132
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
66
133
pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
134
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
67
135
Traffic Flow
Vast majority of traffic is coming in the inside interface and going out the outside interface
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
Inside
Outside
136
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
68
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
137
Only Show Lines That Have the Word host or count/limit in Them
= 146608/unlimited = 0/unlimited
Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
138
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
69
saA saA saA saA saA saA saA saA saA saA
139
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
TCP intercept wont help because the source address is valid Limiting the maximum number of connections each internal host can have is the only option
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
140
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
70
Note: The local-host must be cleared before the new connection limits are applied
pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, The Infected Host Is TCP connection count/limit = 50/50 TCP embryonic count = 50 Limited to 50 TCP TCP intercept watermark = unlimited Connections UDP connection count/limit = 0/unlimited . . .
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
141
Things look much better now Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?
nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
142
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
71
Case Study
High CPU Usage
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
143
For more information on the output of the show processes command, see
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
144
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
72
In One Minute, These Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run, and Are Not Counted in the CPU Usage
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
145
146
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
73
to to to to to to
on on on on on on
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
147
Syslog service was down on the syslog server ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it Cisco PIXs IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem
Syslog Server Lab Outside Syslog Message ICMP Unreachable IDS Syslog Message
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
148
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
74
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
149
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
150
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
75
FWSM
Additional Architecture Information
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
151
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
152
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
76
153
FWSM# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 3 Context (004) Accounting -------------------------------------------Context Name
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
154
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
77
155
FWSMResource Rule
FWSM 3.2 introduced
resource-ruleallows further customization of a partition
resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283
156
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
78
FWSMResource Partition
FWSM 4.0 introduced
resource partitionallows customization of the size of individual partitions (multi-context mode)
FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000. The <size> command leads to re-partitioning of ACL Memory. It will not take effect until you save the configuration and reboot.
Before
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 384 384 833 ACL 14801 14801 14801 Filter 576 576 1152 Fixup 1537 1537 3074 Est Ctl 96 96 96 Est Data 96 96 96 AAA 1345 1345 2690 Console 384 384 768 -----------+---------+----------+--------Total 19219 19219 Partition Limit - Configured Limit = Available to allocate 19219 19219 = 0
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
After
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 20 20 43 ACL 770 770 770 Filter 30 30 60 Fixup 80 80 160 Est Ctl 5 5 5 Est Data 5 5 5 AAA 70 70 140 Console 20 20 40 -----------+---------+----------+--------Total 1000 1000 Partition Limit - Configured Limit = Available to allocate 1000 1000 = 0 157
Traffic sourced from, or destined to, the FWSM also goes through the control point
Syslogs URL filtering (WebSense/N2H2) Management traffic (telnet/SSH/HTTPS/SNMP) Failover communications Routing protocols (OSPF/ RIP) etc.
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
AAA (RADIUS/TACACS+)
158
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
79
159
4 3 2 1
4 2 3 1
This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM Examples: CIFS, FTP, AFP, backups FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the Completion Unit). In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewall.
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
160
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
80
Online Tools
Networking Professionals Connection Bug Toolkit
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
161
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
162
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
81
http://www.cisco.com/go/netpro
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3 Cisco Public
163
Bug Toolkit
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
164
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
82
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
165
Version
Search Keywords
Severity
Status
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
166
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
83
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
167
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
168
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
84
At a minimum include:
Detailed problem description Output from show tech
Optionally include:
Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format)
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
169
Syntax
keywords: Hostname Configures the prompt to display the hostname Domain Context only) Priority State Slot Configures the prompt to display the domain Configures the prompt to display the current context (multi-mode Configures the prompt to display the 'failover lan unit' setting Configures the prompt to display the current traffic handling state Configures the prompt to display the slot location (when applicable)
Example
FWSM(config)# prompt hostname domain priority state slot FWSM/cisco.com/sec/actNoFailover/4(config)#
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
170
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
85
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
171
Starting with Cisco PIX 6.3/FWSM 2.3, all debugs can be disabled simultaneously by issuing no debug all or undebug all (un all for short)
ASA# show debug debug icmp trace debug sip ASA# un all ASA# show debug ASA#
. .
BRKSEC-3020 Troubleshooting Firewalls 14470_04_2008_c3 2008 Cisco Systems, Inc. All rights reserved.
Cisco Public
172
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
86
BRKSEC-3020 Troubleshooting Firewalls 2008 Cisco Systems, Inc. All rights reserved. 14470_04_2008_c3
Cisco Public
173
2008, Cisco Systems, Inc. All rights reserved. Networkers SEC-3020 Troubleshooting Firewalls
87