Professional Documents
Culture Documents
General Troubleshooting 1
A common part of a Security Administrator’s responsibilities is troubleshooting net-
work problems and using basic troubleshooting techniques to resolve VPN-1 issues.
Troubleshooting guidelines are provided in this chapter to assist the Security Admin-
istrator in defining a problem, identifying possible causes, narrowing causes to one or
a few causes, and finding and testing problem fixes.
Objectives:
19
Chapter 1: General Troubleshooting
Key Terms:
IP forwarding
Default Filter
Initial Policy
Secure Internal Communication (SIC)
Network Address Translation (NAT)
Source NAT
Destination NAT
Client-side NAT
Core file
Securecopy (SCP)
Troubleshooting Guidelines
The variety, flexibility, and complexity of the Check Point product suite
can make every problem seem unique. Despite the challenges inherent in
maintaining and administering rapidly evolving security and connectivity
solutions, standard troubleshooting methods are still relevant. Apply the
guidelines in this section when troubleshooting NGX issues.
VPN Example
For example, a remote user is at a hotel trying to connect to the Security
Gateway through SecureClient, and it fails to communicate with the site.
When the remote user is at home the connection works fine, and no
other users are affected. There are no logs in SmartView Tracker for this
connection.
Using the two questions previously stated, you can:
Determine the desired activity: SecureClient PC connects to the
Security Gateway and creates a VPN tunnel. This does not occur for a
particular user.
Determine what is happening, in observable and objective terms: The
Client attempts a connection but there is no response from the
Gateway. SmartView Tracker shows no logs for this user.
monitor on the Security Gateway shows no traffic for the Client. From
this information we assume that main mode traffic from the Client is
being blocked on a router upstream from the Client. The user edits the
Advanced Properties and checks Visitor Mode. After enabling visitor
mode, which works on port 443 instead of 500, the connection succeeds.
IP Forwarding
IP forwarding is the operating system’s ability to forward packets from
one interface to another. The default behavior prior to the VPN-1
installation may vary depending on the OS. By default, SecurePlatform
disables IP forwarding during boot and while the default filter is loaded
to protect the Security Gateway during installation. Once the firewall
services are started and a policy is installed from the SmartCenter
Server, IP forwarding is enabled. When the firewall services are stopped,
IP forwarding is disabled. You can manually enable IP forwarding for
testing.
Enabling/Disabling IP Forwarding
Solaris
Enable IP forwarding on Solaris by running ndd:
ndd -set /dev/IP ip_forwarding 1
SecurePlatform
Verify the IP forwarding setting on SecurePlatform and
SecurePlatform Pro, by checking the value in the ip_forward file:
more /proc/sys/net/ipv4/ip_forward
Windows
Enable IP forwarding on Windows 2000 Server or Windows 2003
Server, check the value of the key IPEnableRouter in the Registry.
Enabling the Remote Access Server (RAS) service can also be used to
enable IP forwarding. The value should be 1. The path to the Registry
key is:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Tcpi
p\parameters\IPEnableRouter
Routing
As a multihomed host, an NGX Gateway has routes automatically
generated for its immediate networks, (external and internal). The
Gateway can only have one default Gateway (or default route) pointing
to its upstream router. If there is more than one internal network
connecting to an internal router behind the Gateway, add static routes
on the Gateway to reach the remote internal networks from the
Gateway.
Connectivity
To test connectivity with the NGX Gateway in place, ping through the
Gateway from internal nodes to nodes on the external side of the
Gateway, or Ping to the upstream router. Run a ping test as follows:
1. Run fw unloadlocal on the Gateway.
When using RFC defined addresses for internal networks, ping test
replies from the Internet will not be received by the internal hosts.
6. If you can Ping from the Gateway to the Internet, but cannot reach
the Internet from an internal network, IP forwarding may not be
enabled on the Gateway’s OS.
7. If you can ping all the way through, install a simple Rule Base with
necessary rules (for example, outbound HTTP), then browse known
Internet sites.
To resolve FQDN names, internal hosts must have a DNS server, either
on an internal network or hosted by an ISP on the Internet. Domain
Name Service over UDP must be allowed.
The default filter can be changed to the drop filter to block all
communication into and out of the Security Gateway during boot. This
is done by using the fw defaultgen command.
1. These two filters are located in $FWDIR/lib/
a defaultfilter.boot - general default filter
b defaultfilter.drop - drop default filter
2. Copy over and rename the relevant desired Default Filter Inspect file
(defaultfilter.boot or defaultfilter.drop) to $FWDIR/conf/
defaultfilter.pf.
3. Compile the default filter by running:
fw defaultgen
If the Security Policy has not yet been installed, run cpconfig to
regenerate the Initial Policy.
Q.) One internal host behind an NGX Gateway cannot connect to the
Internet. This host has just been added to the internal network. All other
hosts from the same network segment can connect to the Internet. In the
Rule Base, there is a rule accepting outbound HTTP traffic for the
entire network, and the rule is tracked as Log. When you open
SmartView Tracker, you find no logs from that problematic host. What
is the next reasonable step for troubleshooting this problem?
A.) Check the routing table on that host, and make sure the default-
Gateway setting is correct. Test connectivity, using ping or traceroute,
from the host to the Gateway, or beyond.
Q.) You find a log indicating HTTP is accepted, the source is that host,
and the rule number is correct. But the host’s browser displays “page
cannot be displayed”. What is the next reasonable step for
troubleshooting this problem?
A.) Run fw monitor, to see if the reply packet returns to the Gateway’s
external interface.
Once the VPN-1 services start, the default filter is replaced by the
Initial Policy. The Initial Policy allows SmartConsole to connect to a
SmartCenter Server, but still blocks all other traffic to the Gateway. The
Initial Policy is enforced on the Gateway until a policy is installed from
the SmartCenter Server. Once a policy is installed, the Initial Policy is
never loaded again.
When the SIC activation key is reset through cpconfig on the Security
Gateway, the default filter is loaded until SIC is re-established with the
SmartCenter Server and a policy is installed.
You can verify whether the Initial Policy is installed by running fw stat
from the Gateway’s Command Line Interface (CLI). To remove the
Initial Policy, run fw unloadlocal on the Gateway. When a Policy is
unloaded from a Gateway, the Gateway is vulnerable, it accepts anything
that is coming to it, like a router without any ACLs. An Administrator
should limit the amount of time a Policy is unloaded from the Gateway.
Q.) You discover no traffic is passing through the NGX Gateway. You
cannot open SmartConsole or SmartView Tracker. You cannot log in to
the Gateway in SSH from the internal or external network. What is the
next troubleshooting step to take?
A.) Unload the Policy, using fw unloadlocal from the console.
Q.) After you log in to the console, you find out IP forwarding is
enabled. From the Gateway, you can ping the upstream router and
known sites. But the internal network still has no Internet access. The
Rule Base has not changed. How do you find out if there are any
objects that have been modified, without viewing each object’s
properties?
A.) Select the SmartView Tracker > Audit screen from the console.
Changes to objects and rules are recorded.
Root Causes
As a baseline for troubleshooting SIC and ICA related issues, test the
following:
Connectivity: Is any traffic (not just SIC) able to reach the Gateway?
Are the necessary ports open and/or available?
Domain name and IP resolution: Verify the entries in the host file are
accurate on the Security Gateway and SmartCenter Server.
Time: If the SmartCenter Server and the Security Gateway are located
in different time zones, verify that a date/time discrepancy is not
causing the conflict.
Certificate Revocation List (CRL): Verify that the SIC Certificate is
not in the CRL, or that the CRL is still reachable for current
Certificates.
Check Point also includes the ICA Management Tool in VPN-1 NGX,
which can be configured on a SmartCenter Server and used
independently of SmartDashboard to view and manage Certificates:
Refer to the SmartCenter user guide and sk30501 “Setting up the ICA
Management Tool” at https://supportcenter.checkpoint.com/
supportcenter/index.jsp, for configuration information.
Maintaining SIC
The following are recommended best practices to set up and maintain
SIC.
Using Correct FQDN to Initialize ICA
If the FQDN for the SmartCenter Server is not correct, the ICA cannot
initialize successfully. Make sure the FQDN has the correct hostname
and domain name. Make sure the SmartCenter Server’s hostname is
entered correctly in the hosts file.
Avoiding Renaming Gateway Object
The Certificate issued by the ICA (SmartCenter Server) is for a specific
hostname. Once the hostname has changed, the Certificate is no longer
valid. Plan carefully in terms of the naming conventions for all of your
Gateways, including the ICA itself, before you start installing and
configuring. If you must rename the Security Gateway after SIC is
established, follow the steps below to re-establish SIC to the
SmartCenter Server:
On the SmartCenter Server, make sure its hosts file has the new
hostname and correct IP address for the relevant Gateway.
From SmartDashboard:
1. Edit the Gateway object.
2. Click Communication > Reset.
3. Rename the Gateway object.
4. Click Communication > Initialize and enter the new Activiation
Key.
5. Click Initialize.
If both conditions are met, try to reinitiate SIC with the following steps:
1. On the Security Gateway, use cpconfig to initialize SIC, by typing a
new one-time password.
2. In SmartDashboard and the Communications screen of the Gateway
object, reset SIC.
3. Initialize SIC by typing the same one-time password as used for the
Gateway.
4. To verify SIC click the Test SIC Status button.
Synchronizing clocks
For SIC to initialize successfully, the clocks on the SmartCenter Server
and respective Gateway must be set accurately and synchronized
properly. Verify correct date and time on the operating systems. If the
SmartCenter Server and remote Gateway reside in two different time
zones, the remote Gateway may need to wait for the Certificate to
become valid.
Q.) Your SmartCenter Server is behind your organization’s perimeter
Gateway, with Static NAT configured on the perimeter Gateway. You
have a new NGX Gateway in another city, and you must set up SIC.
When you try to initialize SIC, you receive the error “initialized but not
trusted”. What are reasonable steps to troubleshoot this error?
A.) Check the hosts file on the remote Gateway, and make sure the
SmartCenter’s hostname resolves to its public IP address. Check if there is
any rule in the Policy blocking traffic between the SmartCenter Server and
remote Gateway.
Resetting SIC
The term “resetting SIC” is often used interchangeably for two different
actions. Each has a different level of severity associated with it,
depending on the context.
From SmartDashboard:
1. Edit the Security Gateway object, click the Communication button,
and click Reset in the Communication screen. This revokes the
Gateway’s Certificate, and changes SIC status to “Not initialized.”
2. Enter the one-time password twice.
3. Click Initialize. If the connectivity is good and no Policy is blocking
the connectivity, SIC status should read “SIC established” in several
seconds.
4. Install the Policy on the Gateway.
Hide NAT
In Hide NAT, the source is translated, the source port is modified and
translation occurs server side. As shown in the illustration below, see the
source of 10.1.1.101 going to destination of 192.9.100.10:
4. The server replies to 172.21.101.1 and the Firewall maps the source
port from the state table to the original IP and translates back to the
original IP of 10.1.1.101 and the original source port of 1305.
Static NAT
Static NAT is assigned to a server that needs to be accessed directly
from outside the Security Gateway, so the packet is typically initiated
from a host outside the firewall. When the client initiates traffic to the
static NAT address, the destination of the packet is translated.
Before VPN-1 NGX, all destination NAT occurred at the “server side”
of the kernel, on the outbound side of the kernel closest to the server.
When NAT occurs in this configuration, a host route is required on the
Security Gateway to route to the destination server. As of VPN-1 NGX,
the default method for Destination NAT is “client side”, where NAT
occurs on the inbound interface closest to the client. Assume the client
is outside the Gateway, and the server is inside the Gateway with
automatic Static NAT configured. When the client starts a connection to
access the server’s NAT IP address, the following happens to the
original packet in a client-side NAT:
Original Packet
1. The packet from outside the Gateway arrives at the inbound interface
(pre-in or i) destined for the webserver, and passes Security Policy
and NAT rules.
2. If accepted, the packet information is added to the connections table
and the destination is translated on the post-in side of the interface
(I) before it is routed.
3. The packet arrives at the TCP/IP stack of the NGX Gateway, and is
routed to the outbound interface (o).
4. The packet is then forwarded through the kernel (O) and routed to
the webserver.
Reply Packet
1. The webserver replies and hits the inbound interface (pre-in or i) of
the Gateway.
2. The packet is passed by the Policy, since it is found in the
connections table and arrives at the pre-out (I) side of the kernel.
3. The packet arrives at the TCP/IP stack of the Gateway, and is routed
to the outbound interface (post-in or o).
4. The packet goes through the outbound interface and is translated to
the static NAT IP address as it leaves the security Gateway (O). The
source port does not change.
Debugging NAT
In this section we will discuss several useful tools to debugging NAT:
SmartView Tracker
fw monitor
Kernel Debugging
SmartView Tracker
After confirming the configuration of the NAT object and rules in
SmartDashboard, open SmartView Tracker to view the traffic leaving
the Security Gateway. Go to View > Query Tree and choose xlate src
and xlate dst. This will show source and destination NAT for packets
running in and out of the Security Gateway.
For example, a host on the inside network is being hide translated
behind the Security Gateway. The src field should show the physical IP
address of the client. The xlate src column should show the translated
address.
fw monitor
Another useful troubleshooting tool is running fw monitor on the
Security Gateway to determine if the packet is translating. Take the
example above. In the fw monitor the src packet will hit the inspection
points i, I, o and then O the packet will translate to the external interface
of the firewall. For more information regarding inspection points
through the firewall kernel, see Chapter 5 Protocol Analyzers.
Another example, is a webserver behind the Security Gateway that is
statically translated. In this example, the webserver is translated to a
specific IP address on the same network as the external interface. The
fw monitor capture can show if translation is happening client-side or
server-side.
Kernel Debugging
The primary command for observing the NGX kernel’s actions on a
packet is fw ctl debug. It is also used for configuring debugging on
almost any action that VPN-1 NGX can take on a packet or connection.
The standard format for the command is as follows:
fw ctl debug
Running this command from the CLI produces a list of currently
running modules and debugging flags. When the command is issued
with an argument following it, the default kernel module acted on is the
fw module.
Some of the more commonly used arguments for fw ctl debug include:
-buf Sets the buffer size used by the debug process for this session
+ <flag_name> Sets the debugging to add the listed flag(s) to the currently
flagged and running session
Note: Adding a flag name to the command without the + ,
such as ...
fw ctl debug smtp
... Sets the debugging session to that flag only.
Debug Explanation
These commands debug the NAT process in the kernel, and produce
output with debug information. To reset debugging flags, run fw ctl
debug 0.