The purpose of this paper is to help troubleshoot your firewall.  This paper does not cover specific  FW-1 errors.  Phoneboy has already done an excellent job of that.  This paper covers general techniques in identifying what the problem is.   These techniques are based on Firewall-1 running on the Solaris operating system.  However, most of the techniques are platform independent, so they should apply in most situations, including  for NT and Nokia.  If you have a technique of your own you would like to add, please let me know at lance@honeynet.org.

1.  CONNECTIONS BEING DROPPED

Often Firewalls drop/deny connections for unknown reasons.  It becomes difficult to determine which Firewall is dropping what.  Here are several techniques in troubleshooting this.

ssh/telnet to the Firewall in question.  At the command line, type

fw log -ft -n | grep <ip address of system that cannot get through the firewall>

This command will watch the Firewall logs for all connections, and then give you the ones you are grepping for.  It is critical to use the "-n" paramter, as this disables name resolution.  This makes the process both faster and more reliable.  This will show you exactly what the firewall logs see.

Another method is to use snoop.  Snoop will pick up ALL traffic before the Firewall filters it.  Regardless if the firewall drop/deny/accepts a packet, snoop will see it first.  An excellent technique is to compare what snoop sees to what the FW-1 logs see.  For more info on snoop, see Secrets of Snoop. For other operating systems, you can use snort, my weapon of choice. You can find both Unix and Windows version of snort at www.snort.org

2.  FW-1 REMOTE FILTER/INSPECTION MODULES CANNOT COMMUNICATE WITH MANAGEMENT MODULE

Remote FW modules must authenticate to communicate with the management module.  The method of uthentication and communication is defined in $FWDIR/lib/control.map.  If systems are licensed for encryption, then the method of authentication will be fwa1, which is also encrypted.  If systems are not licensed for encryption, then the method of authentication will be skey.

If you are having authentication problems (such as logging not working to MM, not able to push a rule base, etc) reset the putkeys (passwords used to authenticate).  One trick to use is the "-n" option, like this.

fw putkey -p <password> -n <local ip> <remote ip>

The '-n' option determines the source ip used.  Ensure that you are using the source IP that communicates with the remote IP.  Otherwise, by default 'fw putkey' used the IP that matches with the hostname.

Also, the log file $FWDIR/log/fwd.log will show any authentication or logging errors that are generated (ver 4.0 only).  See tip #12 for more info.
 

3.  RESTARTING fwd DAEMON

For troubleshooting purposes, you can stop and restart the fwd without having to do a complete fwstop;fwstart.  This is handy for troubleshooting problems, such as logging or authentication.  Remeber, fwd does not do any filtering, the actual firewall module does the filtering.  fwd only handles encryption, logging, admin connections, and the security servers.  If you can momentarily disable these services, then you can kill the fwd and then restart it, without affecting the firewall module (the actual filtering).  What I like about this method is you are not having to do an actual fwstop;fwstart.

To kill the fwd daemon.

mozart #fw kill fwd

When you start up the fwd again, make sure you start it the same way it was running before.  Example:

mozart #ps -aef | grep fwd
root 24347     1  0   Aug 08 ?        0:08 fwd 192.168.5.5

mozart #$FWDIR/bin/fwd 192.168.5.5
 

4.  NETWORK ADDRESS TRANSLATION

It is highly recommend that you manually create your own Address translation rules.  Any NAT rules automatically created will appear in all rulebases, regardless if you want them or not.  This can cause alot of problems if are managing several rulebases for different remote modules.
 

5.  SOME HANDY COMMANDS TO KNOW

This command converts a *.W file to a *.pf file.

fw gen rulebase_name.W > rulebase_name.pf

This command adds the given rulebase to rulebases.fws.  rulebases.fws stores all *.W files for the GUI client to read.  If you have alot of rulebases, you may want to reset the rulebases.fws file, then add your latest *.W file to rulebases.fws.

fwm -g rulebase_name.W
 

6.  VIEWING/EDITING RULES FROM THE COMMAND LINE

If you want to VIEW rules from the command line, I highly recommend you view the .pf file, not the .W file.  The .pf file numbers all rules in the rulebase, .W file does not.  Also, the .pf file contains all global properties, .W does not.

If you want to EDIT rules from the command line, I highly recommend you edit the .W file, and not the .pf file.  The .pf file is overwritten every time you do a "fw load" or push a new rulebase.  So, if you edit a .pf file, then push a new rulebase, the .pf file will be overwritten during the compile process, losing your changes.  Editing the .W file saves the changes.
 

7.  SCRIPTS

The commands 'fwstop' and 'fwstart' are not binaries but csh scripts.  To better understand what they are doing, I highly recommend you review the code.
 

8. $FWDIR/conf/product.conf

The file $FWDIR/conf/product.conf determines the makeup of you system (ie, FW module, management module, etc).  You can easily change your FW setup by editing this file.  Example:

cat /etc/fw/conf/product.conf
Unlimit=1
Auth=1
Management=1
FireWall=0

Anything with a 1 beside it is true, 0 is false.  So this system is defined as a Management Module only.
 
 

9.  CONNECTIONS TABLE

To gain a better understanding of how FW-1's connection's table works, check out Understanding the FW-1 State Table.
 

10.  DEBUGGING

A commong way to debug FW-1 commands is to add '-d' option, which means debug.  For example:

fw load -d rulebase.W

Would give verbose output during the load process, giving you more info for the troubleshooting process.

For the truly hardcore debugging types:

Usage: fw ctl debug [-d] <id | option | 0>  [hostname]

Where known options are: all, cookie, crypt, domain, ex, driver, filter, hold, if, install, ioctl, kbuf, ld, log, machine, memory, misc, packet, q, tcpseq, xlate, xltrc, winnt, profile, synatk, media, align, balance, chain

Or:    fw ctl debug -buf [buffer size] [hostname]

to view Debug output:

fw ctl kdebug [-i | [-f] -o]
 

Usage example:

./fw ctl debug -buf 512
./fw ctl debug kbuf
./fw ctl kdebug -f
 

This would show you the kernal memory allocation actions on the firewall.  The other options are self explanitory.  To view Memory Stats:

fw ctl pstat [-h] [-k] [-n]
 

11.  FWINFO

fwinfo is a troubleshooting script that is included with all FireWall-1 distributions (in otherwords, its installed on your firewall).  The script is used to collect information about your system, then you send the results to Checkpoint support to help them troubleshoot your firewall.  All the information this script gathers in important to troubleshooting firewalls.  You may want to run the script and see what information it gives you.  If nothing else, see what commands the script executes to get ideas of what you should be looking for.

12. LOG FILES

FW-1 ver 4.0 has introduced several new log files that can help you in troubleshooting.  All these new log files are ascii text.

mozart #pwd
/var/opt/CKPfw/log

lisa #file * | grep ascii
ahclientd.log:  ascii text  -----> logs all client athentication
ahttpd.log:     ascii text  -----> logs all http security server issues
fw.logtrack:    ascii text  -----> logs all fw logswitches
fwd.log:        ascii text  -----> logs all fwd (daemon) issues
fwui.log:       ascii text  -----> logs who installed what rulebase when
mdq.log:        ascii text  -----> logs all SMTP security server issues

13. PRODUCTION RULEBASES

Often you have to add new rules or modify an existing rulebase, without effecting production traffic. Your boss has told you to change the rulebase, but you can't afford any dropped production traffic, period! So, you are now have the problem that you need to modify the rulebase, which could accidently drop valid production traffic, but you CANNOT afford to drop production traffic. When this is the scenario, you can troubleshoot the following way. Put in the new rules you need, or modify the existing rule, then change the last rule to Accept everything and log. That way, if the production trafic does not get accepted by the rules you thought it would, the last rule will accept them. Now push your new rulebase. If you see any production traffic in your logs being accepted by the last rule, you know you made a mistake because the traffic should have been allowed by an earlier rule. The advantage here is you find the error, but you don't drop any of the production traffic. HOWEVER, this troubleshooting method throws your firewall WIDE OPEN, accepting unauthorized traffic as well. Use this method only if you have to, and only for a short period of time.

14. RULE ZERO

Often you find rule zero as the rule that is denying your packets. What is rule zero, and why is it dropping your packets? Here is an excelent response to that question by Jerald Josephs.

Anti-Spoofing: This is set on the interface tab of your firewall object. If spoof track is set to "log" or "alert", a rule 0 entry will show in your log. A "drop" on Rule 0 typically means that incoming packet violated your anti-spoofing policy for that interface. A "reject" on Rule 0 typically means that an outgoing packet (one that has been accepted by your security policy and routed by the OS) is violating your anti-spoof rules because the packet is being routed out the wrong interface. If your Network Address Translation is misconfigured, you will often have problems with Anti-Spoofing.

Authentication Failures: This is set in the Authentication tab of the rulebase properties. If this is set to "log" or "alert", any failed authentication attempts will show as a rule 0 log.

SYNDefender warnings may get logged as rule 0. The "Display Warning Messages" checkbox in the SYNDefender tab of the rulebase properties is where this can be disabled.

SecuRemote authentications (the successful ones) can also appear as a rule 0 accept. This is controlled by the "Enable Decryption on Accept" checkbox in the Security Policy tab of the Rulebase Properties.

Anything dropped by FireWall-1's IP Options checking will log as rule 0. The logging is controlled by the "IP Options Drop Track" section of the Log and Alert tab of the Rulebase Properties.

15.  ROUTING

For larger organizations with several firewalls, routing can be one of your problems.  When a packet is accpeted by the Firewall, the firewall builds an entry in the state table and expects a return packet.  But what happens when the return packet comes back through a different firewall?  The packet is then dropped, because the second firewall does not have anything in its state table.  I have seen this problem often in large organizations with complex routing and firewall architectures.  Admins often do not realize that packets may not always return on the same path.  A good way to identify this problem is if a firewall is dropping packets that have a source port with a valid service.  Examples include packets with a source packet of 23 (telnet) or 21 (ftp).  To fix this, you have to resolve the routing issues,  make sure return packets are taking the same path as the original packet.
 
 

Author's bio
Lance Spitzner is an active member of the Honeynet Project. He enjoys learning by blowing up systems in his home lab. Before this, he was an Tanker in the Rapid Deployment Force, where he blew up things of a different nature. You can reach him at lance@honeynet.org .