Packet Filtering with IPSec

$Id: ipsec-filtering.html,v 1.18 2001/12/17 17:50:08 ssklar Exp ssklar $

Introduction

Should I take this tutorial?

This tutorial is about using the IP Security (IPSec) feature of IBM's AIX operating system to implement packet filtering for TCP/IP networks. If you are an administrator of RS/6000 systems, and those systems are connected to a publicly accessible network, packet filtering can help reduce the system's network "footprint", and can provide an additional layer of security for your systems, whether a firewall is in place or if all networked computers are directly connected to the Internet.

It is assumed that the reader is familiar with the concepts of TCP/IP networking and IP addressing. After completing this tutorial, the reader will have an understanding of the steps necessary to install and configure IP Security and define filter rules for limiting access to network services.

Though the basic concepts of IP Security and tunneling are touched upon, they are not explored in detail with this tutorial. For more information about these concepts, the reader is directed to the References and resources section.

About the examples

This tutorial is based on the features and software provided with AIX 4.3.3 and AIX 5.1. Though the examples presented within were tested on an RS/6000 running AIX 4.3.3 Maintenance Level 8, the commands, filesets and procedures are identical in both operating system versions.

It is assumed in this tutorial that the IPSec packet filtering will be implemented on a host or hosts directly connected to the Internet, without an intervening firewall or other network-based security mechanism. If this is not the case in the reader's environment, be sure to carefully read the official documentation, listed in the References and resources section. For all environments, the IBM documentation is the definitive source for information about the commands, procedures, and software components involved in AIX IP Security.

Also, IPv4 is used exclusively throughout the examples presented within this tutorial, though IPSec is fully supported (and, in fact, designed for) IPv6 network architectures. Again, refer to the official documentation for any differences in the implementation of IPSec with IPv6 networks.

What is IPSec?

IP Security, known commonly as IPSec, is a protocol developed by the Internet Engineering Task Force (IETF), designed to provide "end-to-end" cryptographically-based security for IP network connections. Though not yet an official standard, compatible IPSec implementations are available for almost all modern operating systems. Inclusion of IPSec is required in every IPv6 implementation, and it has been designed to work equally well with the more common IPv4 system currently in use by most public and private networks.

All IP Security implementations include a common set of protocols and tools to enable interoperatability between different platforms, and provide the following three benefits:

• Authentication: proof that the identity of the host on the other end of the connection is valid and correct.
• Integrity Checking: assurance that no data sent over the network connection was modified in transit.
• Encryption: the rendering of network communications indecipherable to anyone who might intercept the transmitted data.

IPSec implementations also include a method of restricting connections to various services, based on their origin and destination. This feature, often present in firewall devices, is known as packet filtering.

What is packet filtering?

All packets on an IP network originate from an IP address and a port, and are destined for another IP address and port. A packet filter is a physical or virtual device that sits between the endpoints of a connection and determines whether the packet should be permitted to continue to its destination.

That decision is made by comparing various attributes of the packet to rules that are defined by the administrator of the packet filter. Those attributes include (but are not limited to) source address and subnet, source port, destination address and subnet, destination port, protocol, direction of the connection, and fragmentation of the packet.

Packet filtering is often a included as a feature in firewalls and routers. In these devices, packets can not only be permitted or refused based on their characteristics, but also routed to different destinations. The packet filtering functionality in AIX (as well as many other host-based packet filters) does not have this feature: the only choices available are to either refuse to route the packet, or to let it continue to its destination.

Installation and Initial Configuration

Installing the IP Security pieces

The software components needed to implement IPSec are included with AIX on the base installation media. To determine if the required filesets are installed, run the command:

lslpp -L '*ipsec*'

The output from that command should contain the following filesets:

  Fileset                      Level  State  Description
  ----------------------------------------------------------------------------
  bos.msg.en_US.net.ipsec    4.3.3.0    C    IP Security Messages - U.S.
                                             English 
  bos.net.ipsec.keymgt      4.3.3.50    C    IP Security Key Management
  bos.net.ipsec.rte         4.3.3.50    C    IP Security
  bos.net.ipsec.websm       4.3.3.25    C    IP Security WebSM

If any of the above filesets are missing, install them from the AIX 4.3.3 installation media.

One additional piece of software is required: the bos.crypto fileset, found on the AIX 4.3.3 Bonus Pack. The name of this fileset may differ, depending on the country. To determine if this fileset is installed on the system, run the command:

lslpp -L 'bos.crypto*'

and look for output similar to:

  Fileset                      Level  State  Description
  ----------------------------------------------------------------------------
  bos.crypto                 4.3.3.0    C    40 Bit Encryption for IP
                                             Security 
  bos.crypto-priv            4.3.3.0    C    Triple DES Encryption for IP
                                             Security 
  bos.crypto-wt              4.3.3.0    C    56 bit Encryption for IP
                                             Security 

Set up IPSec logging

The IP Security software uses syslog to process messages and errors that it generates. Messages are sent to syslogd at the local4 facility. It is a good idea to setup logging of these messages before activating IPSec, to make troubleshooting easier.

To have syslogd write all messages received at the local4 facility to the logfile /var/adm/ipsec.log, add the following line to the /etc/syslog.conf file:

local4.debug                    /var/adm/ipsec.log 

Create the empty log file by running the command touch /var/adm/ipsec.log, and then make syslogd aware of the changes to its configuration by running the command refresh -s syslogd.

Activating the IPSec Device

Before activating the IPSec modules, it is important to make sure that there are no filters already defined that might cause the system to refuse connections upon activation. These filters might have been left on the system by a previous IPSec installation. To prevent problems, run the command lsfilt -v4, and examine the output.

There should be three "rules" displayed: these rules are predefined by the operating system, and cannot be deleted. Ensure that the "Rule action" field is not set to "deny", or all network connections may be abruptly terminated upon activation of the IPSec device. If items are listed that are not one of the three predefined rules, they should be removed by executing the command rmfilt -n all -v4 .

To load the IPSec extensions and activate the device, use the SMIT fastpath "ips4_start" to get to the appropriate panel, and select the options as detailed in the following table:

Start IP Security

[Now and After Reboot] 

Deny All Non_Secure IP Packets

[no]

After doing the above SMIT action, the activation of the IP Security device can be confirmed by running the command:

lsdev -Cc ipsec

ipsec_v4 Available IP Version 4 Security Extension

Filter rule types

There are three types of filter rules in the AIX implementation of IP Security:

• Predefined filter rules are created by AIX, and cannot be deleted or deactivated, though certain characteristics (such as logging) can be altered. This type of rule includes, among others, the "all traffic" filter.

• Dynamic filter rules are also created by AIX, and are used with IKE tunnels. This type of rule will not be discussed further in this tutorial.

• Static filter rules are created by the system administrator, and are the type to be used for defining access control. Static filter rules will be used in the implementation of the concepts presented in this tutorial.

Listing active filter rules

Each filter rule (predefined or static) contains eighteen fields, or attributes. There are two output formats available when displaying filter rules: a short form with one line for each rule, and a long form, displaying a line (with label) for each option in the rule. To display all of the currently active filter rules in the short form, run the command:

lsfilt -a -v4 -O

1 *** Dynamic filter placement rule for IKE tunnels *** no
2 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all

To retrieve the same information in the more verbose, detailed format, omit the "-O" flag:

lsfilt -a -v4

Beginning of IPv4 filter rules.
Rule 1:
*** Dynamic filter placement rule for IKE tunnels ***
Logging control     : no

Rule 2:
Rule action         : permit
Source Address      : 0.0.0.0
Source Mask         : 0.0.0.0
Destination Address : 0.0.0.0
Destination Mask    : 0.0.0.0
Source Routing      : yes
Protocol            : all
Source Port         : any 0
Destination Port    : any 0
Scope               : both
Direction           : both
Logging control     : no
Fragment control    : all packets
Tunnel ID number    : 0
Interface           : all
Auto-Generated      : no

End of IPv4 filter rules.

Dissecting a filter rule

To understand the different attributes that make up a single filter rule, the filter presented in the previous panel is broken down here:

Rule 2:

Rule action

: permit

Source Address

: 0.0.0.0

Source Mask

: 0.0.0.0

Destination Address

: 0.0.0.0

Destination Mask

: 0.0.0.0

Source Routing

: yes

Protocol

: all

Source Port

: any 0

Destination Port

: any 0

Scope

: both

Direction

: both

Logging control

: no

Fragment control

: all packets

Tunnel ID number

: 0

Interface

: all

Auto-Generated

: no

Defining the rules

Setting up filters

Filters can be defined via the SMIT panel at the fastpath ips4_add_filter or via the command line, using the genfilt command. The SMIT method presents a screen similar to the table in the previous section. To create the filter "by hand", the following flags to the genfilt command are used to specify the attributes of the filter rule:

-v

-n

-a

-s

-m

-d

-M

-g

-c

-o

-p

-O

-P

-r

-w

-l

-f

-i

Defining a filter does not make it active; enabling a filter is a separate step via the SMIT fastpath ips4_upd_filter or with the mkfilt command. To activate the defined filter rule, issue the command:

mkfilt -v4 -u

To deactivate the filter rules (leaving them in the defined state), issue the command:

mkfilt -v4 -d

Example #1: Logging all traffic

As a simple example, the following steps will add a rule to the filter table that does no blocking, but writes a log entry for each incoming IP packet. For this example, the command-line method will be used.

1. Execute the following command to add the rule to the filter table:

   genfilt -v 4 -a P -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 \
           -c all -o any -O any -w I -l Y -i all

   Filter rule 3 for IPv4 has been added successfully.

2. The details of the filter can be checked for accuracy by running the command:

lsfilt -v4 -n3

Rule 3:
Rule action         : permit
Source Address      : 0.0.0.0
Source Mask         : 0.0.0.0
Destination Address : 0.0.0.0
Destination Mask    : 0.0.0.0
Source Routing      : yes
Protocol            : all
Source Port         : any 0
Destination Port    : any 0
Scope               : both
Direction           : inbound
Logging control     : yes
Fragment control    : all packets
Tunnel ID number    : 0
Interface           : all
Auto-Generated      : no

3. Activate the rule by running the command:

mkfilt -v4 -u

Then, activate logging with the command:

mkfilt -v4 -g start

Note that the above rule will generate a tremendous amount of log messages, as each incoming packet will be recorded.

To stop the logging, issue the command :

mkfilt -v4 -g stop

Remove the newly created rule from the filter by running the command:

rmfilt -v4 -n3

Reload the set of active rules by again running the command:

mkfilt -v4 -u

Example #2: Restrict the FTP service to a single host

In this example, the AIX server will be providing file transfer protocol (ftp) service, via the standard ftpd application, but access to that service will be restricted to a single host with the IP address of 172.44.18.74. Attempts to connect to the ftp server from other addresses will be denied as if ftpd was not available. All configuration will be performed through SMIT for this example.

When implementing a policy that provides access to a service by some hosts and denies access to other hosts, a pair of rules must be defined. Connections are checked against the filter rule table until a match is found, so it is important to ensure that the "PERMIT" rule is higher in that table (i.e., has a lower number) than the "DENY" rule.

1. Enter the SMIT fastpath ips4_add_filter to go directly to the Add an IP Security Filter Rule screen, and enter the values listed below in bold text:

Press "Enter" to execute the SMIT action, and after the command completes, press "F3" to return to the previous screen.

2. Select Add an IP Security Filter Rule from the menu, and create the second rule with the values listed below in bold text:

Again press "Enter" to execute the SMIT action, and after the command completes, press F3 to return to the previous screen.

3. Select List IP Security Filter Rules from the menu. The output from the command should appear similar to:

   1 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 no udp eq 4001 eq 4001 both both no all packets 0 all
2 *** Dynamic filter placement rule for IKE tunnels *** no
3 permit 172.44.18.74 255.255.255.0 0.0.0.0 0.0.0.0 yes tcp any 0 eq 21 both inbound yes all packets 0 en0
4 deny 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 eq 21 both inbound no all packets 0 en0
0 permit 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 yes all any 0 any 0 both both no all packets 0 all

4. Press F3 twice to go to the Advanced IP Security Configuration menu and select the menu item Activate/Update/Deactivate IP Security Filter Rule (or exit SMIT and go to the SMIT fastpath ips4_upd_filter). Select Activate / Update from the following menu. There should be nothing displayed on stdout when the command completes.

5. If the logging of successful FTP connections is desired, again press F3 twice, and select from the menu the item Start/Stop IP Security Filter Rule Log, and then Start Filter Logging.

Example #3: Restricting services to the local network

In this example, access to the "exec", "login", and "shell" services will be restricted to the organization's network, which is the Class B network 172.44/16. The following table details the ports and daemons used to provide those services:

In essence, access to the TCP ports 512-514 will be allowed from the local network, but denied to all other addresses.

1. Create the following three filters to permit access to the above services from the network 172.44/16 (meaning, an IP address of 172.44.0.0 with a subnet mask of 255.255.0.0):

   genfilt -v 4 -a P -s 172.44.0.0 -m 255.255.0.0 -d 0.0.0.0 -M 0.0.0.0  \
           -g Y -c tcp -o any -p 0 -O eq -P 512 -r B -w I -l N -f Y -i all

   genfilt -v 4 -a P -s 172.44.0.0 -m 255.255.0.0 -d 0.0.0.0 -M 0.0.0.0  \
           -g Y -c tcp -o any -p 0 -O eq -P 513 -r B -w I -l N -f Y -i all
   
   genfilt -v 4 -a P -s 172.44.0.0 -m 255.255.0.0 -d 0.0.0.0 -M 0.0.0.0  \
           -g Y -c tcp -o any -p 0 -O eq -P 514 -r B -w I -l N -f Y -i all

2. Next, create the following two filters to deny access to the above services from all addresses:

   genfilt -v 4 -a D -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g Y \
           -c tcp -o any -p 0 -O eq -P 512 -r B -w I -l N -f Y -i all
   
   genfilt -v 4 -a D -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g Y \
           -c tcp -o any -p 0 -O eq -P 513 -r B -w I -l N -f Y -i all

   genfilt -v 4 -a D -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g Y \
           -c tcp -o any -p 0 -O eq -P 514 -r B -w I -l N -f Y -i all

3. Activate the rules by running the command:

   mkfilt -v4 -u

Summary

The pros and cons of IPSec packet filtering

Packet filtering via the IP Security features of AIX gives system administrators a powerful and flexible tool for controlling the network services provided by their RS/6000 systems. Its inclusion and integration into the base operating system helps make deployment and configuration straightforward. By inserting the decision of allowing or denying connections at the IP level, access control can be applied to any network application, without modification, and a system offering a large number of services can have its public "footprint" dramatically reduced.

There are several "cons" to IPSec packet filtering, though. A slight, but measurable reduction in performance may be observed, depending on the speed of the network interface, the complexity of the filter rule table, and the amount of network traffic to and from the system. For larger networks, it might be easier to implement a consistent policy using a dedicated firewall or router with packet filtering capabilities. Unless the RS/6000 is itself serving as a router, it is not possible to use IPSec packet filtering to redirect connections based upon their attributes; this function is better handled by dedicated hardware and/or software.

Further topics for exploration

The tutorial presented above explores only one feature available through the IPSec tools provided with AIX. IP Security encompasses much more than packet filtering: Virtual Private Networks (VPN) and tunnels can provide end-to-end encryption of all communications between individual hosts or entire networks.

Filtering of network traffic is a useful, but small part of system security; its implementation should be seen as one part a comprehensive security policy. It is important to address all aspects of system security from both remote and local connections. A single weakness in one part of infrastructure could lead to a compromise of the organization's entire network.

References and resources

The definitive documentation for the AIX IP Security features is the System Management Guide: Communications and Networks, Chapter 4, available online at <http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixbman/commadmn/ch4_ip_security.htm#A14endr>.

Additional information about the IPSec standard, including mailing lists, Internet-Drafts, and Requests for Comments documents, can be found at the the home page of the Internet Engineering Task Force (IETF) IP Security working group: <http://www.ietf.org/html.charters/ipsec-charter.html>.

Packet filtering implementations are available for a variety of UNIX and other operating systems. IP Filter (ipf) is a popular package, available for Solaris, FreeBSD, and other UNIX systems. With Linux, packet filter functionality is built into the kernel, and is configured using the iptables tool.

IBM offers a number of Redbooks on the subject of networking with AIX. Of particular interest are the publications:

• TCP/IP Tutorial and Technical Overview

• A Comprehensive Guide to Virtual Private Networks, Volume I: IBM Firewall, Server and Client Solutions

Building Internet Firewalls, 2nd Edition, published by O'Reilly, is an excellent guide to the workings of firewalls and packet filters, and contains a wealth of information for security conscious administrators.

Securing AIX Network Services is another tutorial available from the IBM eServer Developer Domain. This tutorial provides an understanding of the network service provided in AIX, and the security implications of each service.

About the author

Sandor W. Sklar is a Unix Systems Administrator at Stanford University, in Northern California. The author invites questions and feedback about this tutorial at ssklar@stanford.edu.