Design Considerations

Design Considerations

We created Libsf to bridge the gap that existed between commonly available tools (such as Fyodor's active fingerprinting facility in nmap and Michal Zalewski's passive fingerprinting tool p0f) and the need for remote OS detection to be built inside additional arbitrary network security tools. While these two tools work well, they are standalone and do not lend themselves very well toward integration into new frameworks. Previous to libsf, there was no way for an application programmer to seamlessly include remote OS detection in his or her programs. The only possible way to get this functionality required serious hacks of existing tools. Libsf offers the application programmer simple and portable primitives to perform remote OS detection. One major advantage of libsf is that its functionality is based upon popular and well-tested tools; the active methods are based on nmap while the passive methods are based on p0f—two tools with proven track records.

Future plans for the library include combining the two separate databases into a single MySQL-based database.

Active Fingerprinting Methods

Libsf's active fingerprinting functionality consists of generating certain TCP packets, waiting for responses, and then analyzing and correlating these responses with its database. These packets transmit to a TCP port with a daemon listening, to an "open" port, and to an inactive or "closed" TCP port. As such, before it can start fingerprint scanning, libsf needs to find an open TCP port and a closed TCP port on the target host. Depending upon initialization parameters, libsf tries to either connect to "likely" open ports such as 80,139, 22, 25, 53, 113, 443, or 6667 or kick off a sequential port scan to determine port status.

At this writing, libsf supports seven different active TCP fingerprint tests. Although each is slightly different, they all use the same general packet template shown in Figure 5.1.

Figure 5.1: Libsf TCP packet with options shown exploded.

Each of the seven tests contains the same TCP options string as shown earlier; we describe the specifics of each test as follows.

Active Test 1: TCP SYN Packet to an Open Port

A TCP packet with the SYN (synchronize sequence numbers) flag is set to an open TCP port on the remote host. SYN packets initialize a TCP session and are ubiquitous across the Internet. This test should almost always succeed.

Active Test 2: NULL Packet to an Open Port

In this test, a TCP packet with no flags is set to an open TCP port on the remote host. In normal operation, you never see a packet with no control flags.

Active Test 3: TCP FIN SYN PSH URG Packet to an Open Port

In this test, a TCP packet with the FIN (finished sending data), SYN, PSH (push data to the application layer), and URG (urgent data present) flags are set to an open TCP port on the remote host. In normal operation, you never see a packet with these controls flags.

Active Test 4: TCP ACK Packet to an Open Port

In this test, a TCP packet with the ACK (acknowledgment) flag is set to an open TCP port on the remote host. The ACK packet, used to acknowledge data, is another standard TCP packet that is often encountered.

Active Test 5: TCP 5YN Packet to a Closed Port

In this test, a TCP packet with the SYN flag is set to a closed TCP port on the remote host. This situation is standard and should result in an RST packet sent back to the original host.

Active Test 6: TCP ACK Packet to a Closed Port

In this test, a TCP packet with the ACK flag is set to a closed TCP port on the remote host. This situation is nonstandard and results in undefined behavior.

Active Test 7: TCP FIN\PSH\URC Packet to a Closed Port

In this test, a TCP packet with the FIN, PSH, and URG flags is set to a closed TCP port on the remote host. This situation is also nonstandard and results in undefined behavior.

Passive Fingerprinting Methods

One problem with active fingerprinting is that it is noisy. You can easily detect the initial port scan or TCP connections and subsequent seven test packets sent out to elicit information from the remote machine. Passive fingerprinting solves this problem by eliminating any packet transmission from the process. In order to gather information, libsf's passive fingerprinting module waits for TCP SYN packets from remote hosts to pass on the network segment and uses information in the packets to differentiate between various operating systems.

At this writing, libsf does not have passive fingerprinting functionality fully working, but it does have the hooks to support eight different passive IP and TCP fingerprint tests based on received SYN and SYN|ACK packets.

Passive Test 1: IP Time to Live

Determine the original IP time to live (TTL) field of the packet as it was sent from the target host. The original value of this header field requires a bit of guesswork to calculate because it decrements by intermediate routers as it travels across the Internet. Generally speaking, increasing the TTL to the next power of two is usually pretty accurate for most remote hosts. For example, a TTL of 17 (2⁴ + 1) would become 32 (2⁵). The expected error between the TTL estimate and the true TTL increases as the distance between the fingerprinting system and the target increases. A traceroute to the host would give the proper number of hops to determine the original TTL but would add over-the-top complexity to this portion of the scan (in addition to making it active instead of passive).

Passive Test 2: IP Packet Size

Determine the size of the packet as reported by the IP header.

Passive Test 3: IP Don't Fragment Bit

Determine whether the remote host set the IP don't fragment (DF) bit.

Passive Test 4: TCP Window Scale Option

Determine whether the remote host set the TCP window scale option.

Passive Test 5: TCP Maximum Segment Size Option

Determine whether the remote host set the TCP maximum segment size option.

Passive Test 6: TCP Selective Acknowledgment Flag Option

Determine whether the remote host set the TCP selective acknowledgment (SACK) option.

Passive Test 7: TCP No-Operation Flag Option

Determin whether the remote host set the TCP no-operation (NOP) option.

Passive Test 8: TCP Window Size

Determine what the remote host set the TCP window size to be.

Database

The database that libsf employs is a precompiled list of operating systems and their responses to the fingerprint tests. Libsf correlates and matches information that it finds through collection techniques against information in the database. For each individual test result that matches a signature in the database, libsf increments a score. At the conclusion of testing, the OS with the highest score represents libsf's best guess.

One major difference that exists between libsf and other existing tools is that rather than implementing an inefficient flat-file database, libsf uses Berkeley libdb for a more efficient model. The fingerprint database is a sorted b-tree (balanced tree) accessed with a much faster seek time than the traditional flat-file formats. As previously mentioned, the database is built and installed at compile time so that it is available to the user of stack fingerprinting applications.