Nmap Hackers,
I'm excited about all of the Google SummerOfCode projects reported
earlier, but am particularly interested in the OS Detection redesign!
I have been working on that with Zhao Lei for weeks. We have been
reading RFCs, casting experimental packets in all directions, reading
related research, etc. His latest draft proposal is below, and
implementation work is just beginning. We are adding quite a few
useful tests, packed into as few probes as possible. But maybe some
of you have other clever ideas? Remember that changing the system
requires (for best results) creating a new DB from scratch, so this
won't be done often! The current OS detection scheme has served us
well for 7 years, and I'd like the new one to last at least as long!
So now is your chance to influence the new system. If you have any
ideas, please post 'em to nmap-dev_at_insecure.org (or send them to me
directly, if you prefer that). Join that list (or read it on
seclists.org) if you wish to participate in ensuing discussion. Of
course we can't add every possible. Here are the "ground rules" for
evaluating tests that we used for the new design:
o The tests must not require Nmap to mess with the host firewall. So,
for example, SYN/ACK retransmission timing is out because Nmap would
need to block the host from receiving the SYN/ACKs and RST'ing them.
o The tests must be quick. RTT timeout is also out because waiting a
minute or two for RTT counting is unacceptable. There is no absolute
cut-off time though -- the amount of time a test takes is
considered against the value of information it provides. Ideal tests
simply require one packet which generates an immediate response. Even
better are tests that can be piggy-backed on an existing test by
modifying the probe or analyzing the response more carefully.
o I'm trying to avoid structurally bogus/corrupt packets. Packets
which blatantly violate the RFCs are more likely to be
dropped/scrubbed on the network, crash the target machine, be
reliably detected by IDS systems, etc.
o I'm trying to avoid sending any fragmented IP packets, as they are
problematic to send and are frequently defragmented en route.
o Tests should require as few new probe packets as possible. A
congestion control test that requires hundreds of sent packets would
be unacceptable.
o I'm trying to avoid making full TCP connections, although I won't
rule that out.
o The new tests must actually work :). Testing an attribute that all
known hosts handle the same way doesn't do much good.
Note that these rules are a gray area. I'm willing to consider
anything if the information gathered is substantial enough.
With that out of the way, I'm including Zhao's latest draft design
below. The grammar is imperfect because I asked him to write it in
English rather than his native Chinese. But the ideas are good! Some
tests are mentioned here but not described in detail -- you can get
more background information on them from
http://www.insecure.org/nmap/nmap-fingerprinting-article.html and
http://seclists.org/lists/nmap-dev/2005/Jul-Sep/0039.html .
PROPOSED NEW NMAP OS DETECTION MECHANISM
by Zhao Lei (zhaolei(a)gmail.com) and Fyodor (fyodor(a)insecure.org)
######################
# Modifications of tests #T2-T7
######################
## Description
o Original test1 is combined with Tseq probes.
o I'd like to use Tseq probes to do option-combination tests,
because we can expect the target really handles and responses to
each combination, not simply return a, say, RST with no options.
** The rest of this section is not changed **
o The modified 6 (test2 - test7) probes of current nmap, mainly
adding the SACK option and setting the Acknowledgement value (to
test seq value in RST+ packets).
* I currently set the ACK value in all of the probe packets. I
checked RFC793 and didn't find there are any requirements for
when ACK shouldn't be set. This seq value test works! I found
one "3Com SuperStack II switch (OS v 2.0), AsanteHub 2072
Ethernet hub" (IP 59.66.79.3-13) set SEQ = Probe's ACK in T5,T7
while others not.
o Add and change some of the tests on the responses.
## Probe Packets Specification
# Test2 - A NULL with the same options to an open port
IP DF flag: set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: none
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Test3 - A SYN|FIN|URG|PSH with the same options to an open port
IP DF flag: not set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN|FIN|URG|PSH
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Test4 - An ACK to an open port
IP DF flag: set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: ACK
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Test5 - An SYN to a closed port
IP DF flag: not set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Test6 - An ACK to a closed port
IP DF flag: set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Test7 - An FIN|PSH|URG to a closed port
IP DF flag: not set
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: FIN|PSH|URG
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
## Tests Specification
** Changed: add TTL check, add A++ in Seq check, use checksum for Rst+
Payload **
Check the following stuffs:
DF, TCP Window size, Seq value, Ack value, Flags, Options, Quirks, RSTData
Different with the current nmap:
o Add seq value check, quirks check and RST payload check.
o Modify the options check.
# The Seq value (Seq=xx)
We expect the sequence number in the response would be one of the
following 3 different patterns:
o Z. It is zero;
o A. It is the same with the ack we sent;
o A++. It is ACK+1;
o O. Other situations.
# The options (Ops=xx)
o SACK-permitted. Denoted as 'S'.
o WScale value. I think we can simply append the value to 'W' when
compare. For example, "W3".
o Timestamp value. Check whether the TSval is zero and whether the
TSecr is zero. I think we can denote this as "Txx" (x=0|1).
# The Quirks (Qrk=xx)
We can denote the quirks as the following:
o R. The "reserved" field of TCP is not zero;
o U. URG pointer value when urg flag not set.
# The Payload (RCK=xx)
If it is RST+ packet and has payload, we count the checksum of the
data and compare the checksum. When there is no data, we use
RCK=0000.
* Don't do this to SYN+ACK packets, because the data in this kind of
packets may be service related.
## Fingerprint example
DF, TCP Window size, Seq value, Ack value, Flags, Options, Quirks,
RSTData CheckSum
T2(Resp=N)
T3(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=&RCK=0000)
T4(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
T5(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
T6(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
T7(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
################
# T1 & TSeq test
################
## Description
o Original 6 TSeq probes, but with different options. The time
intervals are the same with the original.
o Use the first probe as if it is T1, and if there is no reply,
resend the probe after the other TSeq tests are done.
o Now we have the left 5 probes to do option-combination probes. If
some don't response, give them flags and retry after TSeq.
## Probe Packets Specification
# Packet 1
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001
o Window Scale (\012)
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
# Packet 2
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \002\004\001\011\003\003\000\001\004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001
o MSS (\001\011)
o Window Scale (\000)
o SACK permitted
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
# Packet 3
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001\003\003\005\001\002\004\001\011
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
o Window Scale (\005)
o MSS (\001\011)
# Packet 4
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001\003\003\012\001\002\004\001\011
o SACK permitted
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o Window Scale (\012)
o MSS (\001\011)
# Packet 5
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \002\004\001\011\004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001\003\003\012\001
o MSS (\001\011)
o SACK permitted
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o Window Scale (\012)
# Packet 6
TCP Sequence: Random1
TCP Acknowledgement: Random2
TCP Flags: SYN
TCP Option Length: 24
TCP Options: \002\004\001\011\010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001\003\003\012\000
o MSS (\001\011)
o Timestamps (TSval: \077\077\077\077; TSecr: 0)
o SACK permitted
o Window Scale (\012)
o End of List
## Tests Specification
# T1
Do t1 test as T2-T7
# TSeq
Do Seq test as the current nmap does.
# TOps
Get each option and compare.
## Fingerprint example
TSeq(Class=RI%gcd=2%SI=39623%IPID=Z%TS=100HZ)
TOps(O1=W10MT10S%O2=MST11W0=MST11NW0%O4=MST11NW0%O5=MST11NW0)
T1(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=%RCK=0000)
######################
# Two ICMP echo probes
######################
## Description
Two tasks for the probes:
o To cause two IPids reply in ICMP protocol to test if the way is per
<src, des, protocol> triple IPid couter.
o To test if the target echo the ICMP code field back and if any of
Identifier and Sequence Number is zero.
Why two probes?
o We can get one more IPid;
o To confirm some fields are really *echoed* other than set to that
value (eg. DF, TOS)
## Probes Specification
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type | Code | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identifier | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data ...
+-+-+-+-+-
# ICMP probe 1
IP DF: 1
IP TOS: 0
ICMP type: 8
ICMP code: \011
Identifier: random
Sequence Number: \011\021
DataLen: 120 bytes
# ICMP probe 2
IP DF: 0
IP TOS: \004 (IP_TOS_RELIABILITY)
ICMP type: 8
ICMP code: 0
Identifier: the identifier in probe1 + 1
Sequence Number: \011\022
DataLen: 150 bytes
## Tests Specification
# Overview
We have two icmp echo test and expect two icmp echo replies.
Though it is natural to store one fingerprint for each reply (see
below) and compare separately,
TI1(Resp=Y%DF=Y%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
TI2(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
I prefer to combine the two into one single line:
TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
because:
o this will save much place for the db file
o It is possible that the host doesn't reply to icmp echo
messages at all, but it is very rare that the host responds to only
one of them.
o there are not much information dropped during the
combination. Also what dropped might be very rare. (for example, two
different non-zero TOS? two different non-zero Code?)
* I ever thought to check if Icmp ID is zero, but I changed my
mind. It is much easier if we use id to identify our replies and
indeed I didn't see an implcation doesn't reply the ID field.
# Response
1) Two replies: Resp=Y;
2) If we don't get a reply: Resp=N;
3) If we get only one reply, resend the other probe (say, one more
time). Still no reply? Look upon as no reply: Resp=N.
# Test1: DF (DF=xx)
* In the followin text, I'll use R1_xxx and R2_xxx to denote the
value of xxx field of in the two reply messages.
Patterns of DF:
o N. Both R1_DF and R2_DF are not set;
o S. Both use the DF value that the sender uses;
o Y. Both set;
o O. Other.
# Test2: TTL (TTL=xx)
Count initial ttl based on R1_TTL. (The possibility for these two
icmp echo reply have different ttl is very low)
# Test3: TOS (TOS=xx)
Patterns of TOS:
o [Value]. Both set TOS to the same value [Value] (this includes
that both set to zero);
o S. Both use the TOS that the sender uses;
o O. Other.
# Test4: Code (CD=xx)
Patterns of Code:
o [Value]. Both set Code to the same value [Value];
o S. Both use the Code that the sender uses;
o O. Other.
# Test5: Seq (Seq=xx)
Patterns of Seq:
o [Value]. Both set Seq to the same value [Value];
o S. Both use the Seq value that the sender uses;
o O. Other.
# Test6: DataLen (DL=xx)
Pattens of DataLen:
o [Value]. Both truncted to a specific value;
o S. Both the same with the sender;
o O. Other.
## Fingerprint Example
TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
#######################################
# One TCP ECN-setup SYN to an open port
#######################################
## Description
Purpose:
o ECN support test;
o Test if the target echo Urg pointer value with urg flag not set;
o Test if the target echo reserved field.
## Probe Packet Specification
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
| | | C | E | U | A | P | R | S | F |
| Header Length | Reserved | W | C | R | C | S | S | Y | I |
| | | R | E | G | K | H | T | N | N |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
TCP Sequence: Random
TCP Acknowledgement: 0
Reserved(0): 1
TCP Flags: CWR ECE SYN (* we look upon CWR&ECE as flags here)
TCP Options: none
## Tests Specification
# Response
1) one reply. Resp=Y;
2) If we get RST, retry for a second time. Still nothing? then
Resp=N.
3) no reply. Resp=N.
# ECN support test (ECN=xx)
Check if ECE is set, CWR is set, and compare the result to the
following 3 patterns:
1) Y: Only ECE is set. Receiver that supports ECN and replys
correctly;
2) N: Neither is set. Receiver that doesn't support ECN and sets the
"reserved" bit to zero;
3) S: Both are set. Receiver that doesn't support ECN but echoes the
"reserved" bit back;
# Reserved field echo test (RE=xx)
Denote as RE = Y, N
# URG pointer value when URG flag not set test (UP=xx)
Denote as UP = Y, N
## Fingerprint example:
TE(Resp=Y%TTL=64%ECN=Y%RE=N%UP=N)
###############################
# UDP probe (probe not changed)
###############################
The probe is not changed. But we'll get the IPid from the expected
ICMP packet and test if the "unused" field is not zero. (UN = Y, N)
PU(DF=N%TOS=0%IPLEN=38%UN=Y%RIPTL=148%RID=E%RIPCK=E%UCK=F%ULEN=134%DAT=E)
###########################
# Final Fingerprint Example
###########################
TSeq(Class=RI%gcd=2%SI=39623%IPID=Z%TS=100HZ)
TOps(O1=W10MT10S%O2=MST11W0=MST11NW0%O4=MST11NW0%O5=MST11NW0)
T1(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=%RCK=0000)
T2(Resp=N)
T3(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=&RCK=0000)
T4(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
T5(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
T6(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
T7(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
PU(DF=N%TOS=0%IPLEN=38%UN=Y%RIPTL=148%RID=E%RIPCK=E%UCK=F%ULEN=134%DAT=E)
* Some label in PU may be shortened.
_______________________________________________
Sent through the nmap-hackers mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-hackers
Received on Aug 21 2005
Intro
