Dig pcap File For Fun and Productivity

Having worked with networking gear for many years I thought it was about time to jump in and post something to our blog, and why not start by talking about pcap files. As most of you already know, when testing and providing support of networking products, it is common that you will get a big pcap file. Often the file can be so big that it is at best slow when opening in Wireshark, or at worst it may be impossible. Make no mistake, I am a big fan of Wireshark and can not remember a day here on the job where I didn't use this wonderful tool. But the question is, how do you complete tasks such as "grab some TCP sessions where there is no data from server" if opening a 200MB pcap file crashes Wireshark every time?

No worries, programming to the rescue! 

To solve the problem I used Perl (feel free to use your favorite language) to open a pcap file and do some analysis. Let us look at finding sessions where the client sent data but the server didn't send any data in response. To make it easy I've included all the steps I took and, where appropriate, the code.  Since the point is to illustrate how to use script language like Perl to do the job, the code is greatly simplified.   For the convenience of reader,  the complete code is listed at the end.

Step 1. Open the pcap file and put it in binary mode: 

$inputFile = $ARGV[0]; 
#the first command line parameter is the name of the pcap file
open(FD, "<$inputFile") || die "failed to open $inputFile $!\n";
A pcap file is a binary file, so open it in binary mode
binmode(FD); 

Step 2a.  Knowing the structure of the pcap file is helpful here: pcap files typically start with 24 byte file header followed by a sequence of packets. The most important thing from the file header is to read the first 4 bytes to find the endianess.

read(FD, $fileHdr, 24); #skip the 24 byte pcap file header
   checkFileHdr($fileHdr); #the routine is defined later, it checks the file header to find endianness  

Step 2b. Each packet consists of 16 bytes of packet header (timestamp, length etc. Be sure not to mistake it with the protocol header) plus packet data. For details of pcap file format, you can read, for example, pcap file format . We process them by first reading the 16 bytes of header and then read the packat data. Note how endianness plays a role here.

while (!eof(FD))
 {
   read(FD, $pktHdr, 16);
   my ($pktSecond,$pktMicroSecond,$capturedPktLen,$actualPktLen) 
   = unpack($pktHdrFormat, $pktHdr); 
    #endianness determines how time and packet length are stored.
     if (read(FD, $pktBuf, $capturedPktLen) != $capturedPktLen ) 
     { print "Failed to read pkt data\n"; last;}
     $proto = unpack("C", substr($pktBuf, 23,1)); 
     #this is the byte in ip hdr for protocol type, UDP is 17, TCP is 6
     if ($proto == 6) #we are only interested in TCP packet for this task
                     {
            processTCPPkt(); #we will explain this function next 
                     }
              } 

Step 3.  In function "processTCPPkt", we need to decide the offset of various fields, such as IP header protocol field, source IP address, destination IP address, source TCP port, destination TCP port and more. How? Once again I rely on Wireshark (with a small pcap file of course). It clearly shows, for example, source IP address is at offset 26 and source TCP port is at offset 34 (assume the packet doesn't have VLAN tags or ip option fields).This function will look at each tcp packet, if it's a TCP SYN packet, we set up a hash for the TCP session using src IP and port, dst IP and port. If it's a data packet, we want to find whether it's from a client or a server by using the hash, we also make sure the TCP session state keeps track of whether there is client data or server data

              
sub processTCPPkt
  {
      my $srcIp    = substr($pktBuf, 26,4); 
      my $dstIp    = substr($pktBuf, 30,4); 
      my $srcPort  = substr($pktBuf, 34,2);
      my $dstPort  = substr($pktBuf, 36,2);
      my $tcpFlags = unpack("C", substr($pktBuf, 47,1));
      if ($tcpFlags == 2) #2 means TCP SYN
      {
           $hashKey = "$srcIp$srcPort$dstIp$dstPort";
           if (! defined $tcpSessionState{$hashKey})
             {
                   $tcpSessionState{$hashKey} = 0; 
                   #note that TCP session is only hashed from client side to server side, 
                    i.e. $srcIp is TCP client sip
                }
      } elsif ($tcpFlags == 0x18) #TCP PSH ACK pkt, a.k.a data packet
        {
          $hashKey = "$srcIp$srcPort$dstIp$dstPort";
           if (defined $tcpSessionState{$hashKey}) 
           #this pkt is from client, see the above for tcp session hash set up
          {
                  $tcpSessionState{$hashKey} |= 1;  
                  #so if the session has client data, the least significent bit will be 1
             } else
             {
                 $hashKey = "$dstIp$dstPort$srcIp$srcPort";
                  if (defined $tcpSessionState{$hashKey}) 
                  #this pkt is from server
                   {
                       $tcpSessionState{$hashKey} |= 2;  
                       #if the session has server data, the least significient bit will be 1
                        } 

                 }

           }
              }

Step 4.  After we go through all the packets, we go through all the TCP sessions and print the sessions that have no server data.

    foreach $tcpSession (keys %tcpSessionState)
              {
                     if (($tcpSessionState{$tcpSession} & 2) == 0)
                     {
                            printSession($tcpSession);
                     }
              }

We save it to a file, say,  “findBadSessions.pl”.  Here is part of the sample output  for the command:

              perl findBadSessions.pl bigPcap.pcap
              1.1.17.167:7697 --> 1.2.167.89:80

              1.1.172.218:7840 --> 1.2.26.37:80

              1.1.121.19:7698 --> 1.2.122.238:80

              1.1.127.196:7652 --> 1.2.129.100:80

              1.1.172.131:7532 --> 1.2.174.40:80

You can then grab a TCP sessions using tcpdump or windump as: windump -r "bigPcap.pcap"  -w output.pcap host 1.1.17.167 and tcp port 7697 and then your output.pcap will contain the TCP session, and it is small enough to be opened by Wireshark.

Since the code is arranged to make it easy to read you may want to format it in your favorite coding style and add more error check if you see fit. Whatever you do let me know in the comments section. You may also be wondering about the speed of using Perl in processing a big pcap file. Yes, a program written in C is faster, but Perl is also fast. On my Windows XP (3.1GH, duo core), I ran this Perl program on a 234MB pcap file with 13655 TCP sessions and it took about 2 seconds.

 With this method we can also do the following:

• Find which TCP session has retransmissions from a big pcap file
• Open two big pcap files and find out which packets that are present in the first pcap file but not in the second one. This is useful in determining what packets ard dropped by a device under test.
• Determine the average latency between HTTP request and HTTP response.
• Honestly, using this method the options are limitless.

One of the better side benefits of completing this task is when you hear compliments from your colleagues in the form of the question, "How did you find the needle in the haystack?".

For the sake of completeness, here is the entire Perl script:

my $pktHdrFormat;#depending the endianess of file hdr, the 16 byte pkt hdr should be read accordingly 
my %tcpSessionState; #this is used to keep track of all the TCP sessions.
$inputFile = $ARGV[0]; #the first command line parameter is the name of the pcap file
open(FD, "<$inputFile") || die "failed to open $inputFile $!\n";
binmode(FD); #pcap file is a binary file, so open it in binary mode
read(FD, $fileHdr, 24); #skip the 24 byte pcap file header
checkFileHdr($fileHdr); #check the file header to find endianness
#now process each packets
my $pktSecond;
my $pktMicroSecond;
my $capturedPktLen;
my $actualPktLen;
while (!eof(FD))
{
       read(FD, $pktHdr, 16);
       my ($pktSecond,$pktMicroSecond,$capturedPktLen,$actualPktLen) = unpack($pktHdrFormat, $pktHdr);
       #print "$pktSecond,$pktMicroSecond,$capturedPktLen,$actualPktLen\n";
       if (read(FD, $pktBuf, $capturedPktLen) != $capturedPktLen ) { print "Failed to read pkt data\n"; last;}
       $proto = unpack("C", substr($pktBuf, 23,1)); 
       #this is the byte in ip hdr for protocol type, UDP is 17, TCP is 6
       if ($proto == 6) #we are only interested in TCP packet for this task
                     {
                           processTCPPkt();
                     }
              }
foreach $tcpSession (keys %tcpSessionState)
{
                     if (($tcpSessionState{$tcpSession} & 2) == 0)
                     {
                            printSession($tcpSession);
                     }
              }
#this function will look at tcp packet, if it's a TCP SYN, we set up a hash for the TCP session 
#using src IP and port, dst IP and port. If it's data packet, we want to find whether it's from 
#client or server by using hash. For data packet from client, we make sure the tcp session state 
#keep tracks of whether there is client data or server data
sub processTCPPkt
{
           my $srcIp    = substr($pktBuf, 26,4); 
           my $dstIp    = substr($pktBuf, 30,4); 
           my $srcPort  = substr($pktBuf, 34,2);
           my $dstPort  = substr($pktBuf, 36,2);
           my $tcpFlags = unpack("C", substr($pktBuf, 47,1));
           if ($tcpFlags == 2) #2 means TCP SYN
           {
                  $hashKey = "$srcIp$srcPort$dstIp$dstPort";
                  if (! defined $tcpSessionState{$hashKey})
                   {
                         #print "SYN\n";
                         $tcpSessionState{$hashKey} = 0; 
         #note that TCP session is only hashed from client side to server side, 
            i.e.  $srcIp is TCP client sip
                   }
            } elsif ($tcpFlags == 0x18) #TCP PSH ACK pkt, a.k.a data packet
            {
                   $hashKey = "$srcIp$srcPort$dstIp$dstPort";
                   if (defined $tcpSessionState{$hashKey}) 
         #this pkt is from client, see the above for tcp session hash set up
                  {
                         $tcpSessionState{$hashKey} |= 1;  
          #so if the session has client data, the least significent bit will be 1
                  } else
                   {
                         $hashKey = "$dstIp$dstPort$srcIp$srcPort";
                         if (defined $tcpSessionState{$hashKey}) 
        #this pkt is from server
                           {  $tcpSessionState{$hashKey} |= 2; 
      #if the session has server data, the least significient bit will be 1
                                 } 
                                 
                            }
                     }
              }
sub checkFileHdr
{
            my $fHdr = shift;
           my $signature = unpack("N", substr($fHdr,0,4));
           if ($signature == 0xa1b2c3d4)
                     {
                   $pktHdrFormat = "NNNN"
             } elsif ($signature == 0xd4c3b2a1)
                     {
                  $pktHdrFormat = "VVVV";
            } else 
            {
die "unexpected signature bytes";
             }
         }
sub printSession
{
       my $session = shift;
       my @B = unpack("C*", $session);
       printf "%d.%d.%d.%d:%d --> %d.%d.%d.%d:%d\n", 
       $B[0], $B[1],$B[2],$B[3], $B[4]*256+$B[5], $B[6],$B[7],$B[8],$B[9], $B[10]*256+$B[11];
         }

Testing and Validation of Network Security Devices

While catching up on security news and blogs the other day, I came across a blog post from ICSA Labs entitled "Why a Test Lab Needs to be Wary of Commercial Exploit Packet Captures" and thought that it would be a good conversation starter to inform our readers about how BreakingPoint approaches developing test cases for security device testing, our methodology behind why we develop our test cases the way we do, and the thought processes and conclusions behind those decisions.

First, it's important to note that ICSA's blog post is primarily talking about test tools that replay packet captures as their security tests. While the BreakingPoint devices do provide a packet capture replay component, this component is not what we use for security testing. The BreakingPoint devices provide a dedicated security component that execute packaged attacks targeting individual vulnerabilities that we call "strikes". Strikes are not packet captures, and we'll discuss how strikes operate and the benefits derived from them a little later in this post.

Toward the beginning of their blog post, ICSA wrote the following:

"If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy."

This is also one of the primary reasons that we do not use packet captures of attack traffic that we have come across in our research. However, we take it one step further and and don't even use packet captures created in-house. We simply don't use packet captures for security testing at all, which brings me to the first subject I'd like to discuss:

Attack Realism

Let's look at what ICSA has to say on attack realism using third-party packet captures:

"ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control."

The BreakingPoint Labs team builds each strike by hand after performing our own analysis of the vulnerability. We have a high degree of certainty that our attacks are correct because we do this analysis and then we test the strikes afterward when possible against the actual vulnerable target. Then, we use these strikes (not packet captures of them) in testing performed using the BreakingPoint device. There are currently two ways to test using these strikes; passing attack traffic through an intermediary Device Under Test (DUT), and sending attack traffic directly to an endpoint DUT, which I'll cover next.

Attack Simulation

"But what happens if the vendor's IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line."

This is a valid concern when testing an intermediary DUT, and even more so when you're using static data from packet captures. In this scenario, our strikes act as both the attacker and target, and send the attack traffic from one port on our device, through the DUT, and back to a second port. In this way, it's really an attack "simulation" using real attack traffic because we're essentially sending traffic back to ourselves rather than a real target. Because we know what valid attack traffic looks like for each individual iteration of the strike, we know what data we're sending, and we know what the data should look like when we (the target) receive it, if the DUT modifies the attack traffic in transit we consider the attack blocked as it is no longer the attack traffic that we sent and is invalid.

"One-Arm" Strikes

"If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible!"

Once again, we're in total agreement here, which is why we use real attacks. To the extent possible, strikes that target servers can be run in "one-arm" mode where rather than passing attack traffic through a DUT and back to ourselves, the traffic is sent to the DUT as the attack's target server. In this mode, strikes can be used to actually trigger vulnerabilities on actual vulnerable systems. This is what test houses that use BreakingPoint devices like NSS do to verify that the test cases they are using are indeed valid, even though they are provided by BreakingPoint, their vendor.

Custom Strikes

What if BreakingPoint doesn't have a strike for the vulnerability you want to test? Or what if, like ICSA, you don't trust third party content at all? Even though BreakingPoint provides you with real attacks packaged as strikes, users can easily develop their own strikes. I won't cover this topic in any detail here, as we've already had a three part series (1, 2, 3) on this subject posted to the blog.

Strike Development Goals

1. Trigger Just the Vulnerability & Use Unidentifiable Payloads

One of the most frequently raised concerns about our strikes is that they contain no active payloads or executable shellcode. This is by design. Sure, network security devices often have filters for well-known shellcode and common payload encoders, and we have specific strike categories to test those specific cases, however if you are relying on the detection of such by your IPS to protect you from actual vulnerabilities then you have already failed. Most network security devices are reactive in nature, and in order to detect a particular shellcode or payload encoder, it must first be aware of it and/or have a filter for it. We know there are payload encoders and shellcode out there that devices are unaware of, so we simulate this by using completely random data as our payloads. This forces the DUT to identify attacks based on the properties of the vulnerability, not by relying on detecting known shellcode or a decoder stub from an encoded payload. We focus entirely on triggering the vulnerability, not actually exploiting it with an operational payload.

2. Randomness & Uniqueness on the Wire

"ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system."

The problem with ICSA's approach here is that you're initially still testing with static packet captures. Consider the scenario where you replay your packet capture of a malicious TIFF file traversing the wire. The IPS under test blocks it, and you mark that as a success. How do you know that if some unrelated parts of the TIFF file are modified, that the IPS won't miss it? How do you know that if you add a whole lot of padding or superfluous structure to the file and move the evil from the beginning of the file to beyond the padding, that the IPS won't miss it? If you're initially relying on packet captures of static attack traffic and then only breaking out the real exploits and targets when something seems amiss or a customer questions your tests, you're not being thorough in your testing.

BreakingPoint's approach to providing these various attack permutations is to identify all of the components of the attack that are absolutely essential for the attack to work and trigger the vulnerability. We identify these values and their upper and lower bound thresholds as well as identify behavioral protocol and process interactions and what combination and permutations of these are valid. We then develop our strikes to randomize these properties as much as possible while still conforming to the identified valid parameters. Further, we randomize as much other data as possible that is not directly related to triggering the vulnerability while still remaining valid for whatever protocol, file format, or other data structure is being used in the attack. All of this context information and the flexibility provided by dynamic test cases such as strikes as opposed to packet captures is the benefit we get from performing the vulnerability analysis ourselves, understanding the operational bounds of the data involved, and developing strikes that launch attacks that actually utilize that knowledge. You can read more about this subject in one of my previous blog posts, File Format Vulnerabilities and Dynamic Exploit Generators.

3. Evasions

To further the previous point, BreakingPoint can optionally also mutate attack traffic by employing various evasion techniques. When you combine evasion techniques such as IP fragmentation with fragment reordering, using various text encoding methods, and HTTP chunked encoding transmission, among others, with the randomization of the attack traffic that we are already performing as outlined in the previous section, nearly endless permutations of a single attack are dynamically generated which using static packet captures simply can't compete with. Forgive me for quoting a deodorant commercial, but "anything less would be uncivilized." For much more in-depth information on the subject of evasions, please see our recent webcast entitled Harden Security Devices Against Increasingly Sophisticated Evasions or this previous blog post on the subject.

Conclusion

I hope you enjoyed this look into the BreakingPoint strike development and security device testing mindset and found the information both useful and enlightening. Please do follow some of the links above as there is much more information available about the topics discussed.

Tech Talk

Tech Talk

TCP Portals: The Handshake's a Lie!

Whenever I interview someone for an Application Engineer or Security Research position, my favorite introductory question is, "Can you describe for me the TCP three-way handshake?". It is a fine baseline question to understand a candidate's knowledge of modern networking. Answers range from "SYN, SYN/ACK, ACK,", to a full description of ARP, to initial sequence number generation. It's a good springboard question, because then you can start talking about spoofing addresses, port scanning, the significance of IPIDs, and more.

We are hiring a lot here at BreakingPoint, which means I'm asking this question a lot. After the fourth or fifth interview, I decided one morning to look over RFC 793 to make sure that I really did know everything there is to know about the handshake. That is when I found out that we've all been living a lie.

If you've spent any reasonable amount of time around network protocols, you're probably familiar with some version of this diagram:

Here, we see the client on the left starting up a conversation with the server on the right. All pretty normal and familar, right? Well, when I was reviewing the RFC again, I noticed something very, very, odd. Disturbing, even. Allow me to quote at some length:

  The synchronization requires each side to send its own initial
  sequence number and to receive a confirmation of it in acknowledgment
  from the other side.  Each side must also receive the other side's
  initial sequence number and send a confirming acknowledgment.

    1) A --> B  SYN my sequence number is X
    2) A <-- B  ACK your sequence number is X
    3) A <-- B  SYN my sequence number is Y
    4) A --> B  ACK your sequence number is Y

  Because steps 2 and 3 can be combined in a single message this is
  called the three-way (or three message) handshake.

Do you see what I see? Because I'm thinking, "this is not a three-way handshake. This is a four-way handshake." The handshake is a lie, born of coalescing steps 2 and 3.

Now, surely, if I just decided to ACK a SYN, then send my own SYN, that couldn't possibly work, right? Enter PacketFu, my little Ruby library for crafting packets. Turns out, 28 years or so after this RFC was written, clients behave rather strangely when you decide to actually honor ol' RFC 793. After some experimentation, I have a pretty decent proof-of-concept stack that behaves like so:

This is the point where things get a little weird. What's happening here is:

    1) A --> B  SYN my sequence number is X
    2) That's nice. I'm not going to bother to ack that, because...
    3) A <-- B  SYN my sequence number is Y.
    4) A --> B  ACK your sequence number is Y, and my sequence number is X.
    5) A <-- B ACK your sequence number is X

Does this work? You betcha! Take a look at the packet captures, collected from Linux (stock Ubuntu), Apple (stock OSX), and Microsoft (stock Windows XP). These three desktop operating systems are all totally cool with this crazy backwards TCP portal.

But what does it mean? Is this simply a parlor trick, where you can reverse the roles of client and server? How does this affect stateful firewalls? How about inspection devices like IPSes, which often need to have an idea of who the "real" client and server are? How about NAT devices, where the idea of "relatedness" is absolutely tied up with where SYN packets come from.

Clearly, there is a ton of testing work to be done here. Lucky for me, I happen to work at a really advanced testing equipment manufacturer, so I've dropped this nugget in the next StrikePack. Now, strikes can employ the "SneakAckHandshake" TCP override option, and all servers simulated will behave in accordance with this crazy backwards handshake. We'll see how well network inspection gear detects clientside attacks when the client is tricked into behaving like a server.

At the very least, now I have better interview questions and I should at least be able to detect if the next candidate is reading this blog. :)

Port Forwarding Fun

Now that I'm spending a fair amount of time writing and testing our new Client Simulator Super Flows, I often find myself with a need for rapid deployment of services to test against. Normally, this is no problem; we have a VMWare server dedicated to Client Simulator development testing hooked up to our BreakingPoint Elite, and the image runs a friendly stock Ubuntu Server.

This week, I started in on ironing out our new Microsoft SQL Cient Simulator, and as great a product as Microsoft SQL Server is, it sadly doesn't install on Linux very well, and since I'm uncannily lazy, I didn't want to go through the hour or two of building out another VMWare image.

Lucky for me, Kirby Kuehl had already installed MSSQL on his own test server (running QEMU) in case we needed any Network Processor magic to make the MSSQL Client Simulator work correctly.

So, I have this VMWare server sitting over here, and a QEMU guest OS sitting over there, in a different network. "No problem," says I, "I'll just set up some port forwarding love!"

Up until now, whenever I have a need to forward network ports around, I default to using ssh tunnels. But in this case, I really wanted the traffic to be cleartext all along the way since I hadn't decided yet where exactly I'd be sniffing the traffic to verify my implementation. Netcat is a fine cleartext alternative for ad-hoc forwarding, but I got it in my head that I wanted a more "real" service running on my VMWare instance.

Since I had already tested out the diagnostic services using this same host -- Chargen, Echo, Daytime, etc -- I had the venerable xinetd installed to run these basic protocols. Somehow, in my many networking adventures, I had forgotten how powerful xinetd can be. Turns out, with an edit to /etc/services (adding 'externsql' as the name for 1433/tcp), adding "kirbys-target" to /etc/hosts, and an entry under /etc/xinetd.d like so...

# default:on
# description: forward various sql services. In this case, 
various is only one.

service externsql
{
	disable = no
	flags = REUSE
	socket_type = stream
	wait = no
	user = root
	redirect = kirbys-target 1433
}

...I'm able to forward my VMWare guest's 1433/tcp traffic across to Kirby's QEMU machine. Xinet.d also features some pretty easy access control configuration and logging (see Jose Nazario's ancient LinuxJournal article on the subject), so if you're in a more hostile network, that might work out for you.

Once that step was out of the way, I noticed that the bridged interface of Kirby's machine was out of whack (it wasn't set to come up on boot), so I couldn't talk to it directly from my VMWare image. Again, I'd like to reference my supreme sloth: instead of proper troubleshooting with brctl, another xinetd redirect (from the QEMU host to the QEMU guest) had me up and running, shuttling MSSQL packets between the BreakingPoint Elite and the far-off SQL Server.

Now that I'm wrapping up my Client Simulator development (look for it in a StrikePack in the near future), it struck me funny that I had so many layers of redirection in order to test traffic. So funny, in fact, I put together this network cartoon so I could share the hilarity with you.

If you're going to be performing any serious level of performance testing, I probably wouldn't recommend such a labyrinthine setup; QEMU and VMWare both can get a little pokey under load, and while xinetd is pretty lightweight, all those stops along the way are adding friction to your ridiculously fast testing gear. But, in the case of functional R&D, this set up is ideal, if a little silly; I'm never going to recable our test Elite again.