As more and more businesses are moving their infrastructure from physical on premise devices to cloud services they are hitting obstacles when it comes to monitoring their cloud-infrastructure.

This is because the cloud instances you're running aren't really connected to a real network, but to a virtualized version of one. This often makes it impossible to get the same features you would in a physical network like port mirroring for monitoring network traffic in real-time. For medium and large companies this problem can be fixed by buying expensive products, but small companies don't have the same budget and are more than often relying on open-source software to keep cost to a minimum. Since it would be expensive to buy monitoring software with full potential and implement it in the cloud, we need to find a cheap way to do this without changing the infrastructure.

This blog-post presents a way of mimicking the features of port mirroring by configuring a Linux device without relying on physical hardware (this won't work on Windows!). The monitoring system in this case is Security Onion, a great open-source Network Security Monitoring system (NSM) with features like full pcap capture and Intrusion Detection Systems. Be careful since this approach will require extra resources as it runs tcpdump, tcpreplay and tcprewrite. Make sure you choose your cloud instance accordingly. This particular solution is running on Amazon AWS and the Security Onion server is running as a t2.large EC2 instance. The servers being monitored are all t2.micro instances.

Inventory:

• 1 Linux machine running a web server on Amazon AWS
• 1 Linux machine running Security Onion on Amazon AWS

In my case I have two interfaces per machine, one for transmitting the network traffic and one for production. On the Security Onion server we will create an additional dummy interface and set it to promiscuous mode, this will be the monitor interface for Security Onion.

root@bash$ ip link add dev eth1 type dummy
root@bash$ ip link set eth1 promisc on arp off up

To make sure it is added at startup, add this to your /etc/rc.local file. Add this to /etc/rc.local to create the interface on startup

## rc.local ##
ip link add dev eth1 type dummy
ip link set eth1 promisc on arp off up
exit 0
## EOF ##

An optional step is to add specific users to each monitored system to be used only for tcpdump. Create a user (ex. seconion) on each monitored system, add them to a newly created group and change the group of "/usr/sbin/tcpdump" to this newly created group, then edit the permissions on the binary so members of the group can run the command. To simplify the process of remote SSH access, create and add the public SSH key from the Security Onion server to each monitored systems "seconion" users authorized_keys file.

Start a screen terminal (or just start another ssh session) to run tcpdump on the remote machine through ssh. We also need to use tcprewrite to write the correct MCU to the pcap file. Below we have some crazy one-liners. It never saves a thing to disk and only uses standard in and out to send the traffic to the target interface. If you know of any way to improve on this solution let me know!

Note! Be careful what you capture. Make sure you filter out unwanted traffic.

root@bash$ screen
root@screen$ ssh user@remote "/usr/sbin/tcpdump -i eth0 -w /dev/stdout -U 2>/dev/null" | tcprewrite --mtu-trunc --infile=- --outfile=/dev/stdout 2>/dev/null | tail -c +0 -f 2>/dev/null | tcpreplay --intf1=eth1 --topspeed - 2>/dev/null

Explaining the tail command:

-c +0 will start from the beginning of the file
-f will follow the file (never stop reading, wait for more data)

Explaining the tcpreplay command:

--intf1=eth1 will replay traffic from pcap to target interface "eth1" which is the monitor interface created earlier.
--topspeed will send the traffic as fast as possible
The dash (-) will tell tcpreplay to get the input from stdout and the and (&) character will run the command in the background.

Do this for every server you want to monitor. Make sure you use the correct IP address and the correct interface on the target machine.

We use the command tail to continuously print the file to stdout and pipe the output to tcpreplay. Tcpreplay will "play" the file to the listening monitor interface.

Since this is a small trick to get full pcap without port mirroring we will not explain how to setup Security Onion. Please review their documentation on how to do a proper setup.

Now you are ready to go, good luck!