(Tcp)traceroute

Tcptraceroute per the author's website is "a traceroute implementation using TCP SYN packets, instead of the more traditional UDP or ICMP ECHO packets. In doing so, it is able to trace through many common firewall filters."

Before we get into tcptraceroute, let's get started first with traceroute, which has been the standard historical network tool to trace a packet's route across a network. It's a great tool but a lot of sites may simply block that traffic minimizing it's usefulness.

Traceroute works using a few principals:

1) A low, but increasing time to live (TTL), usually starting at 1. (This can be set with the -f flag).

2) An invalid destination port (generally UDP port 33434 for Unix systems, Microsoft tends to use ICMP)

As each router on the network processes an IP packet, it will decrement the TTL by 1 and forward the packet on. However, an IP packet with a TTL of 0 is discarded and then an ICMP packet with a message of "TTL Expired in Transit" is sent to the source IP sending the packet. This reduces the chance of infinite routing loops.

So that's how the Traceroute program builds the route list, by listing the IPs of the router that sent the message as shown here in a traceroute to yahoo.com.

Note that the Source IPs that are associated with the TTL Expired in Transit messages are changing. Those are the routers sending these messages:



The Traceroute program knows it's reached the host because since UDP port 33434 is generally invalid, the remote host will reply with an ICMP packet saying "destination unreachable" (port unreachable).

Here's a network capture showing a Linux host tracing a route:



Note that the initial UDP port is 33434 and it increases for each packet sent and also that the remote host we are tracing to sent us back a "port unreachable" ICMP packet.

Now as we mentioned tcptraceroute uses TCP SYN packets and defaults to port 80, which is generally open on the remote host if it is a web server or at least generally are all allowed across routers.

Compare difference between traceroute and tcptraceroute:


http://michael.toren.net/code/tcptraceroute/examples.txt


Wow, all the way through!

Here's how a network capture looks when we tcptraceroute to pages.ebay.com:



Note when tcptraceroute finally reaches the remote host it sends a packet with the SYN bit set, the remote server responds with SYN and ACK bits set, and tcptraceroute resets the connection with RST.

Pretty cool stuff when you're looking to trace a route!

Deny Hosts with..........DenyHosts!

While I was looking for some security tools recently I had come across DenyHosts.

DenyHosts is a Python script that will allow you to monitor a Linux system for failed SSH logins and add to their IP address to /etc/hosts.deny based on the number of failed attempts. ( This assumes you have set up your SSH install to be protected by TCP wrappers. )

One of the nice features about this script is that you can set different limits for the number of allowed failed logins for the root user, users that do not exist on this system, and users that do exist.

Installing this software was very easy as it was available as "apt-get install denyhosts".

Configuring the script was also very easy and didn't take more than about 10 minutes or so to read through the configuration file and set a few parameters that the author documents very well.

After that I just set up the script to be run from Cron every 5 minutes or so:


and Denyhosts was all ready to go.

As an example, you can see in on the secure log ( /var/log/auth.log on my system), the users Devil, bob, hacz0r, root, and loot have failed login attempts from 1.2.3.4:

Jun 11 15:51:05 RuleNumber1 sshd[1113]: Failed none for invalid user Devil from 1.2.3.4 port 35722 ssh2
Jun 11 15:51:06 RuleNumber1 sshd[1113]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4
Jun 11 15:51:08 RuleNumber1 sshd[1113]: Failed password for invalid user loot from 1.2.3.4 port 35722 ssh2
Jun 11 15:51:18 RuleNumber1 sshd[1117]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4 user=root
Jun 11 15:51:20 RuleNumber1 sshd[1117]: Failed password for root from 1.2.3.4 port 46446 ssh2
Jun 11 15:51:26 RuleNumber1 sshd[1117]: Failed password for root from 1.2.3.4 port 46446 ssh2
Jun 11 15:51:30 RuleNumber1 sshd[1117]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4 user=root
Jun 11 15:51:41 RuleNumber1 sshd[1121]: Failed password for invalid user hacz0r from 1.2.3.4 port 41470 ssh2
Jun 11 15:51:53 RuleNumber1 sshd[1121]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4
Jun 11 16:05:21 RuleNumber1 sshd[1177]: pam_unix(sshd:auth): authentication failure; logname=


and a few minutes later we see that deny hosts has done its job and added the IP address to our deny file:

cmd#cat /etc/hosts.deny
sshd: 1.2.3.4


Another nice feature of DenyHosts is that you can use synchronization mode, where you are accepting information about IP addresses that have been blocked for other users' hosts and/or (it's configurable) upload the IP addresses you have blocked, in a sense collaborating on a large black list of IP addresses.

All in all this was very easy to install and configure, there is one main configuration file this easy to understand and it is well commented.

Deny hosts makes a nice addition to add another level of security to your SSH setup, and I'd suggest adding this to your toolkit.

RST - Who are you?

So the other day I was chatting with some technical (geeky!) friends of mine and the conversation turned to TCP flags. So the question was posed in the group, "What happens if you connect to a remote host and you get a connection refused message?"

Well *one* short answer is the remote side reset the connection, most likely because the particular application or port is not listening on that particular port.

That is to say that when you connect to a Web server on port 80 the connection (the operating system's TCP/IP stack that is running the web browser, or a command line utility, etc) will go through the standard TCP handshake to connect to that well-known port number 80.

Say you connect to port 73, and there is no application running at that IP address and port, the correct thing for that remote host to do is to reply with a packet with the RST flag set, resetting the connection, which is better than the other side just waiting until it times out and gives up.

In layman's terms is basically saying "There is no one here, please go away and try somewhere else".

This is simply one example, more details on TCP (and the RST flag) can be found in this RFC:

Well, so what, why is this important?

Well in the course of troubleshooting it will become important as the following example illustrates:

I was on my customer's network troubleshooting a network connectivity problem.

Basically they were trying to index content on their Web server secured with SSL (https) and they're having a heck of a time with connections from their internal network to the host.

I didn't have any thorough knowledge of their network topology, nor did the customer at the time, however I did have my Linux host and the tcpdump command at the ready so I was able to do a packet capture and then some analysis on the results.

But before that I did some basic tasks:

I did a simple telnet to port 80 to see if there was any Web server running.

Sure enough there was - and when I gave it an HTTP GET request for the / document, it redirected me to port 443. Good to know.

Port 80 is not necessarily directly related to the Web server and its associated port 443 but at least I can confirm to basic IP and TCP connectivity to that host.

So off now to port 443:

First I also issued a telnet to port 443.

That basically didn't give me any real information, nor will it, generally, because the telnet command doesn't really understand the SSL program or protocol. Actually it doesn't at all. For example:

#telnet www.blogger.com 443
Trying 74.125.19.191...
Connected to www.blogger.com (74.125.19.191).
Escape character is '^]'.
Connection closed by foreign host.

All this did was check to see if port 443 was open and listening, which it was, or if it quickly reset, or gave any other information. (plus it's the shortest command to type and remember).

So the next step in troubleshooting would be to use something like the OpenSSL utilty, which is basically an SSL enabled telnet. It's actually quite a bit more than that but for our purposes we're going to connect to the host on port 443 and see what the responses, and then issue some HTTP commands.

However what I found was this response from OpenSSL:

#openssl s_client -connect customerweb:443
CONNECTED(00000003)
write:errno=104

Curl didn't fair much better:

#curl -i "https://customerweb/"
curl: (35) SSL: error:00000000:lib(0):func(0):reason(0)

As you can see neither output was very helpful and clearly doesn't indicate success.

What I would be expecting is something more like this from openSSL:



As you can see the OpenSSL command has connected to the secure blogger website and retrieved some certificate information. This confirms that's a secure Web server at the other end is able to be contacted and we can read the certificate.

So I forwarded the errors and information to the customer and suggested for their network team to look into the network hardware in between these two devices, the first one which incidentally was a Google Search Appliance, which I work often times with.

That's fine and helpful but not really getting to the heart of the issue.

We need to once again check in with TCPdump and really see what's going on a packet level. So I did and this is what I found:

The Google search appliance was making a valid connection to port 443 however the remote host was waiting for a few seconds and then coming back with an RST flag set in the packet, basically resetting the connection, as if to say note is no Web server listening here as shown in the packet trace:



I saw this using Wireshark, which is a great tool for analyzing tcpdump files. It is actually able to resolve the first 3 bytes of the Mac address to the the vendor who has been assigned that block of MAC addresses, which can be really helpful and a time saver.

In this case the default gateway was in fact a device manufactured by the F5 Corporation. Given that, our picture looks a little more clear:

"GSA"->"F5/LB"->"HTTPS Customer WebServer".

Again without even knowing about the full network topology it was clear that the path the Google Search Appliance was going through over network there definitely was F5 device (most likely a firewall or load balancer), just based on the MAC address of the default gateway for the GSA.

The resolution of this in fact was a problem with the F5 load balancer, which taken care of with an update to its configuration.

But here is a good example of starting from what you know, what is available to you, and just knowing a little bit about TCP flags, and how they're used to get closer to a solution.