UDP / TCP Checksum errors from tcpdump & NIC Hardware Offloading

If you’ve ever tried to trace a UDP or TCP stream by using the tcpdump tool on Linux then you may have noticed that all, or at least most, packets indicate checksum errors. This is caused because you have checksum offloading on your network card (NIC) and tcpdump reads IP packets from the Linux kernel right before the actual checksum takes place in the NIC’s chipset. That’s why you only see errors in tcpdump and your network traffic works ok.

So, just to proove my point, here is a tcpdump output while monitoring DNS traffic (udp/53)


$ sudo tcpdump -i eth0 -vvv -nn udp dst port 53

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:04:48.145904 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 61)
10.0.0.2.56497 > 10.0.0.1.53: [bad udp cksum 0x8f54 -> 0xb8fc!] 30234+ AAAA? http://www.twitter.com. (33)
17:04:48.145925 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 61)
10.0.0.2.56497 > 10.0.0.1.53: [bad udp cksum 0x224d -> 0x2604!] 30234+ AAAA? http://www.twitter.com. (33)

After checking active NIC hardware offloading options you can see the obvious


$ sudo ethtool -k eth0 | grep on
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
generic-segmentation-offload: on
generic-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on

After disabling TCO (tcp offloading) for TX/RX on the NIC the problem is gone


$ sudo ethtool -K eth0 tx off rx off


$ sudo tcpdump -i eth0 -vvv -nn udp dst port 53

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:06:09.355411 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 57)
10.0.0.2.18964 > 10.0.0.1.53: [udp sum ok] 292+ AAAA? twitter.com. (29)
17:06:09.355431 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 57)
10.0.0.2.18964 > 10.0.0.1.53: [udp sum ok] 292+ AAAA? twitter.com. (29)

For the sake of performance, remember to turn TCO back on after each tcpdump execution. 😉

If you saved the tcpdump output and later you need to correct the bad checksums then you can do one of the following:


$ sudo tcpreplay -i eth0 -F -w output.cap input.cap

or


$ sudo tcprewrite -i input.cap -o output.cap -C

Edit:
In this excellent article you can see the whole process illustrated as well as the impact of Generic Segmentation Offloading during packet capture.

Ubuntu Precise 12.04: Get rid of NM’s dnsmasq and setup your own!

Anyone from an older version with a working dnsmasq setup must have noticed that after a dist-upgrade to Ubuntu Precise 12.04 dnsmasq is having trouble on startup. The reason is simple. On the upcoming 12.04 Desktop release, Network Manager is starting its own dnsmasq instance which is binded on the localhost ip (127.0.0.1) and is responsible for making sure all LAN, WAN or even VPN connections have a proper DNS handling from a routing prespective. No caching at all! For more info on that change read this blog post.

As already described, NM’s dnsmasq instance listens on localhost. With a little help from the resolvconf package (this is also new to 12.04 Desktop), the /etc/recolv.conf file will look like this.


# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1

The previous resolv.conf entry means that all DNS requests from your Desktop will be headed to the local dnsmasq instance started by the Network Manager.

If you are a power user and you want your dnsmasq custom configuration to be NM dependent all you have to do is comment out the “dns=dnsmasq” parameter in the Network Manager’s configuration


sudo vi /etc/NetworkManager/NetworkManager.conf

and restart the NM service


sudo restart network-manager

After that, try to restart your previously configured dnsmasq and see the results. Everything is in place now!

  1. /etc/resolv.conf is no longer dependant on resolvconf package and is updated based on the DNS entries of each connection in the Network Manager
  2. dnsmasq can not bind freely on the localhost (127.0.0.1:53)

Last but not least, here are some dnsmasq configuration parameters I found helpful until now.
You should use them in “/etc/dnsmasq.conf” (remember to edit it as root with sudo).


# Listen only on localhost
listen-address=127.0.0.1
bind-interfaces

# The default is 1000 dns entries to be cached.
cache-size=10000

# Make sure that DNS requests going out contain a valid domain
domain-needed

# The resolv file which dnsmasq will check for its default nameservers
# The syntax for that file is the same as for classic resolv.conf
resolv-file=/etc/resolv.dnsmasq

# Do not try to check the default /etc/resolv.conf file for any reason
no-poll

# All DNS requests for domain.prive, its subdomains and the reverse maps
# should be sent to the proper DNS. Adjust to your environment.
server=/domain.prive/10.0.0.1
server=/10.in-addr.arpa/10.0.0.1

Restart the dnsmasq service and do some DNS tests to make sure it working as expected


sudo service dnsmasq restart

dig http://www.google.com @127.0.0.1
dig blah.domain.prive @127.0.0.1

If the previous dig results have proven to be ok then edit your Network Manager LAN connections and make sure they have dnsmasq as a primary DNS server. The address is 127.0.0.1.

To monitor dnsmasq’s statistics (cache usage, etc) all you have to do is send the process a USR1 signal and the statistics will be outputed in syslog.


sudo kill -USR1 `pgrep dnsmasq`
sudo less /var/log/syslog

The output should similar to that bellow


Mar 31 18:55:06 pc dnsmasq[16919]: time 1333209306
Mar 31 18:55:06 pc dnsmasq[16919]: cache size 10000, 0/2877 cache insertions re-used unexpired cache entries.
Mar 31 18:55:06 pc dnsmasq[16919]: queries forwarded 1437, queries answered locally 907
Mar 31 18:55:06 pc dnsmasq[16919]: server 194.219.227.2#53: queries sent 0, retried or failed 0
Mar 31 18:55:06 pc dnsmasq[16919]: server 10.32.54.4#53: queries sent 64, retried or failed 32
Mar 31 18:55:06 pc dnsmasq[16919]: server 193.92.3.11#53: queries sent 1118, retried or failed 2
Mar 31 18:55:06 pc dnsmasq[16919]: server 193.92.150.3#53: queries sent 360, retried or failed 0