This all started with a simple request.  A client had purchased some new, shiny networking gear, and in each failover pair the active unit was sending 1 "Runt" per second.

A "runt" is a frame that is smaller than the legal minimum, in other words, 64 bytes.  I guess it doesn't qualify as a packet, since the frame would encapsulate the packet, and since the frame isn't valid to decapsulate, there is no packet to see. Usually you see runt frames when you have a duplex mismatch, in this case it looks like they were inflicted on us by the driver on this piece of gear.  In addition to the switch port's "runt" counter incrementing, the "input error" counter on the switch port is counting up in lock step.

No problem, go to the vendor and get it fixed with an update is the answer you'd expect.  But no, the vendor doesn't believe us, or at least not enough to do anything, they want us to capture some of these frames.

OK, so I start by setting up a capture session on the switch, with a SPAN / mirror port.  At this point I find out that the switch port drops invalid frames before they are even completely recieved, so this approach only forwards the valid frames.

Next, we tried using a TAP (an inline network device that has a "listener" port).  It turns out that most modern TAPs (1Gbps or better) are active devices, they're essentially small switches and have the same issue as the switch I was starting from.  So we tried using a passive tap, which is essentially an electrical device that has the "Rx" ethernet pins "tee'd" off to the listener port.  This works!  On with the capture we said!

Not so fast!  Your ethernet NIC does the same thing as a switch port, the hardware drops invalid frames before they reach the OS.  Luckily in Linux you can adjust this, you can use the "ethtool" tool to disable this feature:

ethtool -K eth0 rx-fcs on  will enable the receive of frames that fail the fcs (frame checksum) check.
ethtool -K eth0 rx-all on  will enable receiving of other invalid frames (like runts)

These options are not available for any Windows drivers I've been able to find - you'd expect to find them under the "advanced" tab for the driver configuration.

OK, all that said, good to go right?
Nope, these options aren't available on many NICs, especially USB NICs like we're forced to use on so many modern laptops.

ethtool -k eth0 | grep rx  will list all the receive options available on your NIC.

Most USB adapters are either Realtek or ASIX, and they both show the two key flags as "fixed", as in not changeable.  I had an older Broadcom USB NIC (the Apple Thunderbolt-2 Ethernet card), same deal:

Luckily, my main laptop has an on-board Intel NIC, which allows you to adjust lots of the knobs available (certainly the ones we're looking for)!

# ethtool -k eth0 | grep rx

rx-checksumming: on
rx-vlan-offload: on
rx-vlan-filter: off [fixed]
rx-fcs: off
rx-all: off
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]

Ok, NOW we're ready to go, right?  We set the whole thing up, with a capture filter of: len < 65 (this is packet length not frame length, so it still sees ARP and other small packets, but at least it filters the majority of the traffic out)

.... And we still don't get our target frames.  We know that we're still receiving runt frames  - we still see them on the switch, and ethtool shows them on the capturing PC - but tcpdump isn't seeing them?

At that point, we go to the wireshark FAQ, and see that even after all this work, libpcap is our last roadbock.  libpcap will not capture invalid frames, so that means tcpdump, wireshark and anything that uses tcpdump won't.
https://www.wireshark.org/faq.html#_how_can_i_capture_entire_frames_including_the_fcs

Looking into various posts on libpcap, we see the same messages echo'd "we'll capture any valid frame" ....

If I remember right, (way way) back in the day, the Network General Sniffer boxes could do this, but that's going back to 10/100mbps ethernet days.

So the question to the community is - has anyone seen a combo of NIC, driver, OS and library that will capture invalid frames?  Please, use our comment form if you've seen anything that works.  Or if you've been in a similar situation of needing to capture traffic but couldn't we're all ears on that too!

===============
Rob VandenBrink
rob@coherentsecurity.com