Data center: Ashburn, VA

Telegram Chat : MBHH_x86

Email Us: Help@mobilehackerforhire.com

Mobile Hacker For Hire, hire a hacker, hiring a hacker, hacker with proof

Live Capture Performance to Rival Wireshark

Table of Contents





From: Matthew Davis <matthewndavis () gmail com>
Date: Sat, 20 Aug 2022 17:46:59 -0500


Hello.

The project I work on is a Windows-only, 64-bit-only application that
collects and processes application layer messages across various
system-specific protocols.  Our legacy application used the old Windows raw
socket mechanism (before migrating to WinPcap 4.1.3) to read the packets
off the wire.  Our revamped application is using the OEM version of Npcap
with MUCH better success.

However...

A recent survey of our log files from the field indicates that we are
missing packets.  Specifically,  converting our log files to .pcapng and
opening them in Wireshark, we see about 1% of the packets showing the [TCP
Previous segment not captured] message.  Due to the nature of this data,
this 1% loss is unacceptable to our users.  As expected, this loss gets
more dramatic with additional network traffic.  Testing in the lab shows
that Wireshark v3.6.7 captures the packets from a stress test with no
apparent packet loss, so I know the problem is on our end.

I think there are some application-level improvements we can make, but I'm
interested in knowing if we are configuring and executing our packet
capture in the most efficient way possible.

Here is a brief summary of how we are doing things currently.

- We call pcap_open() with a snap length of 65536, promiscuous mode
enabled, and a read timeout of 500ms.
- We call pcap_setbuff() to increase the size of the kernel buffer to 16MB.
- We then call pcap_next_ex() inside a for loop to get the next capture.
- Upon successful return, we allocate a byte array using pkt_header.caplen,
copy the pkt_data into the byte array, and add the byte array to a
pre-allocated list.
- We execute this for loop until the pre-allocated list is filled (to avoid
reallocation) or a predetermined timeout is exceeded on the application
side.
- When either of these conditions is satisfied, we hand the pre-allocated
list off to another thread, allocate a new list, and do the loop again.

Here are my questions.

1.  Is pcap_next_ex() the most efficient way of transferring captures to
the application?  It looks like pcap_loop() or pcap_dispatch() might allow
multiple captures to be returned via a single callback.  Is that correct?
And if so, would that be the recommended way to get the captures in a high
data rate environment?

2.  Our understanding is that the kernel buffer *IS* the ring buffer that
must be read from at least as fast as the data is received in order to
minimize/eliminate the occurrence of dropped packets.  We understand the
size of the buffer won't prevent dropped packets if the application can't
keep up (it merely delays the moment when that occurs).  But a bigger ring
buffer can accommodate data spikes, allowing the application to catch up
during data lulls.  To this end, how big can we make the kernel buffer via
pcap_setbuff()?  Is there a practical or rule-of-thumb limit?  Is the ring
buffer associated with each handle???  If we collect simultaneously with
Npcap on multiple NICs, does the size of each ring buffer need to be
limited in any way?  Should we use pcap_set_buffer_size() instead of
pcap_setbuff()?  Can pcap_set_buffer_size() be called after pcap_open()
like pcap_setbuff() can?  Or do we need to use the create-set-activate
pattern?

3.  We are confused about the difference between the kernel buffer and the
user buffer. How does the user buffer work with pcap_next_ex()?  Since
pcap_next_ex() only returns a single packet at a time, does the user buffer
even matter?  Perhaps it comes into play with pcap_loop() or
pcap_dispatch() being able to return more data in the callback?

4.  How does pcap_mintocopy() work with pcap_next_ex()?  Again, since
pcap_next_ex() only returns a single packet at a time, does this even
apply?  Perhaps with pcap_loop() or pcap_dispatch()?

5.  Finally, can stats be enabled for the same handle that is capturing the
data?  If we want to monitor, for example, the number of dropped packets
seen during a capture, how do we do that with the pcap_t returned by
pcap_open()?  If we need two pcap_t handles, one for capture and one for
stats, does that imply a single ring buffer under the hood for a given NIC?

As stated earlier, we are using Windows 64-bit only with Npcap OEM 1.70.

I apologize for the length, but any help/guidance is GREATLY appreciated.
And hopefully this will help others out as well.

Matt Davis
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at https://seclists.org/nmap-dev/




Current thread:

  • Live Capture Performance to Rival Wireshark Matthew Davis (Sep 12)

Leave a Reply

Your email address will not be published. Required fields are marked *

error: Content is protected !!