pcap_next/pcap_dispatch on VMware vmnet device not timing out

This post might be inappropriate. Click to display it.

Chris Morgan

2009-03-10 00:51:08 UTC

Post by Chris Morgan
Opening a live capture as root (using sudo), on a vmware bridge device
on Linux 2.6.27, using a timeout of 1000ms. I'm seeing pcap_next() and
pcap_dispatch() getting stuck reading, no timeouts are occurring. Is
there a robust and efficient way of reading packets that won't block
forever like this?

Well, the first question is "why is blocking forever an issue?"
Is the application also going to, for example, accept input from other
sources while it's reading captured packets?

Well I noticed the issue when I was trying to shut the capture down gracefully.

Blocking may not be a huge issue. We can forcefully terminate the
capture thread if it doesn't respond within a period of time to a flag
that indicates it should shutdown, it just wasn't my first thought.
One nagging question is the issue of passing the pointer of a managed
function into pcap. I'm wondering if terminating the thread will
result in additional events from pcap_dispatch(). The man page
indicates this is possible with pcap_loop() and pcap_breakloop() so we
might as well take care to not have the handler memory be garbage
collected too early.

Post by Chris Morgan
I'm doing this from that c# library so while I've
thought of using poll() on the devices file descriptor I wanted to
avoid doing low level calls like this so the code would work with
winpcap/pcap on windows/linux platforms.

At least in the case where the app is going to accept input from other
sources, that's what select(), poll(),
WaitForMultipleObjects/MsgWaitForMultipleObjects, etc. are intended for -
they'll wake up if packets arrive or if one of the other sources of input
have input available, rather than having a hack wherein one is (incorrectly)
expecting the timer for the timeout to
1) exist (which it doesn't, on some platforms)
and
2) start timing when you start the read rather than when and *if* the
first packet arrives (which it doesn't, on some platforms, e.g. Solaris; the
Solaris timer starts when the first packet is seen, and, at least according
to the bufmod(7) man page, restarts it on each packet - the man page says
To ensure that messages do not languish forever in an accumulating
chunk, bufmod maintains a read timeout. Whenever this timeout expires, the
module closes off the current chunk and passes it upward. The module
restarts the timeout period when it receives a read side data message and a
timeout is not currently active. These two rules insure that bufmod
minimizes the number of chunks it produces during periods of intense message
activity and that it periodically disposes of all messages during slack
intervals, but avoids any timeout overhead when there is no activity.
indicating that the purpose of the timeout is *not* to enable
polling, but to allow batching of packets without having packets remain
undelivered for an indefinite period of time).
Yes, it's platform-dependent, so you might have to find some way to hide
that beneath your C# class.
Given that, note also that there is at least one UN*X where, unfortunately,
poll() *doesn't* work on the descriptor you get for a pcap_t - Mac OS X,
where, unfortunately, in Mac OS X 10.4 and 10.5, poll() doesn't work on
*any* character special files, including not only the BPF devices that
libpcap uses but also ttys and pseudo-ttys (a source of a number of problems
with various bits of software). select() *does* work on them;
unfortunately, a lot of apps - and libraries such as GLib, as used by GTK+,
for example - use poll() in their event loops.
There are also issues with select() and poll() and BPF devices, wherein you
don't get a wakeup if the read completes due to the timer expiring, so you
need to
1) put the descriptor for the BPF device in non-blocking mode;
2) use a timeout in select() or poll();
3) read from the BPF device when select() or poll() returns,
*regardless* of whether select() or poll() indicates that a read is
possible.
-
This is the tcpdump-workers list.
Visit https://cod.sandelman.ca/ to unsubscribe.

We currently create a thread for each capture device and call back the
user code when packets arrive. Does this remove the requirement for
buffering with timeout since we'll pass the packets off after each
arrives?

Is the timeout something checked after each packet is received? Is
this why pcap_next() is blocking, because no packets are arriving?

Chris

Guy Harris

2009-03-10 07:32:10 UTC

Post by Guy Harris
Well, the first question is "why is blocking forever an issue?"
Is the application also going to, for example, accept input from other
sources while it's reading captured packets?

On UN*X systems, there are two possibilities I can think of offhand:

1) if the shutdown done as the result of a signal, one could arrange
that the signal interrupt system calls and, in the signal handler, use
pcap_breakloop() - the system call reading packets will return an
error of EINTR (interrupted system call) when the signal handler
returns, and (at least in newer versions of libpcap that have
pcap_breakloop()) will, when they see that error, check whether the
"break the loop" flag is set and break out of the pcap_loop() or
pcap_dispatch() loop.

2) a poll() or select() loop (but not poll() on OS X!) with a timeout
could be used, and the flag checked whenever select() or poll()
returns. (That will also work with a signal, as select() or poll()
will give an error of EINTR.)

If you want one thread to be able to shutdown the capturing in another
thread that's running the capture loop, that might be harder. From a
quick look at the Single UNIX Specification, one possibility might be that
all threads other than the capturing thread could block a signal such as
SIGUSR1 or SIGUSR2, the thread doing the capturing could catch that signal
and call pcap_breakloop(), and the thread trying to shutdown the capturing
could send that signal to the process with kill().

A similar scheme to the one described in the previous paragraph might work
on Windows - use WaitForSingleObjectEx() or WaitForMultipleObjectsEx(),
with the "bAlertable" argument being TRUE, and with QueueUserAPC() used
to queue for the target thread an APC that calls pcap_breakloop().

Post by Chris Morgan
One nagging question is the issue of passing the pointer of a managed
function into pcap. I'm wondering if terminating the thread will
result in additional events from pcap_dispatch(). The man page
indicates this is possible with pcap_loop() and pcap_breakloop() so we
might as well take care to not have the handler memory be garbage
collected too early.

Yes.

Post by Chris Morgan
We currently create a thread for each capture device and call back the
user code when packets arrive. Does this remove the requirement for
buffering with timeout since we'll pass the packets off after each
arrives?

What do you mean by "requirement for buffering with timeout"?

Post by Chris Morgan
Is the timeout something checked after each packet is received?

The timeout is implemented in the OS code that libpcap uses, not in
libpcap itself. Some platforms don't have a timeout at all (e.g., Linux);
some platforms have a timer that's started when libpcap (or whatever code
is reading the packets) starts reading (*BSD, Mac OS X, Tru64 UNIX,
Windows with WinPcap); some platforms have a timer that's started when a
packet arrives (Solaris).

Post by Chris Morgan
Is this why pcap_next() is blocking, because no packets are arriving?

That's why code that expects pcap_next() - or pcap_dispatch() or
pcap_loop() or pcap_next_ex() - not to block indefinitely until packets
arrive will not work on a number of OSes, including Linux and Solaris.

Chris Morgan

2009-03-10 16:52:58 UTC

1) if the shutdown done as the result of a signal, one could arrange
that the signal interrupt system calls and, in the signal handler, use
pcap_breakloop() - the system call reading packets will return an
error of EINTR (interrupted system call) when the signal handler
returns, and (at least in newer versions of libpcap that have
pcap_breakloop()) will, when they see that error, check whether the
"break the loop" flag is set and break out of the pcap_loop() or
pcap_dispatch() loop.
2) a poll() or select() loop (but not poll() on OS X!) with a timeout
could be used, and the flag checked whenever select() or poll()
returns. (That will also work with a signal, as select() or poll()
will give an error of EINTR.)

Does mac osx have epoll? I'm leaning towards using the poll()/select()
approach with timeout vs. sending signals but I can't really say why.
I guess I know less about which signal to send and the potential side
effects of sending signals.

If you want one thread to be able to shutdown the capturing in another
thread that's running the capture loop, that might be harder. From a
quick look at the Single UNIX Specification, one possibility might be that
all threads other than the capturing thread could block a signal such as
SIGUSR1 or SIGUSR2, the thread doing the capturing could catch that signal
and call pcap_breakloop(), and the thread trying to shutdown the capturing
could send that signal to the process with kill().
A similar scheme to the one described in the previous paragraph might work
on Windows - use WaitForSingleObjectEx() or WaitForMultipleObjectsEx(),
with the "bAlertable" argument being TRUE, and with QueueUserAPC() used
to queue for the target thread an APC that calls pcap_breakloop().

You mentioned that Linux doesn't support a timeout, I see that the
call used by libpcap is a recvmsg() call. Does Windows have a timeout
on the packet read or do we need a os specific way of checking things
there as well?

Yes.

What do you mean by "requirement for buffering with timeout"?

Because we pass each packet off as we receive it we don't buffer
internal to SharpPcap. I thought that for performance reasons
Wireshark would group packets together and upon reaching a high water
mark or exceeding some interval, that the group of packets would be
passed somewhere. Maybe I'm not understanding your initial comments
about buffering but not holding onto packets forever if the buffer
never fills up.

Post by Chris Morgan
Is the timeout something checked after each packet is received?

I know these may be dumb questions since if they made sense they would
likely have been implemented already.

Is there some reason not to implement pcap_breakloop() such that it
will send a signal to wake up the pcap_dispatch() thread? I suppose
that opens a huge can of worms since now you are dealing with cross
platform thread synchronization issues.

Alternatively why not use poll() or select() inside of pcap_dispatch()
to avoid performing the read if there is no pending data? Wouldn't
that let you implement the timeout functionality and be relatively
cross platform?

Post by Chris Morgan
Is this why pcap_next() is blocking, because no packets are arriving?

Ahh. Yeah and I see that pcap_next() calls pcap_dispatch() which
explains the man page recommendation to call pcap_breakloop() when
using pcap_next.

Chris

Guy Harris

2009-03-11 00:02:29 UTC

Post by Chris Morgan
Does mac osx have epoll?

No. It has poll(), but that, as noted, doesn't work with character
special files, such as the BPF devices used for traffic capture in OS
X. (BPF devices *do* work with poll() in *BSD.)

Post by Chris Morgan
You mentioned that Linux doesn't support a timeout, I see that the
call used by libpcap is a recvmsg() call. Does Windows have a timeout
on the packet read

Yes. It's similar to the BPF timeout, in that the timer starts when
the read is done (rather than when the first packet arrives, as is the
case on Solaris).

Post by Chris Morgan
Because we pass each packet off as we receive it we don't buffer
internal to SharpPcap. I thought that for performance reasons
Wireshark would group packets together and upon reaching a high water
mark or exceeding some interval, that the group of packets would be
passed somewhere. Maybe I'm not understanding your initial comments
about buffering but not holding onto packets forever if the buffer
never fills up.

For performance reasons, some in-kernel packet capture mechanisms,
such as:

BPF, as used in *BSD and Mac OS X

the bufmod STREAMS module in Solaris (which is one component of the
mechanism);

the packetfilter mechanism in Tru64 UNIX;

the WinPcap kernel code;

will *NOT* return from a read as soon as the first packet arrives;
they will, instead, try to accumulate multiple packets and hand all of
them up in a single read.

Obviously, if you get one packet an hour, you don't want to have the
application wait several hours until enough packets have been
accumulated, so those mechanisms also include a timer, so that, after
enough time expires, the read will return even if a bufferful of
packets hasn't been accumulated.

This isn't done by Wireshark, or, rather, by dumpcap - and it isn't
done by tcpdump, or any other such application, either.

It's not even done by libpcap; the applications just call
pcap_open_live(), passing it a timeout value, and pcap_open_live()
passes that timeout value to the kernel packet capture mechanism, if
it supports a timeout.

In BPF and WinPcap, and, I think, in packetfilter, the timer starts
when the read starts, and the read will return after the timer expires
even if *no* packets have been accumulated.

In Solaris, the timer starts - and is reset - whenever a packet
arrives, so the read won't return until at least one packet has been
accumulated.

On those platforms, the buffering will be done regardless of what
SharpPcap does, as it's not something that the caller of libpcap/
WinPcap does - it's done by the kernel-mode code that libpcap and the
WinPcap user-mode code use.

Post by Chris Morgan
Is there some reason not to implement pcap_breakloop() such that it
will send a signal to wake up the pcap_dispatch() thread?

Well, one reason not to do that might be that there *isn't*
necessarily a separate thread doing pcap_dispatch() or pcap_loop()
or....

Post by Chris Morgan
I suppose
that opens a huge can of worms since now you are dealing with cross
platform thread synchronization issues.

Yes. The code would be significantly different between UN*X and
Windows.

If there were a pcap_breakloop_thread() operation, which would take a
pcap_t * and a thread, the UN*X version could use pthread_kill() to
send a signal to the thread and the Windows version could use
QueueUserAPC() to queue an APC for the thread.

That would require that WinPcap use a WaitFor...Ex() call; I'd have to
see whether it already uses a WaitFor...() call or not.

Post by Chris Morgan
Alternatively why not use poll() or select() inside of pcap_dispatch()
to avoid performing the read if there is no pending data?

Because, in many cases, you don't *want* to avoid performing the read
if there's no pending data - you want to wait for data. That's what
tcpdump does, and what the program Wireshark and TShark run to do
captures (dumpcap) does.

Post by Chris Morgan
Wouldn't
that let you implement the timeout functionality

To what timeout functionality are you referring? A timeout that
prevents reads from blocking forever even if no packets arrive is, as
noted, not something that all applications need or want - that's *NOT*
what the timeout specified in pcap_open_live() is intended to do, and
not what it was ever guaranteed to do - and a timeout that just
prevents packets from being buffered indefinitely if they're not
arriving fast enough is *already* implemented.

Chris Morgan

2009-03-11 00:40:10 UTC

Post by Chris Morgan
Does mac osx have epoll?

No. It has poll(), but that, as noted, doesn't work with character special
files, such as the BPF devices used for traffic capture in OS X. (BPF
devices *do* work with poll() in *BSD.)

Hmm. Yeah I'll make sure to put in a comment about mac os support.

Post by Chris Morgan
You mentioned that Linux doesn't support a timeout, I see that the
call used by libpcap is a recvmsg() call. Does Windows have a timeout
on the packet read

Yes. It's similar to the BPF timeout, in that the timer starts when the
read is done (rather than when the first packet arrives, as is the case on
Solaris).

Cool. That should mean that we don't need to we don't need to do
anything special on Windows.

BPF, as used in *BSD and Mac OS X
the bufmod STREAMS module in Solaris (which is one component of the
mechanism);
the packetfilter mechanism in Tru64 UNIX;
the WinPcap kernel code;
will *NOT* return from a read as soon as the first packet arrives; they
will, instead, try to accumulate multiple packets and hand all of them up in
a single read.
Obviously, if you get one packet an hour, you don't want to have the
application wait several hours until enough packets have been accumulated,
so those mechanisms also include a timer, so that, after enough time
expires, the read will return even if a bufferful of packets hasn't been
accumulated.
This isn't done by Wireshark, or, rather, by dumpcap - and it isn't done by
tcpdump, or any other such application, either.
It's not even done by libpcap; the applications just call pcap_open_live(),
passing it a timeout value, and pcap_open_live() passes that timeout value
to the kernel packet capture mechanism, if it supports a timeout.
In BPF and WinPcap, and, I think, in packetfilter, the timer starts when the
read starts, and the read will return after the timer expires even if *no*
packets have been accumulated.
In Solaris, the timer starts - and is reset - whenever a packet arrives, so
the read won't return until at least one packet has been accumulated.
On those platforms, the buffering will be done regardless of what SharpPcap
does, as it's not something that the caller of libpcap/WinPcap does - it's
done by the kernel-mode code that libpcap and the WinPcap user-mode code
use.

Right. I was thinking that there might be cases where if the timeout
wasn't supported you would end up with the os buffering packets and
never returning them to the caller.

How does it work if there is no timeout and packets get captured but
not enough for the callback to occur?

Post by Chris Morgan
Is there some reason not to implement pcap_breakloop() such that it
will send a signal to wake up the pcap_dispatch() thread?

Well, one reason not to do that might be that there *isn't* necessarily a
separate thread doing pcap_dispatch() or pcap_loop() or....

Post by Chris Morgan
I suppose
that opens a huge can of worms since now you are dealing with cross
platform thread synchronization issues.

Yes. The code would be significantly different between UN*X and Windows.
If there were a pcap_breakloop_thread() operation, which would take a pcap_t
* and a thread, the UN*X version could use pthread_kill() to send a signal
to the thread and the Windows version could use QueueUserAPC() to queue an
APC for the thread.
That would require that WinPcap use a WaitFor...Ex() call; I'd have to see
whether it already uses a WaitFor...() call or not.

Ahh. Considering the delay in adding such a feature and getting a
release out etc it seems simpler to just handle the issue at the
application level.

Post by Chris Morgan
Alternatively why not use poll() or select() inside of pcap_dispatch()
to avoid performing the read if there is no pending data?

Because, in many cases, you don't *want* to avoid performing the read if
there's no pending data - you want to wait for data. That's what tcpdump
does, and what the program Wireshark and TShark run to do captures (dumpcap)
does.

Wouldn't using poll if the timeout was set and not if it wasn't solve
that problem?

Post by Chris Morgan
Wouldn't
that let you implement the timeout functionality

To what timeout functionality are you referring? A timeout that prevents
reads from blocking forever even if no packets arrive is, as noted, not
something that all applications need or want - that's *NOT* what the timeout
specified in pcap_open_live() is intended to do, and not what it was ever
guaranteed to do - and a timeout that just prevents packets from being
buffered indefinitely if they're not arriving fast enough is *already*
implemented.
-

Right, I'm referring to the to_ms timeout in pcap_open_live(). I see
what you mean, the man page certainly disclaims support for the
timeout on all platforms. The documentation on to_ms does sound like
exactly what I'm trying to do though, cause the read to timeout. I was
trying to use this functionality to have the read abort and return to
the caller.

I'll just use poll() under linux to avoid the read if there are no
packets available. That will let us avoid using signals and keep cpu
usage low by reducing the idle call rate to once a second or
something.

Chris

Guy Harris

2009-03-11 01:01:01 UTC

Post by Chris Morgan
Hmm. Yeah I'll make sure to put in a comment about mac os support.

Note that select() *does* work with BPF devices on OS X - modulo the
traditional BPF bug wherein the timer starts when a read is done,
*not* when a select() is done, so you *could* block forever. At this
point I think all the *BSDs have fixed that in their BPF
implementations, but OS X hasn't fixed it yet.

(If BPF had done Solaris-style timeouts, where the timer is an inter-
packet timer, this wouldn't have been an issue in the first place. It
might also mean that the timer would have done a better job of
batching packets if they're arriving fast and delivering them when
there's a pause.)

Post by Chris Morgan
Right. I was thinking that there might be cases where if the timeout
wasn't supported you would end up with the os buffering packets and
never returning them to the caller.

No. If the timeout isn't supported, the OS doesn't buffer any packets
- it delivers them immediately, even if that means that a burst of
packets is delivered one packet at a time, with a trip to the kernel
done for each packet.

Post by Chris Morgan
How does it work if there is no timeout and packets get captured but
not enough for the callback to occur?

If there's no timeout, that's because there's no in-kernel buffering
to require a timeout, so "enough for the callout to occur" is one
packet.

Post by Chris Morgan
Wouldn't using poll if the timeout was set and not if it wasn't solve
that problem?

No, because tcpdump and dumpcap *do* want a timeout, because they
don't want to, for example, block for several hours if you only get
one packet an hour. They don't care whether the read blocks forever
waiting for the *first* packet to arrive, they just don't want it to
block forever waiting for a bufferful of packets to arrive.

Post by Chris Morgan
Wouldn't
that let you implement the timeout functionality

To what timeout functionality are you referring? A timeout that
prevents
reads from blocking forever even if no packets arrive is, as noted, not
something that all applications need or want - that's *NOT* what the timeout
specified in pcap_open_live() is intended to do, and not what it was ever
guaranteed to do - and a timeout that just prevents packets from being
buffered indefinitely if they're not arriving fast enough is
*already*
implemented.
-

Right, I'm referring to the to_ms timeout in pcap_open_live().

That timeout is, at least on Solaris, "a timeout that just prevents
packets from being buffered indefinitely if they're not arriving fast
enough." On some other OSes, it also happens to prevent reads from
blocking forever if no packets arrive, but applications should not be
depending on that unless they also call uname() and abort if the OS
name string is "SunOS", so that they ensure that they never run on
Solaris.

Post by Chris Morgan
I see
what you mean, the man page certainly disclaims support for the
timeout on all platforms. The documentation on to_ms does sound like
exactly what I'm trying to do though, cause the read to timeout.

I tried as hard as I could to write the documentation *NOT* to sound
like that. The problem is that I didn't mention that until the
description of pcap_dispatch():

NOTE: when reading a live capture, pcap_dispatch() will not
necessarily
return when the read times out; on some platforms, the
read timeout
isn't supported, and, on other platforms, the timer doesn't
start until
at least one packet arrives. This means that the read
timeout should
NOT be used in, for example, an interactive application, to
allow the
packet capture loop to ``poll'' for user input periodically,
as there's
no guarantee that pcap_dispatch() will return after
the timeout
expires.

In libpcap 1.x, the pcap man page discusses the timeout in some
detail, and does explicitly note that it's *NOT* guaranteed to keep
reads from blocking forever:

read timeout
If, when capturing, packets are delivered as soon
as they
arrive, the application capturing the packets will be
woken up
for each packet as it arrives, and might have to
make one or
more calls to the operating system to fetch each packet.

If, instead, packets are not delivered as soon as
they arrive,
but are delivered after a short delay (called a "read
timeout"),
more than one packet can be accumulated before the
packets are
delivered, so that a single wakeup would be done for
multiple
packets, and each set of calls made to the
operating system
would supply multiple packets, rather than a
single packet.
This reduces the per-packet CPU overhead if packets are
arriving
at a high rate, increasing the number of packets per
second that
can be captured.

The read timeout is required so that an application
won't wait
for the operating system's capture buffer to fill
up before
packets are delivered; if packets are arriving slowly,
that wait
could take an arbitrarily long period of time.

Not all platforms support a read timeout; on
platforms that
don't, the read timeout is ignored. A zero value for
the time-
out, on platforms that support a read timeout, will
cause a read
to wait forever to allow enough packets to arrive, with
no time-
out.

NOTE: the read timeout cannot be used to cause calls
that read
packets to return within a limited period of time,
because, on
some platforms, the read timeout isn't supported, and,
on other
platforms, the timer doesn't start until at least
one packet
arrives. This means that the read timeout should NOT
be used,
for example, in an interactive application to allow
the packet
capture loop to ``poll'' for user input periodically,
as there's
no guarantee that a call reading packets will return
after the
timeout expires even if no packets have arrived.

The read timeout is set with pcap_set_timeout().

and the man pages for the individual routines just refer to the read
timeout without giving details.

Chris Morgan

2009-03-11 01:13:17 UTC

Post by Chris Morgan
Hmm. Yeah I'll make sure to put in a comment about mac os support.

Yeah, I've heard that OS X has a lot of bugs that are reported but not
fixed so it doesn't surprise me.

Post by Guy Harris
(If BPF had done Solaris-style timeouts, where the timer is an inter-packet
timer, this wouldn't have been an issue in the first place. It might also
mean that the timer would have done a better job of batching packets if
they're arriving fast and delivering them when there's a pause.)

Post by Chris Morgan
Right. I was thinking that there might be cases where if the timeout
wasn't supported you would end up with the os buffering packets and
never returning them to the caller.

No. If the timeout isn't supported, the OS doesn't buffer any packets - it
delivers them immediately, even if that means that a burst of packets is
delivered one packet at a time, with a trip to the kernel done for each
packet.

Ahh. That makes sense.

Post by Chris Morgan
How does it work if there is no timeout and packets get captured but
not enough for the callback to occur?

If there's no timeout, that's because there's no in-kernel buffering to
require a timeout, so "enough for the callout to occur" is one packet.

Post by Chris Morgan
Wouldn't using poll if the timeout was set and not if it wasn't solve
that problem?

No, because tcpdump and dumpcap *do* want a timeout, because they don't want
to, for example, block for several hours if you only get one packet an hour.
They don't care whether the read blocks forever waiting for the *first*
packet to arrive, they just don't want it to block forever waiting for a
bufferful of packets to arrive.

Right.