Kqueue and threading

[Willem Jan Withagen]

Hi,

Ceph uses a lot of threading, and for any part of it communication it
uses epoll(), which FreeBSD does not use. For that there was already a
EvenKqueue implementation.

But I think I'm now running into:
The kqueue() system call creates a new kernel event queue and
returns a descriptor. The queue is not inherited by a child
created with fork(2). However, if rfork(2) is called without the
RFFDG flag, then the descriptor table is shared, which will allow
sharing of the kqueue between two processes.

Kqueue descriptors are created and events are added, but then the
process starts other threads and expects the kqueue-id to be valid there.

However adding more events returns an error, also waiting on the ID for
events to happen returns an error (descriptor invalid)

Threading is done with 2 different constructions:
std::thread
and creating Workers

Would any of these qualify with the quoted warning? and invalidate the
descriptor table?

If not, how can I (easily) debug the reason why my descriptors go invalid?


thanx,
--WjW

[Eric Badger]

Post by Willem Jan Withagen
Hi,
Ceph uses a lot of threading, and for any part of it communication it
uses epoll(), which FreeBSD does not use. For that there was already a
EvenKqueue implementation.
The kqueue() system call creates a new kernel event queue and
returns a descriptor. The queue is not inherited by a child
created with fork(2). However, if rfork(2) is called without the
RFFDG flag, then the descriptor table is shared, which will allow
sharing of the kqueue between two processes.
Kqueue descriptors are created and events are added, but then the
process starts other threads and expects the kqueue-id to be valid there.
However adding more events returns an error, also waiting on the ID for
events to happen returns an error (descriptor invalid)
std::thread
and creating Workers
Would any of these qualify with the quoted warning? and invalidate the
descriptor table?
If not, how can I (easily) debug the reason why my descriptors go invalid?

Sharing a kqueue between threads of a process works. Are the workers
created using rfork without RFFDG as suggested in the manpage? I've
never had reason to do this, but a quick test seems to indicate that it
works as advertised. A normal fork closes the kqueue file descriptor. If
you suspect that's what's happening, you might run "procstat -f {worker
pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
column appear (if not, they were presumably closed on the fork).

Eric

[Slawa Olhovchenkov]

Post by Eric Badger
Sharing a kqueue between threads of a process works. Are the workers

Can you comment PR207261?

Post by Eric Badger
created using rfork without RFFDG as suggested in the manpage? I've
never had reason to do this, but a quick test seems to indicate that it
works as advertised. A normal fork closes the kqueue file descriptor. If
you suspect that's what's happening, you might run "procstat -f {worker
pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
column appear (if not, they were presumably closed on the fork).
Eric

[Eric Badger]

Post by Slawa Olhovchenkov

Post by Eric Badger
Sharing a kqueue between threads of a process works. Are the workers

Can you comment PR207261?

Hmm, I don't think I have the netmap expertise to comment.

Eric

[Adrian Chadd]

netmap is a special snowflake case, as the last time I checked a whole
bunch of assumptions are made about the process/thread contexts doing
the work.



-adrian

Post by Eric Badger

Post by Slawa Olhovchenkov

Post by Eric Badger
Sharing a kqueue between threads of a process works. Are the workers

Can you comment PR207261?

Hmm, I don't think I have the netmap expertise to comment.
Eric

[Willem Jan Withagen]

Post by Eric Badger

Post by Willem Jan Withagen
Hi,
Ceph uses a lot of threading, and for any part of it
communication it uses epoll(), which FreeBSD does not use. For
that there was already a EvenKqueue implementation.
But I think I'm now running into: The kqueue() system call
creates a new kernel event queue and returns a descriptor. The
queue is not inherited by a child created with fork(2). However,
if rfork(2) is called without the RFFDG flag, then the descriptor
table is shared, which will allow sharing of the kqueue between
two processes.
Kqueue descriptors are created and events are added, but then
the process starts other threads and expects the kqueue-id to be
valid there.
However adding more events returns an error, also waiting on the
ID for events to happen returns an error (descriptor invalid)
Threading is done with 2 different constructions: std::thread and
creating Workers
Would any of these qualify with the quoted warning? and
invalidate the descriptor table?
If not, how can I (easily) debug the reason why my descriptors go invalid?

Sharing a kqueue between threads of a process works. Are the
workers created using rfork without RFFDG as suggested in the
manpage? I've never had reason to do this, but a quick test seems
to indicate that it works as advertised. A normal fork closes the
kqueue file descriptor. If you suspect that's what's happening, you
might run "procstat -f {worker pid}" to see if file descriptors
with a "k" (kqueue) in the "T" (type) column appear (if not, they
were presumably closed on the fork).

Yup, I know what the man page writes. :) But now the trick question
is: how does the C++ worker/thread library function....

Some of the code actually does: std:move on a lambda-function, which
should instantiate a complete fresh environment. And that would
certainly qualify for expected closure of the kq descriptors.
OTOH it keeps regular file descriptors available in the thread.

I guess I'll have to start testing with a few simple C++ programs to
get things clear.

--WjW