Discussion:
Linuxisms in s6
(too old to reply)
Jonathan de Boyne Pollard
2016-08-25 04:55:32 UTC
Permalink
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you
get around the linuxisms, and the lack of sensible time code in it
(its calculations for daemon run duration is based on system time, not
wall clock, so if your box boots jan 1, 1970 then gets NTP, things
are.. hilarious), and some of the arcane bits to get logging working
right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
Jan Bramkamp
2016-08-25 09:57:01 UTC
Permalink
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you
get around the linuxisms, and the lack of sensible time code in it
(its calculations for daemon run duration is based on system time, not
wall clock, so if your box boots jan 1, 1970 then gets NTP, things
are.. hilarious), and some of the arcane bits to get logging working
right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
The skalibs library used by s6 to calculate the deadlines should use
clock_gettime(CLOCK_MONOTONIC) on FreeBSD and as such shouldn't be
affected by changes to the wall clock.

I'm currently working on a FreeBSD only potential init replacement as
well just without the mandatory per service supervisor process. The new
kqueue EVFILT_PROCDESC filter type in FreeBSD 11 combined with pdfork()
should make it really easy to deal child processes in a single unified
kevent loop. Forking services could still be handled by a supervisor
using procctl(PROC_REAP_ACQUIRE).

At the moment I'm fighting with some corner cases in the file descriptor
passing code and redesigning the API to work without access to a
writable file system. My last API required a writable file system
because FreeBSD doesn't support listen()ing on unbound unix domain
seqpacket sockets and I don't want to require something like the Linux
/run tmpfs. Instead my new API uses socketpair() to create a connected
pair of anonymous unix domain sockets for each supervised process. Next
I have to find out if fexecve() works at least for fd 0, 1 and 2 without
a mounted fdescfs.

I want to implement the following features in a single process capable
of running as PID 1:
- Track service dependencies (want, require, bind, conflict)
- Store, Retrieve and close file descriptors.
- Spawn and supervise processes in a well defined environment.
- Reliable event notification with coalescing.
- Bootstrap the system with help from a default service.

With those features it should be able to wrap existing rc.d scripts
without resorting to polling.
Adrian Chadd
2016-08-25 15:46:35 UTC
Permalink
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.

It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)



-adrian
Lars Engels
2016-08-25 20:12:11 UTC
Permalink
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Adrian Chadd
2016-08-25 21:09:28 UTC
Permalink
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.



-adrian
Warner Losh
2016-08-25 21:13:44 UTC
Permalink
Post by Adrian Chadd
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.

What devd doesn't do is launchd / job control. That would be a whole
new kettle of fish for it, and one for which it may be ill suited. Though
viewed in the right way, it's all just a series of notifications: This service
is up, this is down, user wants to start this other one, etc, so maybe
it wouldn't so bad...

Warner
Adrian Chadd
2016-08-25 21:17:47 UTC
Permalink
Post by Warner Losh
Post by Adrian Chadd
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.
Right. I don't want to reinvent the wheel if it can be avoided.

But there are other things that want to produce and consume events.
eg, openvpn bringing up a VPN triggering possible ipfw rule changes.
Or openvpn coming down triggering other ipfw rule changes.
Post by Warner Losh
What devd doesn't do is launchd / job control. That would be a whole
new kettle of fish for it, and one for which it may be ill suited. Though
viewed in the right way, it's all just a series of notifications: This service
is up, this is down, user wants to start this other one, etc, so maybe
it wouldn't so bad...
Well, ideally the jobd would sit on the message bus and take commands
to do things. Like dbus and udevd did in linux, before systemd
steamrolled over them. But then if I suggest we need a message bus
daemon up and going so arbitrary system pieces could talk to other
system pieces, I'll likely be shouted at.

But not by jkh. He'd likely be "YOURE ONLY JUST GETTING AROUND TO THIS
NOW?" and laugh a lot.

(jkh - please come to the next bafug so we can talk shop..)


-a
Jan Bramkamp
2016-08-26 10:45:21 UTC
Permalink
Post by Adrian Chadd
Post by Warner Losh
Post by Adrian Chadd
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.
Right. I don't want to reinvent the wheel if it can be avoided.
Afaik devd is limited handling events reported by the kernel on
/etc/devctl. There is no way to inject events into arbitrary events from
userspace into devd (no ptrace hacks don't count).
Post by Adrian Chadd
But there are other things that want to produce and consume events.
eg, openvpn bringing up a VPN triggering possible ipfw rule changes.
Or openvpn coming down triggering other ipfw rule changes.
FreeBSD offers several IPC APIs but non of them can implemented reliable
multicast as this would require an unbounded journal in stable memory.
For most use-cases reliable notification of the current state is enough.
Instead of reliably multicasting each message to each recipient just
send each observer the latest state of each observed value e.g. in your
OpenVPN example the IPFW wrapper doesn't care how many time the tunnel
flapped. The user just wants the right firewall configuration for his
current network environment. He doesn't want to replay every change on
the way.

In macOS has notifyd offers this service. The optimizations in notifyd
make it too large to just import its services into an init process, but
having a simpler reliable notification mechanism early would be useful.
Post by Adrian Chadd
Post by Warner Losh
What devd doesn't do is launchd / job control. That would be a whole
new kettle of fish for it, and one for which it may be ill suited. Though
viewed in the right way, it's all just a series of notifications: This service
is up, this is down, user wants to start this other one, etc, so maybe
it wouldn't so bad...
Well, ideally the jobd would sit on the message bus and take commands
to do things. Like dbus and udevd did in linux, before systemd
steamrolled over them. But then if I suggest we need a message bus
daemon up and going so arbitrary system pieces could talk to other
system pieces, I'll likely be shouted at.
But not by jkh. He'd likely be "YOURE ONLY JUST GETTING AROUND TO THIS
NOW?" and laugh a lot.
(jkh - please come to the next bafug so we can talk shop..)
Process spawning and supervision should be separate from the policy
engine(s) as the process supervision graph should be a tree rooted in
pid 1, but the user might want to run multiple rule/policy engines. An
init process should just offer the required mechanisms and nothing more.
Convenient policies can be implemented on top of those mechanisms.

For my own init system I'm still not sure if the init process should
track services and their dependencies at all or just processes and keep
the concept of services and dependencies in a service management process
instead of the init process.
Warner Losh
2016-08-26 14:12:12 UTC
Permalink
Post by Adrian Chadd
Post by Warner Losh
Post by Adrian Chadd
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.
Right. I don't want to reinvent the wheel if it can be avoided.
Afaik devd is limited handling events reported by the kernel on /etc/devctl.
There is no way to inject events into arbitrary events from userspace into
devd (no ptrace hacks don't count).
It would be trivial to add though, especially since the author is easy to
work with :). It already fans out data to multiple listeners.

Warner
Jan Bramkamp
2016-08-26 15:55:18 UTC
Permalink
Post by Warner Losh
Post by Adrian Chadd
Post by Warner Losh
Post by Adrian Chadd
Post by Lars Engels
Post by Adrian Chadd
On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
We're using s6 at work, and it works out mostly ok. Mostly once you get
around the linuxisms, and the lack of sensible time code in it (its
calculations for daemon run duration is based on system time, not wall
clock, so if your box boots jan 1, 1970 then gets NTP, things are..
hilarious), and some of the arcane bits to get logging working right.
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.
It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)
Can't this be done with devd?
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.
Right. I don't want to reinvent the wheel if it can be avoided.
Afaik devd is limited handling events reported by the kernel on /etc/devctl.
There is no way to inject events into arbitrary events from userspace into
devd (no ptrace hacks don't count).
It would be trivial to add though, especially since the author is easy to
work with :). It already fans out data to multiple listeners.
In that case dear author I have two requests:

* Read messages from sequential packet sockets in addition to
/dev/devctl.

* Please offer a flag to keep devd from forking without spamming the
log with debugging messages.

The first would allow userspace tools to inject messages into devd and
the second would allow devd to run under a process supervisor without
ugly workarounds.

--
Jan Bramkamp
Jonathan de Boyne Pollard
2016-08-27 16:37:47 UTC
Permalink
Post by Adrian Chadd
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.
You're reinventing upstart. The lesson of upstart is that whilst the
event-driven paradigm looks like the bright shiny future, once one gets
down to the details it is a lot harder than it at first appears. I
strongly recommended learning about upstart, and especially learning the
problems that people hit with it, to anyone going down the same route.
The Debian systemd Hoo-Hah had some lengthy discussion of upstart.

(I regret not having bookmarked the discussion that I once came across,
where someone opined that xe preferred systemd to upstart because at a
Linux conference the systemd presentation had been exciting and had been
put forward as the wave of the future, where upstart had been presented
as old-school, traditional, and boring. Ironically, this person wasn't
aware that the designs are exactly the opposite of that. upstart has
the novel event-driven design where the system is configured with the
information that event A triggers programs P, Q, and R, and the system
starts by raising a "first event", that runs programs, that raise
further events, that run further programs. Whereas it is systemd that
has the conventional design, shared by Mewburn rc and others, of
starting from a goal, working through a dependency tree, and doing
topological sorts.)

The Debian people chose to improve a non-event-driven architecture
instead. It's a lesson to be learned from SMF, in fact. One can have a
lot more additional abstract targets, such as "/milestone/name-services"
and "/milestone/system-clock", and dependencies to and from them. The
world is not 2 to 4 run levels plus "DAEMON", "NETWORKING", and "$local-fs".

That said, something like this hypothetical "/milestone/system-clock" is
a milestone that would need to be reached *very* early on in the
bootstrap process. Fixing up the clock is something that both the nosh
system manager and systemd handle themselves directly, outwith of
service management. More on this in a moment.
Adrian Chadd
2016-08-28 01:58:29 UTC
Permalink
On 27 August 2016 at 09:37, Jonathan de Boyne Pollard
Post by Jonathan de Boyne Pollard
Post by Adrian Chadd
Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the date,
we can now do crypto services" doesn't happen there right now.
You're reinventing upstart. The lesson of upstart is that whilst the
event-driven paradigm looks like the bright shiny future, once one gets down
to the details it is a lot harder than it at first appears. I strongly
recommended learning about upstart, and especially learning the problems
that people hit with it, to anyone going down the same route. The Debian
systemd Hoo-Hah had some lengthy discussion of upstart.
Oh yeah, I'm aware of the differences between systemd and upstart.
Post by Jonathan de Boyne Pollard
(I regret not having bookmarked the discussion that I once came across,
where someone opined that xe preferred systemd to upstart because at a Linux
conference the systemd presentation had been exciting and had been put
forward as the wave of the future, where upstart had been presented as
old-school, traditional, and boring. Ironically, this person wasn't aware
that the designs are exactly the opposite of that. upstart has the novel
event-driven design where the system is configured with the information that
event A triggers programs P, Q, and R, and the system starts by raising a
"first event", that runs programs, that raise further events, that run
further programs. Whereas it is systemd that has the conventional design,
shared by Mewburn rc and others, of starting from a goal, working through a
dependency tree, and doing topological sorts.)
For some environments (servers, desktop environments, etc) where most
of the dynamicness comes from "which user is logged in" and "maybe I
don't have my network hardware plugged in until later", sure, I can
see that the dependency tree model works great. Your aim is some
grander set of checkpoints, like "What do I need to run basic network
services", "What do i need up to run a desktop login environment",
etc.

But for things like "I'm a vpn server, and I need to speak to a vpn
server to provide access to my vpn clients, oh and I have firewall
rules that need applying based on which dynamic interfaces come/go" it
still maps to an event driven mechanism. Sure you can map that event
driven mechanism to a set of targets, but those targets may be
per-interface. Like, when a vpn client interface comes up, I have a
set of things that need to happen that depend upon which client. Same
with wireless clients associating. I may hit some situations (eg above
a certain threshold of associated clients) where I trigger events such
as "clean up old clients", "look at migrating clients to other access
points", etc.

If I can do this with s6, then cool - please let me know how and I'll
re-consider it. But regardless of that, I also do need some
generalised dbus style mechanism so all the pieces of the system can
talk to the other pieces of the system without having to .. well, wrap
it all in 'service' style shell scripts and calling ifconfig wlanX
list sta periodically from everywhere.
Post by Jonathan de Boyne Pollard
The Debian people chose to improve a non-event-driven architecture instead.
It's a lesson to be learned from SMF, in fact. One can have a lot more
additional abstract targets, such as "/milestone/name-services" and
"/milestone/system-clock", and dependencies to and from them. The world is
not 2 to 4 run levels plus "DAEMON", "NETWORKING", and "$local-fs".
That said, something like this hypothetical "/milestone/system-clock" is a
milestone that would need to be reached *very* early on in the bootstrap
process. Fixing up the clock is something that both the nosh system manager
and systemd handle themselves directly, outwith of service management. More
on this in a moment.
So, this is where it gets exciting in some of these appliances.

Sometimes there's no 'date/time' RTC hardware. Sometimes, you have to
present some UI to the user so they can enter a date/time, and some
services need to run before that, but some services (notably ntpdate)
won't work. So we can't, say, hold all network services back until we
have valid date/time or a bunch of the UI infrastructure won't be
there.

I can't expect the whole system to stop waiting for a system-clock
time to be valid. In fact, i ended up adding some stuff to our
appliance images that store the current clock value in a file every 15
minutes - the rootfs is read-only, so i can't just boot up from /its/
concept of "last mounted", as that filesystem is not modifiable.
Trouble is, that gets read from the system during boot, after FSes
have been mounted, etc. It's all terrible.

Thanks,



-adrian

Jonathan de Boyne Pollard
2016-08-27 13:40:16 UTC
Permalink
What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have
Linuxisms, obviously. But what Linuxisms does s6 have?
We just had a bunch of fun trying to get it to build right, [...]
Such as what, specifically?
Jonathan de Boyne Pollard
2016-08-27 16:52:29 UTC
Permalink
[...] the uptime stuff really threw us.
It's unfair to lay such system time problems at s6's door. Systems whose
system clock jumps 46 years during system bootstrap don't get to blame
s6 for mad time gaps that appear in logs and service start time
records. There is a *lot* of the Unix and Linux worlds that depends
from time being right. It's not just s6 that is affected by such
things. You note crypto. There are a lot of other things as well that
have unstated, sometimes undocumented, and sometimes surprising
dependencies upon system time being current.

Here's one such.

For quite a while, Linux distributions had rather an odd problem at
bootstrap. They'd repeatedly fsck volumes at every bootstrap when they
need not have. And this didn't affect U.S. or U.K. people, which is in
part why it persisted for so long.

* https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/63175

* https://bugs.archlinux.org/task/17438

* http://lwn.net/Articles/264498/

The problem was that people were erroneously running their real-time
clocks in local time rather than UTC, and this triggered an odd hidden
dependency upon having the right time in the system clock in countries
where local time was in advance of UTC. The Linux method for handling
RTCs erroneously running in local time is for the system bootstrap to
make a special settimeofday() call that effectively tells the kernel
what the UTC offset is for the RTC hardware. This could happen *after*
the fsck of the root volume, however. So whilst that fsck was
happening, the kernel was assuming that UTC was the local time that it
had taken from the RTC and initialized its system clock with. In
effect, as soon as the special settimeofday() call was executed, the
system clock would jump backwards by one or more hours, to what UTC
actually was.

But the ext2/3/4 filesystem format has last checked/mounted/written
timestamps in its superblock. Part of the checking to see whether a
full fsck is needed at bootstrap is comparing them to the current time.
If they are in the future by hours or more, something is clearly wrong,
thinks fsck, and it runs the full check. At bootstrap, when the initial
fsck (of at least the root volume and sometimes other volumes as well)
is run, the system clock is not UTC yet. Comedy results.

Both systemd and the nosh system manager have to ensure that they do the
special settimeofday() system call before they start off service
management and thus run mount/fsck services, or indeed anything else
that might have a closet dependency from not stepping the system time by
hours partway through bootstrap. The nosh system-manager's manual page
has a whole section on this subject.

FreeBSD/PC-BSD has a mechanism for correctly reading a RTC that is
erroneously in local time. One sets up the RTC's offset from UTC in the
machdep.adjkerntz variable in /boot/loader.conf{,.local} and the system
clock never has to jump by hours during bootstrap. I've yet to
experience a FreeBSD/PC-BSD system where the installer actually
configures this, though.

Interestingly, FreeBSD/PC-BSD also has a fallback mechanism that uses
the latest volume mount timestamp that it can find as the initial system
time when no hardware clock device registers at bootstrap. Presumably
you have a clock device that registers but it is not battery-backed,
your volumes don't preserve (or reset) their mount timestamps, or you
are encountering the comedy situation where FreeBSD/PC-BSD is setting
the system clock to 1970-01-01 because the last time around it mounted
the filesystems before the clock was corrected.
Continue reading on narkive:
Loading...