tuxad blog

The daemontools softwaresuite are widely used for supervising processes. Unlike the common way of launching daemons or other processes in background, writing their PID to a file and watch for a process (actually any process) with this PID by crappy tools like monit the daemontools offer a direct supervision of processes with an immediate restart on a crash.

Using the traditional SysV-init it is most easy to launch the daemontools processes:

$ grep -B1 SV /etc/inittab
si::sysinit:/etc/rc.d/rc.sysinit
SV:12345:respawn:/command/svscanboot3

Note: svscanboot3 is a binary replacement of svscanboot. It is part of ngtx.

SysV-init process ID 1 is very small, approx. 700 KB RSS. With many manual optimizations it is possible to start a system / server as fast as with upstart or systemd.

Upstart offers event-based booting. A time ago it was dealed as a successor to SysV-init. To start daemontools as a upstart-service you must create a config file:

$ cat /etc/init/daemontools.conf 
description     "DJB daemontools"
start on filesystem
stop on runlevel [06]
respawn
exec /command/svscanboot3

Some latest Linux (actual "linux-only") distros switch to systemd. Systemd also offers event-based and parallel booting with linux-only features like cgroups or fanotify. Its features are very nice but its PID 1 RSS is 13 MB which is very big compared to SysV-init. For systemd you must also create a config file (and a symlink):

$ ls -l /etc/systemd/system/multi-user.target.wants/ \
  daemontools.service
lrwxrwxrwx. 1 root root 39 21. Dez 08:34
  /etc/systemd/system/multi-user.target.wants/ \
  daemontools.service -> /lib/systemd/system/ \
  daemontools.service

$ ls -l /lib/systemd/system/daemontools.service
-rw-r--r--. 1 root root 151 21. Dez 08:33
  /lib/systemd/system/daemontools.service

$ cat /lib/systemd/system/daemontools.service
[Unit]
Description=DJB daemontools
After=sysinit.target

[Service]
ExecStart=/command/svscanboot3
Restart=always

[Install]
WantedBy=multi-user.target

Filtering all queries of one user out of a MySQL Slow Query Log is a common task. Maatkit is a toolkit for analyzing log files and more. Here is an example run to filter all queries of user123 of a 707 MB big slow-query-log:

$ time mk-query-digest --filter \
 '($event->{user} || "") =~ m/user123/' \
 --print --no-report log-slow-queries.log >1
log-slow-queries.log:  15% 02:49 remain
log-slow-queries.log:  29% 02:26 remain
log-slow-queries.log:  44% 01:51 remain
log-slow-queries.log:  59% 01:20 remain
log-slow-queries.log:  76% 00:45 remain
log-slow-queries.log:  90% 00:18 remain

real    3m20.097s
user    0m16.616s
sys     3m2.592s

Well, 707 MB in more than 3 minutes? Simple filtering can also be done with awk, try it:

$ time awk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    0m58.844s
user    0m2.501s
sys     0m3.268s

Nice speed, but awk is a symlink to mawk, try gawk:

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    3m30.287s
user    0m29.934s
sys     2m58.800s

Ooops, but didn't we already know, how slow GNU-software can be? Try a different server:

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    2m4.899s
user    2m2.240s
sys     0m1.890s

This server runs more fast, but it lacks mawk, is virtual and it runs like the first server only 32-bit-linux. Try gawk and mawk on 64-bit-linux (same physical host):

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    1m49.348s
user    1m40.242s
sys     0m1.528s

$ time mawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >3

real    0m13.115s
user    0m1.000s
sys     0m2.392s

Conclusions:

• simple filter can be done faster with awk than with maatkit
• mawk does this filtering faster than gawk
• mawk and gawk run faster on 64-bit-linux