Site (RSS, Atom)
Weblog status
Total entries: 50
Last entry: 2016-10-17 22:22:03
Last updated: 2016-10-18 07:57:57
powered by vim, bash, cat, grep, sed, and nb 3.4.2

December 2011 Archives

2011-12-31 18:37:32

DJB daemontools with upstart or systemd

The daemontools softwaresuite are widely used for supervising processes. Unlike the common way of launching daemons or other processes in background, writing their PID to a file and watch for a process (actually any process) with this PID by crappy tools like monit the daemontools offer a direct supervision of processes with an immediate restart on a crash.

Using the traditional SysV-init it is most easy to launch the daemontools processes:

$ grep -B1 SV /etc/inittab

Note: svscanboot3 is a binary replacement of svscanboot. It is part of ngtx.

SysV-init process ID 1 is very small, approx. 700 KB RSS. With many manual optimizations it is possible to start a system / server as fast as with upstart or systemd.

Upstart offers event-based booting. A time ago it was dealed as a successor to SysV-init. To start daemontools as a upstart-service you must create a config file:

$ cat /etc/init/daemontools.conf 
description     "DJB daemontools"
start on filesystem
stop on runlevel [06]
exec /command/svscanboot3

Some latest Linux (actual "linux-only") distros switch to systemd. Systemd also offers event-based and parallel booting with linux-only features like cgroups or fanotify. Its features are very nice but its PID 1 RSS is 13 MB which is very big compared to SysV-init. For systemd you must also create a config file (and a symlink):

$ ls -l /etc/systemd/system/ \
lrwxrwxrwx. 1 root root 39 21. Dez 08:34
  /etc/systemd/system/ \
  daemontools.service -> /lib/systemd/system/ \

$ ls -l /lib/systemd/system/daemontools.service
-rw-r--r--. 1 root root 151 21. Dez 08:33

$ cat /lib/systemd/system/daemontools.service
Description=DJB daemontools



Posted by Frank W. Bergmann | Permanent link | File under: redhat, fedora, shell

2011-12-03 20:53:11

MySQL Slow Query Log Filter

Filtering all queries of one user out of a MySQL Slow Query Log is a common task. Maatkit is a toolkit for analyzing log files and more. Here is an example run to filter all queries of user123 of a 707 MB big slow-query-log:

$ time mk-query-digest --filter \
 '($event->{user} || "") =~ m/user123/' \
 --print --no-report log-slow-queries.log >1
log-slow-queries.log:  15% 02:49 remain
log-slow-queries.log:  29% 02:26 remain
log-slow-queries.log:  44% 01:51 remain
log-slow-queries.log:  59% 01:20 remain
log-slow-queries.log:  76% 00:45 remain
log-slow-queries.log:  90% 00:18 remain

real    3m20.097s
user    0m16.616s
sys     3m2.592s

Well, 707 MB in more than 3 minutes? Simple filtering can also be done with awk, try it:

$ time awk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    0m58.844s
user    0m2.501s
sys     0m3.268s

Nice speed, but awk is a symlink to mawk, try gawk:

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    3m30.287s
user    0m29.934s
sys     2m58.800s

Ooops, but didn't we already know, how slow GNU-software can be? Try a different server:

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    2m4.899s
user    2m2.240s
sys     0m1.890s

This server runs more fast, but it lacks mawk, is virtual and it runs like the first server only 32-bit-linux. Try gawk and mawk on 64-bit-linux (same physical host):

$ time gawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >2

real    1m49.348s
user    1m40.242s
sys     0m1.528s

$ time mawk \
'/^# Time: /{t=1;s=$0;p=0}'\
'!/^# /{t=0}/^# User@Host: /{p=0}'\
'/^# User@Host: user123/{p=1};'\
'{if(p){if(t){print s;t=0;}print$0}}' \
 <log-slow-queries.log >3

real    0m13.115s
user    0m1.000s
sys     0m2.392s


  • simple filter can be done faster with awk than with maatkit
  • mawk does this filtering faster than gawk
  • mawk and gawk run faster on 64-bit-linux

Posted by Frank W. Bergmann | Permanent link | File under: mysql, logging, shell