djbdns logging to syslog instead of multilog

Recently I came across an issue in the djbdns service on a linux host that I am setting up. After resolving around 250 remote queries, tinydns just stopped responding with no error messages. The same happened with dnscachex. A reboot would give me another allowance of ~250 queries. I was not happy.

After chasing this bug for a ridiculous amount of time, which included a fair bit of strace, tcpdump, reboots, googling, scripting tests, comparing it with a working installation, etc, I got to the bottom of it.

To make a long story short, turns out that daemontools - a support package that monitors djbdns services - was firing up multilog which in turn was failing to run due to invalid permissions set in its log directory (multilog was executed under dnslog:nofiles and its log directory was owned by root:root, so there was little chance of multilog writing files there).

Logging in djbdns is implemented with a FIFO pipe between tinydns and the logger process - usually multilog. As multilog never really started, tinydns locked up after pumping enough data into the log pipe which apparently filled up its input buffer (guess I could find out exactly how large it is...). Changing ownership of the log directory to dnslog:nofiles fixed the problem.

drwxr-sr-x 2 dnslog nofiles 4096 Mar 19 03:18 main
-rwxr-xr-x 1 root   root      98 Mar 24 02:32 run
-rw-r--r-- 1 dnslog nofiles    0 Oct 14 17:59 status
drwx--S--- 2 root   root    4096 Mar 24 02:33 supervise

Then I took the chance to make a further tweak: in a Linux system with no storage, I was not interested in having djbdns writing log files locally. Having a syslog daemon available, configured to forward data to a remote server made the solution quite obvious. All I needed was to get djbdns to forward logs to the local syslog daemon.

This is surprisingly easy to do but not at all obvious. The logger process is started by daemontools via the wrapper script /service/tinydns/log/run.

#!/bin/sh
exec setuidgid dnslog multilog t ./main

Instead of multilog, logger can be used to pipe data to syslog like so:

#!/bin/sh
exec /usr/bin/logger -p local5.debug -t tinydns

I chose facility local 5, log level debug and program name tinydns to mark these log entries, but these parameters are user defined. See the logger man page for more information. The same technique can be applied to dnscachex.

On the remote syslog server, I segregate these logs with the following configuration rules [remember to rotate these log files!]:

$template HostDirFile_tinydns,"/var/log/%HOSTNAME%/tinydns.log"
$template HostDirFile_dnscache,"/var/log/%HOSTNAME%/dnscache.log"
# DNS services
if $syslogfacility-text == 'local5' and $programname == 'tinydns' then ?HostDirFile_tinydns
& ~
if $syslogfacility-text == 'local5' and $programname == 'dnscachex' then ?HostDirFile_dnscache
& ~

Job done.