Written by J. Moellenkamp
on 
Reading time: 6 minutes
English   Solaris   Oracle

Less known Solaris features - logadm

The mighty root couldn’t sleep at night, so root walked around the castle. Deep down in the foundation of super users castle there was a strange room. It was filled with scrolls and some of the serving daemons of root filled this room with even more scrolls while root was standing there. So root looked in some of them: There were new ones, interesting ones. Then root took some old ones, blew away the dust and after a short look root thought “Damned … those scrolls are so old, there aren’t true anymore. Someone has to clean up this place”. So root spoke a mighty spell of infinite power and another daemon spawned from the ether: “You are the keeper of the scrolls. But don’t keep everything. Just the last ones.” And so it was done since the day of this sleepless night.

Housekeeping in your logfile directories

One of the regular task of any decent admin should be the housekeeping in the logfile directory. Logfiles are really useful, when something went wrong, but often they just filling the directories with data. Albeit sometimes it’s useful to have even very old logfiles, most of the times you just need the recent ones in direct access.

logadm

One of the big conundrums with solaris is the point that few people know the logadm tool. It’s available with Solaris since the release of Solaris 9. However it’s still one of the well-kept secrets of Solaris despite the fact, that the tool is well documented and already in use Solaris. I’m often wondering what users of Solaris thing, where this .0-files were created ;)

Capabilities

For all the usual task surrounding logfile handling you could use the command logadm . It’s a really capable tool:

</noautobr>

How it works

logadm is the tool to configure it as well it’s the tool that do the log rotation. To automatically rotate the logs, logadm is executed by cron once a day. Let’s look into the crontab of the root user.

jmoekamp@hivemind:/var/squid/logs# crontab -l
[... CDDL Header omitted ...]
#
10 3 &#42; &#42; &#42; /usr/sbin/logadm
15 3 &#42; &#42; 0 [ -x /usr/lib/fs/nfs/nfsfind ] && /usr/lib/fs/nfs/nfsfind
30 3 &#42; &#42; &#42; [ -x /usr/lib/gss/gsscred_clean ] && /usr/lib/gss/gsscred_clean
30 0,9,12,18,21 &#42; &#42; &#42; /usr/lib/update-manager/update-refresh.sh

So as you may have already recognized, the logadm script is executed daily at 3:10 am in a Solaris default configuration. But how is the rotation itself done? Well, exactly as you would do it by typing in commands on the shell. logadm does the same job - just automatically. It generates a sequence of commands to rotate the logs and send them to a c shell for execution.

Practical side of logadm

Configuring it

Albeit it’s possible to invoke the rotation totally from command line each time, this isn’t a comfortable method to rotate logfiles. In almost every case you would configure the logadm facility in order to do the rotation task automatically. So how is this done?

logadm.conf

The daily run of logadm depends on a config file. When you don’t specify a config file, it’s /etc/logadm.conf per default. The configuration is rather short and just rotates the usual suspects. The following file is a little bit extended, as the default configuration file doesn’t use all important options:

<br />
jmoekamp@hivemind:~$ cat /etc/logadm.conf<br />
[... CDDL Header omitted ...]<br />
/var/log/syslog -C 8 -P 'Sat Feb 20 02:10:00 2010' -a 'kill -HUP `cat /var/run/syslog.pid`'<br />
/var/adm/messages -C 4 -P 'Sat Feb 20 02:10:00 2010' -a 'kill -HUP `cat /var/run/syslog.pid`'<br />
/var/cron/log -P 'Thu Dec 17 02:10:00 2009' -c -s 512k -t /var/cron/olog<br />
/var/lp/logs/lpsched -C 2 -N -t '$file.$N'<br />
/var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv /var/fm/fmd/errlog.0- $nfile' -N -s 2m<br />
/var/fm/fmd/fltlog -A 6m -M '/usr/sbin/fmadm -q rotate fltlog && mv /var/fm/fmd/fltlog.0- $nfile' -N -s 10m<br />
smf_logs -C 8 -s 1m /var/svc/log/&#42;.log<br />
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm -m 664 -o adm -p never<br />
/var/log/pool/poold -N -a 'pkill -HUP poold; true' -s 512k<br />
/var/squid/logs/access.log -P 'Tue Feb 23 06:26:23 2010'  -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n' -z 1

You can edit this file directly, but it’s preferred to change it with the logadm command itself. Let’s dissect some lines of this configuration.

Standard log rotation - introducing -C,-P and -a 

[...]<br />
/var/log/syslog -C 8 -P 'Sat Feb 20 02:10:00 2010' -a 'kill -HUP `cat /var/run/syslog.pid`'<br />
[...]

This line is responsible for rotating /var/log/syslog. The -C 8 specifies, that the logadm should hold 8 old versions before it expires (read: deletes) old logfiles. With the -a 'kill -HUP `cat /var/run/syslog.pid` option the syslog gets an HUP signal after the log rotation. The syslogd needs this to recreate the logfile and to restart logging. The -P isn’t there in the pristine version of the file. Those options will are inserted when you run the logadm command. With this option, you tell logadm when when the last rotation was done. logadm inserts it each time it rotates a logfile. It’s important to know, that this time is GMT time, so don’t wonder about the offset to the local time shown with your logfiles. In this statement you don’t find a configuration of the time or size that leads to a log rotation. In this case the default values “1 byte and 1 week” are kicking in. So /var/log/syslog is rotated when the last rotation was at least one week in the past {\emph and} the file is at least one Byte in size.

Explicit definition of a rotation time span - introducing -p

With -p you can control the period between log rotations. For example 1d specifies that you rotate the log files on a daily schedule.

[...]<br />
/var/squid/logs/access.log -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n'<br />
[...]

So this logfiles is rotated every day by the logadm execution initiated by the entryin the crontab.

A template for the name of the rotated log - introducing -t

-t specifies the way, how the names for logfiles are created by logadm. The template for the new name of the logfile isn’t just a fixed string, you can use several variables to control the name of the file.

[...]<br />
/var/squid/logs/access.log -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n'<br />
[...]

This is a relatively simple example. $n is just the version number of the file, starting with 0. Thus a configuration would lead to a filename like this one:

<br />
-rw-r----- 1 webservd webservd 652486 2010-02-22 16:36 /var/squid/logs/access.log.0<br />

$n is just one possible variable available for use in the templates. The logadm man page specifies further possible variables.

Explicit definition of maximum logfile size - introducing -s

Sometimes you want to set your current logfile a limit based on file size and not based on a time interval. The -s option allows you to do so:

[...]<br />
/var/cron/log -P 'Thu Dec 17 02:10:00 2009' -c -s 512k -t /var/cron/olog<br />
[...]

With this option logadm will rotate the logfile as soon it’s 512k or larger at the time logadm runs.

Specifying both: Space and time

When you specify a maximum logfile size (-s) as well as a maximum logfile period (-p), both conditions are connected with an AND. So the default “1 byte and 1 week” can be translated: Rotate when the logfile has a size of at least 1 byte AND it’s one week old, thus a week old zero-length logfile is not rotated by the default configuration.

Copy and truncate instead of moving - introducing -c

Rotating logfiles isn’t unproblematic. Some application don’t like it if you simply move the file away. They may use the rotated log instead of a new one, or they simply don’t create a new logfile. Thus you have to restart the service or send a signal. There is an alternative. It’s called truncating.

/var/cron/log -P 'Thu Dec 17 02:10:00 2009' -c -s 512k -t /var/cron/olog<br />
[...]

The -c option forces logadm to use cp instead of mv to rotate the log. To get a fresh start in the new log /dev/null is copied to the current logfile effectively make a 0 byte file out out it. This circumvents the need for a restart of the application to restart logging.

Compress after rotation - introducing -z

Due to their structure, logfiles can yield an excellent compression ratio. So it’s sensible to compress them after rotation. You can configure this with the -z session. However often it’s somewhat unpractical to work with compressed files, thus the parameter after -z forces logadm not to compress the stated number of the most recent log files. By doing so you have the recent logfiles available for processing without decompressing, without sacrifying space by leaving all logfiles uncompressed.

[...]<br />
/var/squid/logs/cache.log -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n'  -z 1<br />
[...]

-z 1 configures logadm to leave the most recent rotated logfile untouched and compresses the next one. This results to the following files:

jmoekamp@hivemind:/var/squid/logs$ ls -l /var/squid/logs/access&#42;
-rw-r----- 1 webservd webservd      0 2010-02-23 07:26 /var/squid/logs/access.log
-rw-r----- 1 webservd webservd 652486 2010-02-22 16:36 /var/squid/logs/access.log.0
-rw-r----- 1 webservd webservd  39411 2010-02-22 09:22 /var/squid/logs/access.log.1.gz</pre</code></blockquote>

<h4>A log rotation that is never executed automatically - introducing  <code>-p never</code></h4>At the first moment, this line looks really strange.
<!-- Migration Rule 1 --> 

<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext">[...]
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm -m 664 -o adm -p never
[...]</code></pre></figure>

What is the sense of a configuration line that is never used automatically in a log rotation script. Well, the fact that you don't use it automatically, doesn't mean that you can't use it manually from the command line  or that other scripts can't trigger the logadm functionality for log rotation instead of implementing their own.

The config line printed above is such a line enabling the use of logadm log rotation in other scripts. When you look into the script <code>/usr/lib/acct/turnacct</code>, you will see a line triggering logadm with <code>-p now</code>:
<!-- Migration Rule 1 --> 

<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext">&lt;pre&gt;[...]
switch)
        pfexec /usr/sbin/logadm -p now /var/adm/pacct
        if test ! -r pacct
        [...]</code></pre></figure>


The <code>turnacct</code> script triggers the log rotation. Due to the <code>-p never</code> in the <code>logadm</code> it's the only way to trigger the rotation. I will explain later on what happens due to this command.
<h4>Wildcards</h4>
The man page of logadm gives a good example for the usage of wildcard and "regular expressions":<br />

<!-- Migration Rule 4 --> 

<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext">logadm -w apache -p 1m -C 24\
            -t '/var/apache/old-logs/$basename.%Y-%m'\
            -a '/usr/apache/bin/apachectl graceful'\
            '/var/apache/logs/&amp;#42;{access,error}_log'</code></pre></figure>


With this statement you rotate all log files ending on <code>access_log</code> or <code>error_log</code> in your apache log directory. At the beginning of the line you may have recognized the <code>apache</code> name. This is a shorthand for this configuration. So you can simply rotate all logfiles with <code>logadm -p now apache</code> instead of the longer name of the directory. Additionally it's important that the command specified  by <code>-a</code> is executed once after all matching logfiles were rotated and not for each one. 
<h4>More options</h4>
There are several other option controlling the behaviour of a rotation. I will describe those options by short examples.
<ul>
<li><b><code>-A 1y</code>:</b><br> With <code>-A</code> is an additional way to control the expiration of a rotated logfile. For example, this example forces <code>logadm</code> to expire rotated logfiles when they are older than a year.</li>
<li><b><code>-S 100m'</code>:</b><br> Perhaps you don't want to specify a expiration limit by age, but by the size all the rotated logfiles takes. You can do so by using the <code>-S</code> statement. This example will expire rotated logfiles when they take more than 100 Megabyte.</li>
<li><b><code>-E 'mv $nfile /longtermstorage'</code>:</b><br> This option controls what happens when a rotated logfiles expires. The standard behaviour is a simple deletion. This example moves the expired file to a long term storage, for example an SamFS filesystem mounted via NFS on a different server. </li>
<li><b><code>-b 'svcadm disable nonbehavingapp' -a 'svcadm enable nonbehavingapp'</code>:</b><br> There isn't just an option to execute a command after log rotation with <code>-a</code>, there is one for executing commands before log rotation as well. You use the <code>-b</code> option to specify this account This is really useful to stop and start a non well behaving application for log rotation.</li>
<li><b><code>-R 'loganalyser.pl $file'</code>:</b><br> the command specified with this option is executed after each log rotation. At first this sounds redundant to <code>-a</code>  but there is an important difference. <code>-a</code> is executed once, even when the line in <code>logadm.conf</code> matched several logfiles. The command <code>-R</code> is executed for each rotated log. So <code>-a</code> is more for signaling a process, that it should start a new log, <code>-R</code> is more useful for log processing. When you use <code>-R</code> for signaling, the process would get a signal for each logfile you rotate.</li>
<li><b><code>-M 'mv $file $nfile</code>:</b><br> With <code>-M</code> you can specify your own command for rotation. This  example  is the default for a <code>logadm</code> run.</li>
</ul>
<h3>Control options</h3>
So far i just mentioned the options needed for configuration. But there are some additional options to control what 
<h4>Changing <code>logadm.conf</code> with <code>logadm</code></h4>
Okay, how do you put the commands to the <code>logadm.conf</code> file. You can edit the file directly , it will work. But <code>logadm</code> can be used to add and delete statements to the config file as well. When you add a line to the <code>logadm.conf</code> file, prepend the line with a <code>-w</code><br />
<blockquote><code>jmoekamp@hivemind:/var/squid/logs# logadm -w /var/squid/logs/cache.log -C 8 -c -p 1d -t '/var/squid/logs/cache.log.$n' -z 1<br />

You can check the addition with the logadm -V command. -V triggers the validation of the file.Thus it’s sensible to run this command after a direct addition to the file as well.

jmoekamp@hivemind:/var/squid/logs# logadm -V<br />
[...]<br />
/var/squid/logs/cache.log -C 8 -c -p 1d -t '/var/squid/logs/cache.log.$n' -z 1

Okay, we’ve worked a while with this log rotation configuration. You will notice that, that the configuration line has changed.

jmoekamp@hivemind:/var/squid/logs# logadm -V<br />
/var/log/syslog -C 8 -P 'Sat Feb 20 02:10:00 2010' -a 'kill -HUP `cat /var/run/syslog.pid`'<br />
[...]<br />
/var/squid/logs/access.log -P 'Tue Feb 23 19:19:27 2010' -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n' -z 1<br />
/var/squid/logs/cache.log -C 8 -P 'Tue Feb 23 19:59:20 2010' -c -p 1d -t '/var/squid/logs/cache.log.$n' -z 1<br />

Now you want to get rid of this configuration statement. This is done with the -R option in conjunction with the name of the logfile.

<br />
jmoekamp@hivemind:/var/squid/logs# logadm -r /var/squid/logs/cache.log<br />
jmoekamp@hivemind:/var/squid/logs# logadm -V<br />
/var/log/syslog -C 8 -P 'Sat Feb 20 02:10:00 2010' -a 'kill -HUP `cat /var/run/syslog.pid`'<br />
[...]<br />
/var/squid/logs/access.log -P 'Tue Feb 23 19:19:27 2010' -C 8 -c -p 1d -t '/var/squid/logs/access.log.$n' -z 1<br />
jmoekamp@hivemind:/var/squid/logs#

It’s gone …. the /var/squid/logs/cache.log will not be rotated the next time logadm is executed by cron or at the command line.

Forcing an immediate rotation

Sometimes you want to execute an immidiate log rotation. For example because you want to restart a service with a fresh log to ease analysis. -p now executes the log rotation right in the moment you start logadm

jmoekamp@hivemind:/var/squid/logs# date
Dienstag, 23. Februar 2010, 20:48:25 Uhr CET
jmoekamp@hivemind:/var/squid/logs# logadm -p now squid_cachelog
jmoekamp@hivemind:/var/squid/logs# ls -l cache.log&#42;
-rw-r----- 1 webservd webservd      0 2010-02-23 20:48 cache.log
-rw-r----- 1 webservd webservd 636799 2010-02-23 20:41 cache.log.0
jmoekamp@hivemind:/var/squid/logs#

With -p now you can even execute log rotations that are never executed automatically.

What happens under the hood?

As i wrote at the beginning, logadm works by sending a lot of commands to a c-shell. The logadm has an interesting mode of operation. When you use the -v option the tools shows the commands logadm uses to rotate the logfiles.

jmoekamp@hivemind:/var/squid/logs#  logadm -p now -v squid_access<br />
# loading /etc/logadm.conf<br />
# processing logname: squid_access<br />
mkdir -p /var/squid/logs # verify directory exists<br />
cp -fp /var/squid/logs/access.log.2.gz /var/squid/logs/access.log.3.gz # rotate log file via copy (-c flag)<br />
cp -f /dev/null /var/squid/logs/access.log.2.gz # truncate log file (-c flag)<br />
mkdir -p /var/squid/logs # verify directory exists<br />
cp -fp /var/squid/logs/access.log.1.gz /var/squid/logs/access.log.2.gz # rotate log file via copy (-c flag)<br />
cp -f /dev/null /var/squid/logs/access.log.1.gz # truncate log file (-c flag)<br />
mkdir -p /var/squid/logs # verify directory exists<br />
cp -fp /var/squid/logs/access.log.0 /var/squid/logs/access.log.1 # rotate log file via copy (-c flag)<br />
cp -f /dev/null /var/squid/logs/access.log.0 # truncate log file (-c flag)<br />
mkdir -p /var/squid/logs # verify directory exists<br />
cp -fp /var/squid/logs/access.log /var/squid/logs/access.log.0 # rotate log file via copy (-c flag)<br />
cp -f /dev/null /var/squid/logs/access.log # truncate log file (-c flag)<br />
touch /var/squid/logs/access.log<br />
chown 80:80 /var/squid/logs/access.log<br />
chmod 640 /var/squid/logs/access.log<br />
#     recording rotation date Tue Feb 23 18:15:44 2010 for /var/squid/logs/access.log<br />
gzip -f /var/squid/logs/access.log.1 # compress old log (-z flag)<br />
# writing changes to /etc/logadm.conf

As you may have recognized pretty much the same as you would do manually.

Tips and Tricks

Multiple instances of logadm

It’s perfectly possible to have more than one configuration file. You can choose the config file with -f. You just adds an additional entry in the cron table that specifies a different logfile.A colleague of me use this “feature” to have an independent log rotation config file for SamFS, so he don’t have to edit the file supplied by the distribution.

Conclusion

Albeit it’s just a really small tool, the possibilities of logadm to help you with your logfiles are vast and infinite. I’m really wondering why many people don’t know about it. I hope i changed that at least a little bit by writing this tutorial.

Do you want to learn more?

man pages
docs.sun.com - logadm(1M)
docs.sun.com - logadm.conf(4)