Quick Index
|
|
This paper was originally published in the Proceedings
for The First World Conference on System Administration,
Networking andrSecurity, Washington DC, 1992.
Three Useful Tools
for
the UNIX System Administrator
Linda Bissum
ABSTRACT
It seems to be a common problem that system
administrators tend to re–invent the wheels in form
of many of the small tools required for the daily
administration. Part of this problem is that many tools are
not well distributed, and also that many of the tools which
are written are done as ad–hoc shell scripts, which
must be modified on an ongoing basis. This paper describes
three tools, police , which removes files,
cleanlog , a log maintainer and free , a
file system monitor — all written as generalized
utilities. They should therefore be able to adapt to
changes in policy or environment without modification of
the program themselves.
Introduction
This paper describes three useful system administration
tools. The tools where all originally written in the Bourne
Shell, back in the mid–eighties, and have proved
themselves useful. However, as they where placed on an
increasing number of new platforms, it became increasingly
difficult to maintain a single source for all platforms.
Over the last year, they therefore all have been rewritten
in Perl(1). programming language. The rewritten utilities
are based on the lessons learned and are also utilizing the
more flexible programmability provided by Perl. The tools
which are covered in this paper is police , which
removes stale files, cleanlog , a log file
maintainer and free , a file system space
monitor.
The utility free was previously the most
difficult script to maintain, and it received several
improvements during the rewriting in Perl. Therefore, a
discussions of this transition will also be covered, as an
illustration of the differences between implementing such
tools in Bourne Shell and in Perl.
Police, a file removal daemon
Temporary and stale files can be a problem on systems
with limited disk space. For example, the emacs
editor and the ispell spell checker leaves
*.bak files which an be deleted automatically
after a period of time. In a similar manner, core
files can be immediatly deleted on most systems which has a
binary–only license.
Police is a generalized program to remove such
files. Which files should be removed is determined in a
external configuration table. This table enables the system
administrator to specify which files should be removed, and
how old they should be before removal.
This approach has strong advantages over ad–hoc
shell scripts as no modifications are necessary in the
program when changes in policy require files be added or
removed from the automatic deletion process.
However, as for all monitoring tools, which are actual
making alterations to the system, there is a danger that
files could be unintentionally be removed. The answer to
this problem is twofold. First, no such tool should ever be
employed on the systems, without a clear policy of which
files will be removed. Second, as for police , the
implementation should allow for efficient warnings before
the removal. The time period of warnings before removal, as
well as the time since last modification date is determined
by an external configuration table, as shown in figure
1.
Finally, all file removal can be stopped by the system
administrator by creating a lock file. When this is done,
e–mail will be generated as usual, but no file will
actually be removed. This is useful to ensure that the
regular expressions in configuration performs as
expected.
core:1:3
*.bak:1:3
*.i:1:3
Canceled.mail:5:7
dead.letter:5:7
Fig 1 – Configuration file for police
The first column in the configuration file is a regular
expression specifying the name of the file. In the first
line it is core . The second column shows how long
a time the file should be left unnoticed (one day). When
one day has passed, mail will be sent to the owner of the
file, warning him that his file soon will be deleted
automatically. When the number of days shown in column 3
(two days) are up, the file will be deleted unless its
owner has touched it (e.i changed the date of last
modification, which starts the game all over), or renamed
it to keep the file permanently. The second line is for the
backup files from emacs and ispell . Note
that regular expressions are allowed.
The main advantages of this utility over ad hoc
scripts is that the user is warned that the file is about
to be deleted. If necessary, the user can change the name
of the file, if it is needed in the future. Also, it is
very easy for the system administrator to add files, or
change any of the time periods, without any changes are
required in the police utility.
The disadvantage with this utility is that it takes a
long time to run. This is caused by the fact that it is
necessary to run the equivalent of a find command
from the root directory. The implementation is of caused
based on find2perl command. The perl script
generated by find2perl are then executed from an
open statement (in a manner similar to popen in C).
As the utility is intended to be run at night from
cron , this is a minimal problem unless the system
has a large number of disks. However, in that case it can
be assumed that disk space is less of a premium, obsoleting
the need for a utility like police .
Cleanlog, a Log Maintainer
Various UNIX utilities are writing log files. Some of
these utilities, like sendmail and
syslogd are ensuring that the logs are not growing
without bounds. Other utilities are not cleaning up in a
similar manner. The system administrator is expected, for
each log file, to invent a scheme to remove such log files
on a regular basis. However, if the information in those
log files is not kept for some time it can make recovery
after failure more difficult.
Cleanlog is generalized tool to maintain the
log files from one simple table, while giving the
administrator the ability to decide for each individual
file how many days they should be kept.
In principle, the system administrator specifies how
many days a log file should be kept, and cleanlog
will the keep the logs on a rotation basis (similar to what
is done by sendmail ). Each night, when
cleanlog is executed from cron , it will
remove the oldest log file, and pushing all other log files
one place ahead. For a seven day rotation scheme, it would
be the equivalent of:
rm logfile.7
mv logfile.6 logfile.7
mv logfile.5 logfile.6
. .
. .
mv logfile logfile.1
touch logfile
In praxis, cleanlog provides some additional
functionality, as can be seen from the cleanlog
configuration file in figure 2.
Configuration file for cleanlog
/usr/lib/news:log:3:news:news:600:
/usr/lib/news:errlog:7:news:news:600:
/usr/lib/nn:Log:3:news:news:600:
/usr/smail/log:log:7:root:whell:600:
Fig 2 – cleanlog configuration example
The six fields of this table are used in the following
manner: The first field is the directory where the log file
is located and the second field is the basename of the log
file. The third field is number of generations which should
be kept of the log file. The sixth field is the file
permission for the log file. The next two fields are the
owner and group which the log file must belong to.
Cleanlog will change the access permission on any
newly created log file. The last field (unused in the
example above), is intended for a shell command which must
be executed after the log files have been moved. This is
required in some versions of UNIX for a utility like
cron , which in some cases must be killed and then
restarted to ensure the log file change has taken effect.
This problem is caused when a utility is opening the log
files, and is keeping it open for the lifetime of the
process. In that case the utility will continue to write to
the same file, independent of any any file name change, or
even file removal. In all cases, the older log files will
be kept in a directory named OLDLOGS which will be
created in the same directory as the main log file.
Free, a File System Monitor
It is a common problem that file systems are running out
of free space. Unix has no way to report this prior to the
event, unless the system administrator does a manual disk
free check, using the df command, on a regular
basis (e.i. several times a day). This is no big deal, if
the system administrator is responsible for one or two
systems. However, if the number of systems becomes much
larger five or ten, it becomes impossible keep up.
Free is an example of solving this problem,
partly through automation, partly through delegation of
responsibility. The idea behind the file system monitor is
based on the fact that the system administrator, in most
cases, does not know which files can be deleted or archived
(with the obvious exceptions of file system like
root and /usr.) For most file systems,
the users will have a better knowledge of what can be
deleted or archived. The idea behind free is
therefore based on the concept that it is appropriate to
let the users look after their own file space. Sites which
are adopting this view, will of cause need to have written
policies stating that this is required of the users.
Procedure must also be in place describing the process to
request files placed on archive tapes, or to request a
larger file system.
However, the problem with this policy is, that many
users do not look after the file space, and many do not
even know how. This is where Free is providing a
useful service. free checks the amount of free
space for mounted file system. If it falls below a certain
level, it uses wall to broadcast a message to all
users currently logged on.
Free is intended to be run from cron
every hour, and will in this manner ensure that the users
will get a timely warning before a file system is filling
up.
If this still is happening, and if the users do not take
any action, a point of view could be that they have asked
for problems they are about to encounter. However, such an
attitude is neither very positive, nor will it get goodwill
from the user community. Therefore, the free disk
monitor has a second level of checks build in. If a file
systems reaches the first limit (the message level), and no
action is taken, the fall back position is that when the
second level (the panic level) is reached, the system
administrator is notified by e–mail.
In this manner, on a reasonably well configured and
administrate system, the system administrator will no
longer need to worry about the free space problem on a
daily basis. Most likely, the users will be able to take
whatever action is necessary themselves.
There are two problems with this approach, however.
First, no default will work well for all sizes of file
systems, and second, not all file systems should be
reported to the users. These problems have been resolved
through the use of an external configuration table, as
shown in figure 3.
/dev/sd0a:3607:3150
/dev/sd0g:95%:90%
/usr2:10%:5%
Fig 3 – Configuration File for Free
Free has builtin defaults, which can be set a
reasonable value for most systems (e.i. 10% warning level,
and a 5% panic level). However, for any file system, where
such values are not reasonable, the values can be set in
the configuration file. This file has three fields, where
the first is either the device name or the pathname of the
mount point, and the two following fields are the values
for the warning and panic levels of free space. The latter
can either be specified in blocks, on in a percentage of
the size of the file system. File systems where no warning
message should be send to the users, the warning level is
set to 0.
Free Implementation Details
It is a common misconception that shell scripts are
portable between UNIX systems. The syntax and semantics, at
least for the Bourne Shell, are reasonable constant.
However, many of the commands called from the shell script
will vary in both style and location, making the
portability difficult. One of the advantages of Perl over a
shell script is, that operations in the shell script which
are done by calling external utilities, in Perl mostly are
builtin functions. However, in the case of free
some of the essential operations are still done through
call of external utilities, leaving the problem of
portability unresolved. These are:
- The name of the host must be determined. Depending of
the version of UNIX, this is done through calling of
different utilities, typically either hostname
(BSD) or uuname (USG).
- A mail user agent must be called to deliver
e–mail. Less important, dependent on the mailer, it
may not be possible to specify a subject line.
- The core of the free utility is the
determination of the amount of occupied and available
disk space. The only truely portable solution, was to
call the df program. Unfortunately, the format
of the output of this program differs wildly between UNIX
versions, and even between similar versions from
different vendors.
$HostName = ( "/etc/hostname",
"/bin/hostname",
"/usr/bin/uuname —s"
);
sub $HostName {
unless ( defined $HostName ) {
foreach $Cmd ( @HostName ) {
chop( $HostName = `$Cmd` );
last if ( $? == 0 );
}
}
$HostName;
}
Fig. 4 – Perl subroutine HostName
The two first problems are solvable in a shell script by
calling each possible utility, until one is found and
executed. The same solution can be use in Perl, as shown
for hostname in Figure 4.
I have found no truly portable solution to solve the
last item. Therefore, the interface to the df
program has been left dependent on the architecture.
The free utility is assuming that an external
command, arch , exists, which will print the name
of the current architecture to standard output. This output
string is then used to make a lookup in a second
configuration table, to find the name of the interface file
which shall be used to execute the df command.
Figure 5 is showing this table.
sun4:bsd
sun3:bsd
bsdi:bsd
hpux:hpux
marc:bsd
svr4:svr4
sv3.2:usg
sco:sco
Fig 5 – Architecture conf. file for free
In this table, the first field contains the name of the
architecture, and the second is field contains the name of
the interface file.
To give the greatest possible flexibility, neither of
the configuration files are required. However, if the file
system table is missing, the defaults will be used for all
mounted file systems, and if the architecture table is
missing, the interface programs known internally to
free will be executed sequentially, until one is
giving a useful data, or the list is expired.
Comparison between Perl and Shell
Because Free has been implemented in both
Bourne Shell and Perl, it gives a basis of comparison
between the two implementation approaches. It has been my
experience that many programs which can only with
difficulties be implemented as a shell script are easily
implemented in Perl. The code for police and
cleanlog are smaller and less complex than the
older shell script. Free ended up with being
slightly bigger, however, much additional functionality was
added to resolve problems which existed previous.
ReferencesLarry Wall and Randal L. Schwartz.
Programming Perl. O'Reilly & Associates, Inc,
1990.
|
