DESCRIPTION
arclog.pl archives the log files monthly. It strips
off previous months' log records from the log file, and save them to
compressed archive files named logfile.yyyymm. It then
saves the hard disk space and prevents potential attacks on log
files.
Current, it supports Apache access log and Syslog log file
formats, and gzip and bzip2 compression methods.
CAUTION
- Archiving takes time. From version 2.0.0, it will copy
the source log file to a temporary working file and restart it first.
Then it can take its time working on that temporary working file.
This can maximum reduce the time occupying the source log file,
avoiding the need of foreign log file restarting program. However,
please note: (1) If you have a huge log file (several tenths of
MBs), it still sucks copying. You had better stop logging first,
archive the log file and restart logging, to avoid racing condition
in writing. If you archive the log file periodly, it shall not grow
too big. (2) If it stops in the middle of execution, it will leave
a temporary working file. The next time it runs, it will stop when
it sees that temporary working file at the first sight. You have to
process that file first in order to keep working. You can change it
to a different filename and archive it like an ordinary log file, to
solve this.
- You can optionally sort the records in the archived files by
time (and then by record order), to solve some problems on merged log
files. But BE CAUTIONED: Sorting eats huge memory and
CPU. If you work on large log files, it may kill the
whole system in one minute. I had made some simple steps to avoid
that, but I CAN'T GUARANTEE THAT IT WORKS. Whether it kills
depends on which CPU you have, how much memory you have
and how large your archived file is. Because of this, sorting is by
default turned off.
- Be careful on the Syslog log files: Syslog does not record the
year. arclog.pl uses Date::Parse to parse
the date, which smartly judges the year between this month and last
next month. For ex., if today is 2001-06-08, it will then judges the
year between 2001-06-30 back to 2000-07-01 if the year is missing. I
think this is smart enough. However, if you do have a Syslog log
file that has records older than one year, don't use
arclog.pl. It will destroy your log file.
- From version 1.5.0 you can specify "-" to
read from STDIN. But notice here: (1) You MUST
specify the output prefix if you want to read from STDIN,
since what it needs is an output pathname prefix, not an output file.
(2) If you read from a file, the records of this month will be
returned to that file, to be archived next month; but if you read from
STDIN, there's no place to return the records, so the
records of this month will be archived, too. (3) The answers of the
ask mode is obtained from STDIN, too.
Since you have only one STDIN, you cannot specify the
ask mode while reading from STDIN. It will
fall back to the fail mode in that case. (4) You cannot
read compressed log files from STDIN. This is not
implemented yet.
SYSTEM REQUIREMENT
- Perl, version 5.005 or above. This program uses the File::Temp
module, which requires some new features introduced from perl 5.005.
Perl is distributed with most Unix now. If not, or if you have an
older version of perl, you can download and install/upgrade it from
http://www.perl.com/.
If you are using MS-Win32, you can download and install from
http://www.activestate.com/.
- Required perl modules:
File::Spec::Functions,
File::Temp,
Date::Parse.
You can download and install them from the CPAN archieve:
http://search.cpan.org/.
- Optional perl modules:
Compress::Zlib,
File::MMagic.
If you want to install them, you can download and install them from
the CPAN archieve:
http://search.cpan.org/.
DOWNLOAD
arclog.pl's official website is at...
You can always download the newest version of
arclog.pl from...
imacat's PGP public key is at...
INSTALL
-
Extract the downloaded tar ball
% tar -xzf arclog.pl-x.xx.xx.tar.gz
-
Install using the autoconf configuration script
Although arclog.pl needs no configuration, it is
still good if you run the autoconf configuration script. It checks
your environment and tell you in advance if something is missing. It
also installs the man page and other documents, and supports
uninstall from your system.
Goto the extracted directory and run configure:
% ./configure
It will check your environment and create an appropriate
Makefile. Then you can run make:
% make
Now, su to root and install it:
% su
Password:
# make install
That's all. You can also try
% ./configure --help
for a list of available configuration options.
-
Manual set up and install
Copy the arclog.pl to your desired directory and
run it. That's all. ^_*'
-
Run
Run it and see if anything goes wrong. You can try:
% ./arclog.pl --help
for a list of available command line options. Try:
% perldoc arclog.pl
for a complete documentation.
OPTIONS
./arclog.pl [options] logfile... [output]
./arclog.pl [-h|-v]
- logfile
- The log file to be archived. Specify "-"
to read from STDIN. Multiple log files are supported.
gzip or bzip2 compressed files are
supported, too.
- output
- The prefix of the output files. The output files will be named as
output.yyyymm, ie: output.200101,
output.200102. If not specified, the default is the same
as the log file. You must specify this if you want to read from
STDIN. You cannot specify "-"
(STDIN), since this is only a name prefix, not the
output file.
- -c, --compress[=method]
- Specify the compression method for the archived files. Log files
usually have large number of simular lines. Compress them saves
you lots of disk spaces. (And this is why we want to
archive them.) Currently the following compression methods
are supported:
- g, gzip
- Compress with gzip. This is the default.
arclog.pl can use Compress::Zlib to
compress instead of calling gzip. This can be safer and
faster for not calling foreign binaries. But if
Compress::Zlib is not installed, it will try to use
gzip binary instead. If gzip binary is not
available, either, but gzip compression is required, the
program will fail.
- b, bzip2
- Compress with bzip2. Because
Compress::Bzip2 has still very limited bzip2 library
support, arclog.pl will always use bzip2
binary. If bzip2 binary is not available, but
bzip2 compression is required, the program will fail.
- n, none
- No compression at all. (Why? :p)
- --nocompress
- Don't compress the archived files. This is equal to
--compress=none.
- -s, --sort
- Sort the records by time (and then the record order). Sorting
eats huge memory and CPU, so it is disabled by default.
See the description above for a detailed illustration on sorting.
- --nosort
- Don't sort the records. This is the default.
- -o, --override[=mode]
- Whether we should overwrite the existing archived files.
Currently the following modes are supported:
- o, overwrite
- Overwrite existing target files. If --override was
specified without mode, it assumes that you want to
overwrite. You will lost these existing records. Use with care.
This is helpful if you are sure the master log file has the most
complete records.
- a, append
- Append the records to the existing target files. You may destroy
the log file completely by putting irrelevant entries altogether
accidently. Use with care. This is helpful if you append want to
merge 2 or more log files, for ex., 2 log files of different periods.
- i, ignore
- Ignore any existing target file, and discard all the records of
those months. You will lost these log records. Use with care. This
is helpful if you are supplying log records for the missing months,
or if you are merging the log records in a complex manner.
- f, fail
- Stop processing whenever a target file exists, to prevent
destroying existing files by accident. This should be mostly wanted
when run from some automatic mechanism, like crontab. So, this is
the default if no terminal is found at STDIN.
- ask
- Ask you what to do when a target file exists. This should be
most wanted if you are running arclog.pl interactively.
So, this is the default if a terminal is found at STDIN.
The answers are read from STDIN. Since you have only one
STDIN, you cannot specify this mode if you want read the
log file from STDIN. In that case, it will fall back to
the fail mode. Also, if arclog.pl cannot
get its answer from STDIN, for ex., on a closed
STDIN like crontab, it will fall back to
fail mode.
- -k, --keep[=mode]
- What to keep in the source file. Current the following modes
are supported:
- a, all
- Keep the source file after records are archived. If
--keep was specified without mode, it
assumes that you want to keep all.
- r, restart
- Restart the source log file after records are archived.
- d, delete
- Delete the source log file after records are archived.
- t, this-month
- Archive and strip records of previous months off from the log
file. Keep the records of this month in the source log file, to be
archived next month. This is designed to be run from crontab
monthly, so this is the default.
- -d, --debug
- Show the detailed debugging messages.
- -q, --quiet
- Shihhhhhh. Only yell when errors.
- -h, --help
- Display the help message and exit.
- -v, --version
- Output version information and exit.
COPYRIGHT
Copyright © 2001-2002 imacat. All rights reserved.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General
Public License as published by the Free Software Foundation;
either version 2 of the License, or (at your option) any later
version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty
of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE. See the GNU General Public
License for more details.
You should have received a copy of the GNU
General Public License along with this program; if not, write
to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA
02139, USA.
RELEASE NOTES
Please read to the NEWS file, for newly added
features, bug fixes, etc. Please refer the ChangeLog
file for the revision history.
BUGS
arclog.pl has a mailing list at SourceForge:
arclog-users@lists.sourceforge.net. It is for
arclog.pl's users to discuss and report problems. Its
web page is at http://lists.sourceforge.net/lists/listinfo/arclog-users.
If you have any problem or question on arclog.pl, please
go to this page, join the list, and send your questions on this list.
Thank you.