arclog.pl v2.1.0

DESCRIPTION

arclog.pl archives the log files monthly. It strips off previous months' log records from the log file, and save them to compressed archive files named logfile.yyyymm. It then saves the hard disk space and prevents potential attacks on log files.

Current, it supports Apache access log and Syslog log file formats, and gzip and bzip2 compression methods.

CAUTION

Archiving takes time. From version 2.0.0, it will copy the source log file to a temporary working file and restart it first. Then it can take its time working on that temporary working file. This can maximum reduce the time occupying the source log file, avoiding the need of foreign log file restarting program. However, please note: (1) If you have a huge log file (several tenths of MBs), it still sucks copying. You had better stop logging first, archive the log file and restart logging, to avoid racing condition in writing. If you archive the log file periodly, it shall not grow too big. (2) If it stops in the middle of execution, it will leave a temporary working file. The next time it runs, it will stop when it sees that temporary working file at the first sight. You have to process that file first in order to keep working. You can change it to a different filename and archive it like an ordinary log file, to solve this.
You can optionally sort the records in the archived files by time (and then by record order), to solve some problems on merged log files. But BE CAUTIONED: Sorting eats huge memory and CPU. If you work on large log files, it may kill the whole system in one minute. I had made some simple steps to avoid that, but I CAN'T GUARANTEE THAT IT WORKS. Whether it kills depends on which CPU you have, how much memory you have and how large your archived file is. Because of this, sorting is by default turned off.
Be careful on the Syslog log files: Syslog does not record the year. arclog.pl uses Date::Parse to parse the date, which smartly judges the year between this month and last next month. For ex., if today is 2001-06-08, it will then judges the year between 2001-06-30 back to 2000-07-01 if the year is missing. I think this is smart enough. However, if you do have a Syslog log file that has records older than one year, don't use arclog.pl. It will destroy your log file.
From version 1.5.0 you can specify "-" to read from STDIN. But notice here: (1) You MUST specify the output prefix if you want to read from STDIN, since what it needs is an output pathname prefix, not an output file. (2) If you read from a file, the records of this month will be returned to that file, to be archived next month; but if you read from STDIN, there's no place to return the records, so the records of this month will be archived, too. (3) The answers of the ask mode is obtained from STDIN, too. Since you have only one STDIN, you cannot specify the ask mode while reading from STDIN. It will fall back to the fail mode in that case. (4) You cannot read compressed log files from STDIN. This is not implemented yet.

SYSTEM REQUIREMENT

Perl, version 5.005 or above. This program uses the File::Temp module, which requires some new features introduced from perl 5.005. Perl is distributed with most Unix now. If not, or if you have an older version of perl, you can download and install/upgrade it from http://www.perl.com/. If you are using MS-Win32, you can download and install from http://www.activestate.com/.
Required perl modules: File::Spec::Functions, File::Temp, Date::Parse. You can download and install them from the CPAN archieve: http://search.cpan.org/.
Optional perl modules: Compress::Zlib, File::MMagic. If you want to install them, you can download and install them from the CPAN archieve: http://search.cpan.org/.

DOWNLOAD

arclog.pl's official website is at...

SourceForge's home page: http://arclog.sourceforge.net/
SourceForge's project: http://sourceforge.net/projects/arclog/
Tavern IMACAT's home page: http://www.imacat.idv.tw/tech/arclog.pl.html

You can always download the newest version of arclog.pl from...

SourceForge: https://sourceforge.net/project/showfiles.php?group_id=62499
Tavern IMACAT's FTP: ftp://ftp.imacat.idv.tw/pub/arclog.pl/

imacat's PGP public key is at...

SourceForge: http://arclog.sourceforge.net/pgpkey.asc
Tavern IMACAT's: http://www.imacat.idv.tw/me/pgpkey.asc

INSTALL

Extract the downloaded tar ball
```
% tar -xzf arclog.pl-x.xx.xx.tar.gz
```
Install using the autoconf configuration script

Although arclog.pl needs no configuration, it is still good if you run the autoconf configuration script. It checks your environment and tell you in advance if something is missing. It also installs the man page and other documents, and supports uninstall from your system.

Goto the extracted directory and run configure:
```
% ./configure
```
It will check your environment and create an appropriate Makefile. Then you can run make:
```
% make
```
Now, su to root and install it:
```
% su
Password:
# make install
```
That's all. You can also try
```
% ./configure --help
```
for a list of available configuration options.
Manual set up and install

Copy the arclog.pl to your desired directory and run it. That's all. ^_*'
Run

Run it and see if anything goes wrong. You can try:
```
% ./arclog.pl --help
```
for a list of available command line options. Try:
```
% perldoc arclog.pl
```
for a complete documentation.

OPTIONS

./arclog.pl [options] logfile... [output]
./arclog.pl [-h|-v]

logfile

The log file to be archived. Specify "-" to read from STDIN. Multiple log files are supported. gzip or bzip2 compressed files are supported, too.

output

The prefix of the output files. The output files will be named as output.yyyymm, ie: output.200101, output.200102. If not specified, the default is the same as the log file. You must specify this if you want to read from STDIN. You cannot specify "-" (STDIN), since this is only a name prefix, not the output file.

-c, --compress[=method]

Specify the compression method for the archived files. Log files usually have large number of simular lines. Compress them saves you lots of disk spaces. (And this is why we want to archive them.) Currently the following compression methods are supported:

g, gzip: Compress with gzip. This is the default. arclog.pl can use Compress::Zlib to compress instead of calling gzip. This can be safer and faster for not calling foreign binaries. But if Compress::Zlib is not installed, it will try to use gzip binary instead. If gzip binary is not available, either, but gzip compression is required, the program will fail.
b, bzip2: Compress with bzip2. Because Compress::Bzip2 has still very limited bzip2 library support, arclog.pl will always use bzip2 binary. If bzip2 binary is not available, but bzip2 compression is required, the program will fail.
n, none: No compression at all. (Why? :p)

--nocompress

Don't compress the archived files. This is equal to --compress=none.

-s, --sort

Sort the records by time (and then the record order). Sorting eats huge memory and CPU, so it is disabled by default. See the description above for a detailed illustration on sorting.

--nosort

Don't sort the records. This is the default.

-o, --override[=mode]

Whether we should overwrite the existing archived files. Currently the following modes are supported:

o, overwrite: Overwrite existing target files. If --override was specified without mode, it assumes that you want to overwrite. You will lost these existing records. Use with care. This is helpful if you are sure the master log file has the most complete records.
a, append: Append the records to the existing target files. You may destroy the log file completely by putting irrelevant entries altogether accidently. Use with care. This is helpful if you append want to merge 2 or more log files, for ex., 2 log files of different periods.
i, ignore: Ignore any existing target file, and discard all the records of those months. You will lost these log records. Use with care. This is helpful if you are supplying log records for the missing months, or if you are merging the log records in a complex manner.
f, fail: Stop processing whenever a target file exists, to prevent destroying existing files by accident. This should be mostly wanted when run from some automatic mechanism, like crontab. So, this is the default if no terminal is found at STDIN.
ask: Ask you what to do when a target file exists. This should be most wanted if you are running arclog.pl interactively. So, this is the default if a terminal is found at STDIN. The answers are read from STDIN. Since you have only one STDIN, you cannot specify this mode if you want read the log file from STDIN. In that case, it will fall back to the fail mode. Also, if arclog.pl cannot get its answer from STDIN, for ex., on a closed STDIN like crontab, it will fall back to fail mode.

-k, --keep[=mode]

What to keep in the source file. Current the following modes are supported:

a, all: Keep the source file after records are archived. If --keep was specified without mode, it assumes that you want to keep all.
r, restart: Restart the source log file after records are archived.
d, delete: Delete the source log file after records are archived.
t, this-month: Archive and strip records of previous months off from the log file. Keep the records of this month in the source log file, to be archived next month. This is designed to be run from crontab monthly, so this is the default.

-d, --debug

Show the detailed debugging messages.

-q, --quiet

Shihhhhhh. Only yell when errors.

-h, --help

Display the help message and exit.

-v, --version

Output version information and exit.

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

RELEASE NOTES

Please read to the NEWS file, for newly added features, bug fixes, etc. Please refer the ChangeLog file for the revision history.

BUGS

arclog.pl has a mailing list at SourceForge: arclog-users@lists.sourceforge.net. It is for arclog.pl's users to discuss and report problems. Its web page is at http://lists.sourceforge.net/lists/listinfo/arclog-users. If you have any problem or question on arclog.pl, please go to this page, join the list, and send your questions on this list. Thank you.