Maintenance

FIP (FingerPost Information Processor) maintenance is normally controlled by a shell script called ‘zapfiplog’ which can be found in the directory /fip/local.

The daily maintenance typically covers:-

• Ageing of the item log files
• Maintaining the number of files in the archive directory (/fip/log/data)
• Purging various ‘done’ directories

The script ‘zapfiplog’ is normally run at midnight each day and is controlled by the Unix scheduler ‘crontab’.

To see what tasks are scheduled to run, execute the command ‘crontab –l’. To edit the scheduled list, use the command ‘crontab –e’, which opens the task list in ‘vi’.

A typical ‘crontab’ entry looks like:-

0 0 * * * /fip/local/zapfiplog

The columns represent:-

• The number of minutes past the hour
• The hour in the day
• The day of the month
• The month of the year
• The day of the week (0 = Sunday, 1 = Monday, 2 = Tuesday etc)

The entry for ‘zapfiplog’ means that it will run at midnight every day of the week.

Ageing of the item log files

The item log file, which contains all the message lines logged by the various programs, is called ‘ALL’ and is stored in the directory /fip/log. This file is accessed when using the following commands in the text interface – f, l, m, t, and y.

Example item log entry

Wed Aug  4 14:28:42 iproute !i : AP   MAW2818 (sxx-4) w4apsport   ^BC-EU-SPT-OLY--Athens-Security<
Wed Aug  4 14:28:44 ipw4 !o : 20040804_ap   maw2818 SPORT_GENERAL

The program ‘ipmaint’ is used to age this file and by default a minimum of five log files are saved. ‘ipmaint’ uses a parameter file called /fip/tables/sys/MAINT.

The first time that 'ipmaint' is run, the file ‘ALL’ is renamed ‘ALL.1’ and a new ‘ALL’ file is created. The second time ‘ipmaint’ runs, ‘ALL.1’ is renamed ‘ALL.2’, ‘ALL’ is renamed ‘ALL.1’ and a new ‘ALL’ is created. After six days there will be six log files ‘ALL’ and ‘ALL.1’ ...... ‘ALL.5’. The next time maintenance occurs, the oldest file ‘ALL.5’ is deleted and the rest of the files are incremented by 1 and a new log file is created.

To keep a different number of item log files, run the ‘ipmaint’ program with the –d switch e.g.

/fip/bin/ipmaint –d 10

will keep 10 days of item log files.

When a new ‘ALL’ file is created, ‘ipmaint’ can start the new log file with a few lines from the end of the previous days log file. Therefore if you would like the log to contain 10 minutes from the previous day, make sure the ‘MAINT’ parameter file contains the entry:- 23:50

Each entry in the item log file has a ‘flag’ which is an exclamation mark followed by one or two characters. The ‘flag’ defines the type of log entry i.e.

!i – the entry signals a file that is coming into the FIP system e.g. iproute, ipspool
!o – a file that is leaving the FIP system e.g. ipftp, ipsmtp
!z – an administrative entry – nightly maintenance e.g. ipmaint, ipdelque
!f – an error entry

It is possible to save certain flagged entries by adding a line to the ‘MAINT’ file in the form:- string_to_match filename e.g.

!x         /fip/log/ERRORS

When ‘ipmaint’ runs it will look for any entries that contain the string ‘!x’ and will either create the file /fip/log/ERRORS, or append them to the existing file.

Maintaining the number of files in the archive directory (/fip/log/data)

Archive files are created by the program ‘ipwheel’ and are a compilation of every file that has been received for a source on each day. The archive files are stored in the directory /fip/log/data and the filename takes the form NNN_SSSSS, where NNN is the julian day (number of days since the beginning of the year – Jan 1st is day 0) and SSSSS is the value of the SU (source name) header field. For files received by one of the ‘wire’ programs, this is the argument supplied to the –n switch e.g. –n APDS.

Note: when a file is resent using the ‘r’ command in the text interface, the data originates from these archived files.

To control the number of archive files saved, the ‘MAINT’ file must contain a number of entries in the form:-

/directory/SU      NN

where ‘directory’ is the name of the directory in /fip/log, SU is the name of the source and NN is the number of archive files for that source e.g. /data/APDS 15 means keep 15 days of files from the source APDS, in the directory /fip/log/data.

The number of archived files saved, normally depends on the amount of spare disk space available, though a figure somewhere between 5 and 20 days is most common. Sometimes if one source provides a lot of the files that are data formatted, it may make sense to keep a lot more of these files. Example of the MAINT parameter file

;
; 'MAINT' - Maintenance file for ipmaint
;
; Note: ‘ipmaint’ is started by the script /fip/local/zapfiplog
; which is run by crontab
;
; Start the new item log file at ‘23:45’
23:45
;
; Purge the archives
;
; Archive      Number of days
; Name         to keep

/data/APDS      5
/data/APSPT     5
/data/BLOOM     5
/data/DOWJO     5
/data/MAIL      5
/data/NYT       5

; Save the item log file errors
!x         /fip/log/ERRORS

; *** End ***

Purging various ‘done’ directories

Purging of miscellaneous directories is done with the program ‘ipdelque’. This program can delete all the files in a directory or files older than a few minutes, hours or days. Alternatively the program can delete files whose filenames match/does not match a specific string and it will even look inside the file in order to delete files that contain/do not contain a specific string.

Examples

ipdelque -q /fip/tables/form/bin

Deletes all files from the directory /fip/tables/form/bin

ipdelque -q /fip/log/ftp -m 5

Deletes all files more than 5 days old from the directory /fip/log/ftp

ipdelque -q /fip/x -k TMP -M 360

Deletes all files more than 6 hours old (360 minutes), whose filenames contain the string ‘TMP’ from the directory /fip/x.

ipdelque -q /fip/data/w4/files -A 28

Deletes all directories more than 28 days old. The purging is based on the directory name, therefore it must take the form YYMMDD_xxxx or YYYYMMDD_xxxx where YY/YYYY is either a 2 or 4 digit year, MM a two digit month and DD a two digit day. xxxx can be any other string.

The program ‘ipdelque’ can also be run from the command line. After pressing Enter/Return the user is given a 30 second grace period. If during this time, the key sequence Control-C is pressed, the program will not have started its purging. This grace period can be reduced, as is often the case in the ‘zapfiplog’ script, by using the –i switch e.g.

ipdelque -q /fip/spool/formsave -m 2 -i 0

Delete all files more than 2 days old from the directory /fip/spool/formsave and start the purging immediately.