ipspool

IPSPOOL is used to bring files into FIP from another system. It runs against an
input queue and any file found is sent to the output queue possibly with FIP
header information added possibly running a program/script against it.

Mac resource forks may be zapped at the same time. For files arriving across
the network, a file creep time may be used to wait for the file to complete
before it is processed.

The file contents are normally left unaltered but a small change - removing the
CR of a CR NL combination may be used for incoming PC files.

A filename mask may be specified where other programs use the same queue for
temporary files. Only files matching the mask are processed.

The incoming file name is used for the output name except that certain
characters are removed and replaced by '_'. These include space, tab, control
chrs (deleted), $, *, /, |, \, < and > and single and double quotes.
This can be stopped used the -n switch and/or quoted withe to -N (quotename)
switch.

Note that in order to pass parameters to a script specified in -p, put double
quotes around. For example :
	ipspool -i eyeye -o delete -p "scripto param1 \SN"
will run script called 'scripto' (which MUST be in /fip/local if no path is
specified) and pass it parameters :
	Parameter 1 : the fixed string 'param1'
	Parameter 2 : whatever is in the Fip Hdr field SN
	Parameter 3 : The full path and filename of the input file

Errors from the script will be logged - 256 or higher means that the script
cannot be found or does not have execute permissions.

Note that all files starting with a dot are ignored and, on Unix, any files
called 'core' are assumed to be core dumps and are ignored !

If the '-A' switch or 'allowsubques:' parameter is used, if a queue is dropped
into the input queue, then the contents of that queue are treated in the normal
manner but any sub-subqueues are flagged as error. If there is a done queue,
the subqueue is moved into that and the a subfolder in the output folder is
created too unless the folder is '2go', 'xchg', '2edsys' or '2brouted' (or the
no-output-subqueues: parameter is specified).
- Normally the sub folder is removed when empty, so use input switch -a or
parameter 'preserve-subqueues:' to leave alone.
- Normally the sub folder is created in the output folder (as mentioned above),
so use input switch -Q or parameter 'no-output-subqueues:'.

Using subqueues and any MAC switch means the program will ignore queues like
'.rsrc', 'Trash', Network Trash Folder' and 'DeskTop'.

When accessing/writing to queues that are physically on another system - NFS
drives for example - it is a good idea to check that drive is actually mounted
beforehand.
Two switches, -I and -O (and their identical parameter file keywords,
chkexists-input and chkexists-output) can be used to check whether a specified
file or queue exists before the input queue is scavenged (or output written).
As filenames starting with a dot are ignored, one way to implement this is to
touch a file in this input queue. eg :
	cd /mountedOnRemore/spool/input
	touch .DontZapMePls
While in the SYSTEM file we could have something like :
remo	local	ipspool -i /mountedOnRemore/spool/input -I .DontZapMePls -o 2go -h
"#DU:Toinfinity#SN:"


Input switches are :
	Mandatory parameters :
	-i : input q (normally in ~/spool)			no default
		if the path starts with a '/', it is considered a hard path
	-o : output q (normally in ~/spool) or 'delete' to delete it.  no def
		if the path starts with a '/', it is considered a hard path

	Advised parameters :
Either	-Z : name of the parameter file to use		default: none

Or	-h : hdr string to be added BEFORE the filename in the output q
		This parameter is advised but not mandatory.	no default
		Note for NT this has no effect on the FipHdr,
		so use the 'extra-FipHdr-fields' parameter to create a progs.FipHdr
		with meaningful fields that can be used for other Fip programs.
		For Unix, please do NOT add any '/' as it will create subqueues.
	-H : hdr string to be added AFTER the filename in the output q
		This parameter is advised but not mandatory.	no default
		Note for NT this has no effect on the FipHdr,
		so use the 'extra-FipHdr-fields' parameter to create a progs.FipHdr
		with meaningful fields that can be used for other Fip programs.
		For Unix, please do NOT add any '/' as it will create subqueues.

	Optional parameters :
	-b :
	-B : balance queue for save-data - same as save-data-balance-folder	default:
none
	-c : copy the file NOT link. This is for files residing on
		a different volume to /fip/spool		default: link
	-C : supercede:	Overwrite any existing files		default: NO
	-d : done queue for a copy of the raw input file	default: none
		if the path starts with a '/', it is considered a hard path
	-l : do NOT log each file through			default: log.
	-f : file wait (ie wait x secs for file to finish before processing)
								default: 5 secs.
	-t : sleep between scans of the directory 		default: 5 secs
	-p : program or script to execute BEFORE sending	default: none
	-P : parameters for the script specified by '-p'	default: none
	-e : error queue if script returns an error		default: same as output
		if the path starts with a '/', it is considered a hard path
	-E : ignore the file if the script returns an error	default: move file to '-o'
	-z : log file for the executing program or script
		if you have a program running against each file, this can
		be used to log the result.			default: none
	-T : maximum time allowable for a script before it is killed. def: no limit
		This is to catch scripts/programs which are looping or are
		flow controlled or for someother reason they have hung.
	-m : for Mac files : zap the progs.ResFork file pls
		Removes the resource fork			default: no
		See also -R for the resfork Path/name
	-M : for Mac files : move the progs.ResFork file to the done que. def: no
		Note that you must ONLY move resource forks within progs.AppleShare
		volumes as they contain unique ids which are guaranteed to
		screw up if moved between vols!
		See also -R for the resfork Path/name
	-R : for Mac Files Res Fork path/prefix			default: .rsrc/
		For Mac NFS where the resourcefork is a second file in the
		queue with a '%' in front : use	-R%
	-q : quiet mode - do not message if we find directories or other
		non-files are found in the input queue - just ignore them.
					default: log message if directory found
	-r : replace CR or CR LF with just LF
					default: leave file contents alone.
	-k : filename mask. Send only files containing this string.
					default: send all files
	-K : filename unmask. Do NOT Send files containing this string.
					default: send all files
	-X : If both Mask and Unmask are set and a file meets both criteria,
		Send it.		default: do NOT send it.
	-n : do NOT look for Unix meta characters in the filename.  Normally
		$, * <> |\/ etc are changed to '_'.	default: change to '_'.
	-N : Put single quotes around the filename	default: no quotes
		when running a script. This enables filenames
		with spaces and other metacharacters to be passed.
	-s : Minimum size for file			default: no
		Files smaller than this are deleted
	-S : Maximum size for file			default: no
		Files larger than this are deleted
	-I : Check File/Queue name for input		default: none
		see above
		If the path starts with a '/', it is considered a hard path
		otherwise it is considered relative to the input queue.
	-O : Check File/Queue name for output		default: none
		see above
		If the path starts with a '/', it is considered a hard path
		otherwise it is considered relative to the output queue.
	-A : allow files to be placed in sub queues in the main input queue.  default:
no
	-a : if using subqueues, do NOT zap  empty subfolders	default: zap em
	-W : add a progs.FipHdr and add normal progs.FipHdr fields such as the
		date and time (HH, HM etc) plus internal fields
		such as EN, EP and EQ 			default: no
	-b : default round robin maximum	default is none
		see also parameter 'round-robin:'
	-9 : do NOT run speedy (valid on a speedy system only)
	-8 : input is NOT speedy (valid on a speedy system only)
		ie the input folder is NOT piped but the rest of the system is.
	-v : version no and exit

Using the -Z switch you may (optionally) specify a parameter file in
tables/spooler can be used to provide much the same info as the other input
switches (plus a bit more !!). If both switches and parameter file are used,
the parameter file takes precedent.
The parameter files are held in tables/spooler.

Parameters are the usual FIP style of :
	; comment line
	(keyword): (subparams) ....
and include :
	hdrbefore:	same as switch -h
	hdrafter:	same as switch -H
	script:		same as switch -p
	param-script:   same as switch -P
	logscript:	same as switch -z
	timeout:	same as switch -T
	copy:		same as switch -c
	nolog:		same as switch -l
	minsize:	same as switch -s
	maxsize:	same as switch -S
	wait:		same as switch -f
	outque:		same as switch -o
	doneque:	same as switch -d
	errorque:	same as switch -e
	strip:		same as switch -r
	zapmac:		same as switch -m
	movemac:	same as switch -M
	respath:	save as switch -R
	quiet:		same as switch -q
	mask:		same as switch -k
	unmask:		same as switch -K
	meta:		same as switch -n
	quote:		same as switch -N
	allowsubques	same as switch -A	allow-subqueues and allow-subfolders are all
identical
	preservesubques	same as switch -a	preserve-subqueues and preserve-subfolders
are all identical
	no-output-subqueues:	same as switch -Q
	add-all-fiphdr-fields:	same as switch -W
	no-fiphdr:	same as switch -X

Plus these extra ones
	wait-for-data: (progs.FipSeq)	Do not process the file until this string
		appears in the data. In this case the -f input switch is used as a
		MAXIMUM time.
	extra-fiphdr-fields:	progs.FipSeq to add to the FipHdr.
		eg extra-fiphdr-fields:#SU:outgoing#SC:notes#SN:\EN#PR:\EQ
	extra-fiphdr-script:	program which will create more progs.FipHdr fields
		\E3 hold the name of a TMP file to create that will be read for the list.
		\E4 hold the name of a TMP file that holds the contents of the incoming file
		eg extra-fiphdr-script:/jpeg/rdjpgcom -fiphdr \E4 > \E3
	winnt-input-drive:
	winnt-output-drive: drive letter of the queue if not the default
	wait-for-remote-server: (secs) On flaky networks, the remote drives
		may not be available all the time. Use this parameter to log it
		has gone missing, pause a few seconds and retry - but do not stop the
program.
		Default is none. If specified with (secs) missing, the default is 10
	number:	octal/dec/hex	Number system to use for progs.FipSeq data
	first:		progs.FipSeq data to insert at the top of the file
	last:		progs.FipSeq data to insert at the bottom of the file
	prefile:	full path and filename of a file to be included
			after any 'first' text and before file text.
	endfile:	full path and filename of a file to be included
	chkexists-input: same as -I switch
	chkexists-output: same as -O switch
	chkexists-input-script: Script (in progs.FipSeq) to run before accessing folder
		If this script does NOT return '0', it is assumed to be an error.
	allowsubques:	same as -A switch
	supercede:	Overwrite any existing file	default is NO.
		This also sets the default if you have several masks/unmasks
	save-data-path: (pathname for data)
		This puts the data of the incoming file into this folder and creates a progs.FipHdr
		file that contains 2 progs.FipHdrs containing the full path/filename
			SX: and	FTP_EXTERNAL_FILE:
		(ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
			eq	save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/
		Use this for big files that you do not want to copy around the Fip Spool
area.
	save-data-filename: (progs.FipSeq name)
		Use this to specify exactly what the the 'save-data-path' name should be
		default is (incoming filename).(time).(seqno)
		eg 	save-data-filename:\HR-\SU.raw
	save-data-fiphdr: (yes/no)
		Save any incoming progs.FipHdr with the saved file	default: yes
	save-data-balance-group: (Balance group)
		Balance all save-data files to the following group. default: do not balance
	save-data-balance-folder: (Balance folder)
		If balancing, put the token in this folder under spool. default: 2balance
	create-checksum: (2 letter progs.FipHdr field)
		Create a 32 letter MD5 signature checksum in this progs.FipHdr field
		Many web-type organisations which allow downloads will also give, separately,
an MD5
		signature. Use this to compare so you can verify the data has not been
tampered with.
		eg	create-checksum:CS
			Create a progs.FipHdr field CS like 'CS:62663683bbbd3b84779d072f53178919'
	round-robin: (number)
		Send to output queues ending with a digit up to this number
		default: none
		ie if round-robin:3 and outque:/data/today
		The first file through will be put in a folder /data/today1, the 2nd in
/data/today2 etc
	round-robin-fiphdr: (2 letter progs.FipHdr field)
		Add this Fiphdr to the output file so it can be used downstream
		default: none
	tracker-script: (fullpath to script name)
		default: none

MASKING
-------
	wildmask:*	Define a character to be used as a wild-card for masks.
			default: none
	wildsingle:?	Define a character to be used as any single chr for masks.
			default: none
	exact-match:yes/no	NO (default) - the mask is a string which can be ANYWHERE
in the
					eg mask:8	will match any file with '8' in the name
				YES - the match is an exact match
					eg mask:1954*	will ONLY match files starting 1954....
	mask0,...,mask9
	unmask0,...,unmask9
	script0,...,script9
	param-script0,...,param-script9
	logscript0,...,logscript9
	outque0,...,outque9
	supercede0,...,supercede9
	fiphdr0,...fiphdr9
	no-fiphdr0,...no-fiphdr9
	filename0,...filename9
	file-type0,...file-type9	check the first 4 bytes of the file for its 'magic'
number
	wait-for-data0...wait-for-data9
	check-mask-first: yes/no
			Change logic ....
			Either check mask/unmask before checking file-type (YES)
			or check file-type BEFORE checking masks (NO)

	Up to 10 mask and/or unmasks can be specified.
	Any of the numbers can be used - they do not have to be sequential.
	If ANY unmask if found, the file is ignored.
	If more than one mask is found, the one with the lower number is used.
	If the optional script is required, if there is a script of the SAME number,
it is run else 'script' is run. Masks/Unmasks can be in FipSeq.
	The file which matches the mask/unmask can also be moved to a different folder
with 'outqueX' where X is the index of the (un)mask.
	Supercede2 etc can be either YES or NO.
	The base 'wait-for-data' is used as a default for any index which does NOT
have a wait-for-data.
	Note that 'fiphdr0' etc add additional fiphdr fields to files matching that
one mask. While 'extra-fiphdr-fields' is added to ALL fields irrespective of
the mask.

	Note that if 'no-fiphdr' is specified, all other options which use the fipHdr
to store metadata are ignored including (for example)
		save-data-path
		extra-fiphdr-fields
		extra-fiphdr-script


	Example of using masks
		; we have an input folder which can receive several different
		; types of data. Each needs a different Fip destination (and
		; other progs.FipHdr info) so it can be treated differently
		; using '*' as the bits to ignore
		wildmask:*
		; 1. stream off all xml files
		mask2:*xml*
		; 1.1 stuff into /fip/spool/sgml
		outque2:sgml
		; 1.2 add the relevant fiphdr info (DY is the ipsgml param file)
		fiphdr2:DY:MYXML
		; 2 stream off the pdfs
		mask4:*pdf
		; 2.1 just copy to edsys to place in a nice folder somewhere
		outque4:2edsys
		; 2.2
		fiphdr2:DF:SEND.PDF
		; 3. catch all for all the other files
		mask9:*
		; 3.1 send them via iproute to check more
		outque9:2brouted
		; 3.2 route needs a SU (source) and ZI for archive
		fiphdr9:SU:TXT#ZI:

	check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
		Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
		that is used whether the current host should be getting the files or not.
		ie in the parameter file REMOTEGET is
			check-primary-server:remotewire
		and in the DEST_REDUN is
			; psuedohost	primary secondary
			remotewire	fip1	fip2
		and in the SYSTEM file for both fip1 AND fip2 there is a line
			rem1	local   ipspool -i /nfs/data/input -o 2go -Z REMOTEGET
		Then if fip1 is up, the ipspool on fip1 will always get while on fip2 it will
just check/loop.
			if fip1 is down, the 'ipspool' on fip2 will start getting the files.
	hostname: (hostname)
		Hostname to use for this server for using with redundancy
	check_redun-every: (no of secs)
		If running redundant systems with check-primary-server, this is the delay
between checks for the secondary system.
		default is 60 secs.
	timing-stats:yes/no
		Show timings statistics		default: no
	wait-between-files: (tenths of a sec)
		small pause between each file.. in tenths of a sec
		default is 10 for 1 sec (max is 100 for 10 secs)
		eq	wait-between-files:2

	fiphdr-original-time:(2 letter progs.FipHdr field)
		Add the UTC time when the file was created as this progs.FipHdr field.

	copy-file0...copy-file9
		There can be up to 10 copies of the same file
		Take care to speicfy the FULL path AND filename in fipseq
		copy-file:/fip/spool/2balance1/#SN:SN#SU:bal1#XX:\$u.fip
		copy-file2:/fip/spool/2balance2/#SN:SN#SU:bal2#XX:\$u.fip
		copy-file6:/fip/spool/2balance6/#SN:SN#SU:bal6#XX:\$u.fip

SAVING FILES FOR IPFTP
----------------------

Use this for big files that you do not want to copy around the Fip Spool area.

Use save-data-path to squirrel a copy of the data file while the 'output' file
is just a small progs.FipHdr file.

Normally you would also want	'add-all-fiphdr-fields'
2 extra progs.FipHdrs fields are created containing the full path/filename
	SX: and	FTP_EXTERNAL_FILE:
	(ipbalan uses SX and ipftp uses FTP_EXTERNAL_FILE)
		eq	save-data-path:/fip/data/jpegs/\$e\$y\$i\$d/

Any version of ipftp (from version 14k) will pick it up.

** Note it is up to you when you purge the external or save folder - in the
above example /fip/data/jpegs/(todays date)

If you have one file in -> one ftp out, then get ipftp to delete it by adding
FTP_ZAP_EXTERNAL: to the fiphdr - again use ipspool for this
	extra-fiphdr-fields:#FTP_ZAP_EXTERNAL:#

Otherwise if you have masses of disk space, just add a line in the nightly
maintenance /fip/local/zapfiplog (or zapfiplog.cmd) to wipe out the files after
a day or so.
On Unix/Linux :
	# save 8 days of jpegs ..
	if [ -d /fip/data/jpegs ]; then
	   /fip/bin/ipdelque -q /fip/data/jpegs -A 8 -i 0
	fi

NOTES
-----
- For AIX 4.1 : If calling a script using the '-p' switch or the 'script:'
parameter, you MUST make sure the first line of the script starts with a '#!'.
	eg for a bourne shell :		#! /bin/sh

- Always make sure you exit from a script with an 'exit 0' if it has finished
correctly.

- Note that some internal progs.FipHdr fields are available to the program
	They are NOT added to the output file automatically.
	But you can do automatically with the keyword 'add-all-fiphdr-fields'
		which will also add all the HH/HD/HM fields for date and time
	SN - output filename
	EN - input filename
	EQ - lastpart of the input folder/queue
	EP - full input folder

WIN 2K and UNC drives
---------------------

If using UNC drives, note that the Win2k 'security model' no longer allows the
default service logon (progs.LocalSystem) to access external drives. So you will need
to start the 'Fip Comms Manager' service under a Logon which exists on the
remote system AND has permissions to read AND WRITE to the folders you want to
read and/or write.

You will need to
either	- The easiest method is to have the same logon/password for both systems
		- if running a Domain server, use a single logon for both
		- if not create/use the same logon/password on both
or 	- The remote system needs to know about the local system's logon and have
permissions to read/write

How do you know it will work
	- Logon the Fip/local server with the logon
	- Open RUN and type the full UNC path
		- try reading and/or writing a file
		- if it does not work, there is no way Fip will either ! So fix it first
	- When that works..
		- Open CMD window and change to the drive  where Fip is.
		- type in the complete ipspool parameter line from the SYSTEM file
			(remember to have NO spaces between input switch and parameter ie -i2proc
not -i 2proc)
			- if that does not work, check your syntax and ogon again
	- When that works..
		- Add the logon to the ControlPanel->Admin Tasks->Services->Fip Comms Manager
			- stop and start

- Note for Fip UNC drives, always use '/' rather than '\' as the latter is the
escape chr (so you need '\\' for every '\' !)

- For NT note that the path name for a script is using forward slashes NOT
backwards
 The '-P' or param-script keyword however is not changed to forward slashes


Version Control
;032s25	21oct05 added create-checksum:
	;a-b 17jan06 added -b and round-robin and slightly better logging
	;c 23jan06 added -8
	;d 29mar06 -q also affects locked files
		and possible bugette with NOT cleaning out remnants of missing files.
	;e-h 05aug06 for -A subqueues : added -a for preserve-subfolders and -Q
no-output-subfolders
	;i 18jan07 bugette - supercede0 was missing and added param-script0-9 and
logscript0-9
	;j-k 22mar07 bugette - do not use DEST_REDUN if not needed
	;l 04oct07 added filesize at the end of the log message
	;m-n 24oct07 added extra-fiphdr-script and save-data-filename:
	;o 31oct07 check for primary server before each file in addition to once every
folder scan
	;p2 08nov07 added copy-file0-9 and no-fiphdr0-9
	;r6 29nov07 added rename in doneque ;r1 added fiphdr-original-time: ;r2 added
E4 ;r3 bugette with -p /-P ;r4 for fipredunBalanced
	;s15 31jul07 added save-data-balance-group/queue and -B ;3456 bugette WINNT
curdrv and speedy
		;7-8 bugette in wait-for-data ;9 redid -r ;10 added round-robin-fiphdr ;11
added file-type: 0-9
		;12 added tracker-script ;13 added secs to wait-for-remote-server ;14
file-type tuned ; 15 usleeps tuned
		;16 allow dbl qtes around any input switch que - input, balance, error etc
		;17 added chkexists-input-script
		;18-20 5oct11 bugette with subques (both linux and win2k) and added filter
;20 added wildsingle
		;23-25 19oct11 added check-mask-first:yes/no,subques - do NOT allow '..'
;031z1	01feb01 added wait-for-data ;a bugette with masks when using -k/-K
	;b 01may01 masks are now progs.FipSeq
	;c 11may01 added outque as a parameter
	;d 14jun01 added outque1-9 according to mask/umask
	;e 17aug01 small problem with mask/unmask and wild at the END
	;f 13sep01 added supercede2-9
	;g 02jan02 added mask1/unmask1/supercede1 etc
		and found the bugette where SN sometimes disappeared!
	;h 03oct02 added fipseq functions - partial/repeat/style etc
	;i 16oct02 bugette in progs.FipHdr bit
	;j 15nov02 added redundancy/check-primary-server
	;k/l 01jan03 MACOSX
	;m 23may03 added fiphdr0-9
	;n 06aug03 bug in outque/outque1
	;o 10feb04 added filename0-9
	;p 15may04 added timing-stats
	;q 01jul04 bugette in incoming fipHdrs.
	;r-x 07oct04 speedy (w allow drive letter for script)
	;y 18jul05 added wait-for-remote-server
	;z1 11oct05 added save-data-path:/fip/data/incoming/\$d/
		doneque is now progs.FipSeq and not just hard paths
;030	30nov00 upto 9 masks/unmasks and scripts
;029f	01sep99 added extra-fiphdr and add-all-fiphdr-fields
	;a 13sep99 NT mods for drives
	;b 15sep99 added -E
	;c 28oct99 added toomanyfiles check
	;d 14jan00 small WINNT hdr mod
	;e/f 20jan00 supercede added

(copyright) 2011 and previous years progs.FingerPost Ltd.

Notes and Comments

How does ipspool cope with subdirectories in the spooler queue ?

A simple 'ipspool' syntax looks like this:-

ipspool -i abc -o 123

If you create the sub-directory /fip/spool/abc/tst, 'ipspool' doesn't like to see the sub-directory, so complains e.g.

ipspool !x : ** Directory found in input queue - pls remove : /fip/spool/abc/tst 2 (No such file or directory)

If you really want the sub-directory to be there, to stop the errors you just add the -q switch

e.g.

ipspool -i abc -o 123 -q

The -A switch is used where files and directories maybe stored in the input directory, and you want both lots of data processed.

Therefore a file called 'gunshot', saved into /fip/spool/abc, will be moved to /fip/spool/123 e.g.

/fip/spool/abc/gunshot ---> /fip/spool/123/gunshot

and the input file will be deleted.

A directory called 'bullet', containing a file called 'rifle', saved into /fip/spool/abc, will be moved to /fip/spool/123 e.g.

/fip/spool/abc/bullet/rifle ---> /fip/spool/123/bullet/rifle

and the input directory and file will be deleted.