ipbalan'ce

This program is used to balance data files on Fip systems. It sends files so
marked to IPSVRD or equivalent on other systems

It can either check the progs.FipHdr for information where to send or can be forced
to send all data to a group/single box.

If for some reason the remote box is down, 'ipbalan' will move the files to an
offline queue and resend when that host is restored.

The progs.FipHdr of the file is checked and the following headers are used :
	DG - name of the group		- mandatory if not running 'force-group'
	SX - full pathname of data file	on both source and remote systems
			If there is NO SX field, the data is assumed to follow
			the progs.FipHdr in the file or there is NO data.
	DN - full pathname of data file	on the remote system
			This overrides the SX progs.FipHdr field for the REMOTE only.
	DK - action on the remote	- optional, defaults to STORE
			'store'		- Store this file
			'append'	- Append data to this file
			'link'		- Link this file - unix only
			'delete'	- Delete this file
			'script'	- run this script
			'ignore'	- just this one
			'notes-upload'	 - upload to notes
These  are optional and only available on Unix
	DD - if 'store' or 'link', link the new file TO a this name
	DV - if 'link', link the file FROM this file. If the file is NOT found
		and there is data attached to the incoming XML, it is used instead.

As the '#' is an end-of-fip-hdr marker, SX and DN are parsed so that # should
be encoded as the 4 printable chrs '\043' (otherwise SX and DN will be
truncated to end on the first '#').
	eg	DN:/fip/spool/testdata/\SN
	or	DN:/fip/spool/\DQ/\043SN:\SN\043DQ:endfolder
		If DQ is 'pdf' and SN is 'abc123' this gives
			DN:/fip/spool/pdf/#SN:abc123#DQ:endfolder

if the progs.FipHdr field ZBALANCE-DATA exists, the SX field is used for the file
name on the remote and the Data following the progs.FipHdr is used.

if the progs.FipHdr field ZBALANCE-STUB exists as well as an SX progs.FipHdr field (with an
external filename), then ONLY the external file is sent. Normally the two are
merged unless the 'nofiphdr:' subkeyword is specified for that 'group:(server)'

If the file cannot be sent, it is moved to the offline queue and the progs.FipHdr
field 'CY' used to denote the hostname of the failed system.

The offline folder is scanned once a minute to see what needs to be retried.

If the progs.FipHdr field ZBALANCE-ID exists and the remote system is offline, then
only a single copy of the file to be sent will be saved - normally this
mechanism is used for balancing skipfiles or other standing files. When the
remote system reappears, only a single copy of the file will be enough to bring
it up-to-date.

If the progs.FipHdr field ZBALANCE-ZAP-INPUT:(filename) exists, the file pointed to
will be zapped after the data has been sent (successfully). The file MUST be in
/fip/data/.. /fip/spool/... or in /fip/x starting 'TMP...'

The parameter file in tables/sys, normally called BALANCE has the following
parameters :
	; comment

	group:(name)	host:(host name) port:1234  fippath:/data4/fip notjustfip:
		name is the name of this group - there can be one or many lines
			all with the same group name.
		hostname is a valid IP address or name
		port if NOT the default which is 9140 for 'ipsvrd'
		fippath default is /fip
			only need if NOT /fip
		notjustfip:
			default is just under /fip
		nolog:		do NOT log the item
		nofiphdr:	Do NOT send the progs.FipHdr
			Normally we send the progs.FipHdr as well as data.
			This really refers to the incoming file in
			spool/2balance and NOT any external file.
			So if the data follows the progs.FipHdr (ie there is NO
			external file - SX:), this flag will send the data only
			and NO FipHdr.
			if there is an external file and it has a FipHdr, this
			is considered part of the data anyway and is NEVER
			stripped off for transmission. So the complete external
			file is ALWAYS sent.
			So if you have an external file which has a FipHdr,
			always specify this option - eg for progs.FipRemotes
		noxmlhdr:	The file already contains progs.FipAgent XML so we
				do NOT need to send one.
		notfipagent:
			NO= none, YES=fipagent, MAYBE= just text
		nowait
			Do NOT wait for the response from the other end
			There may be someone who needs this but we have not
			worked out why !
		failover-mode:
			Normally 'ipbalan' sends to all in the Group. This
			flag denotes only one copy needs to be sent to any
			in the group.
			So instead of one file sent to ALL systems in the group,
			We send one file to ANY ONE of the group - usually the
			first one specified.

		ignore-localhost:
			For this group, do NOT send if the remote host
			is actually the local machine.
		force-queue: (queue or folder on remote system)
			For STORE, all files are to be sent to this folder on the remote system
			The default is the contents of the DN progs.FipHdr field (or SN/SX)
		newname: (replacement name on the remote system if there is no DN progs.FipHdr
field)
			Otherwise The name on the remote system is (in sequence)
				either	contents of the DN progs.FipHdr field (if there)
				or	contents of SX
				or	contents of SN

		use-permissions:no/yes
			Use same permissions on the remote as on the source (unix only)
			So if the source file has permissions of RWXRWXRWX (777) - it will be made
so on the remote.
			The default is NO (or YES if -P input switch is set) meaning the remote
permissions will be the same as the logon which started 'ipsvrd'

		overwrite: yes/no
			Overwrite the file at the remote host - default yes unless -U



	unique-name: SYSTEM
		name of a file which will be different on the remote systems
		hostname is automatically suffixed.
			ie SYSTEM -> SYSTEM.fip1

	default-group: (name of a group)
		use this Group if none is specified (same as -d Input switch)

	retry-timeouts:(yes/no)
		if a message to 'ipsvrd' timeout or the link is broken,
		the default is to retry ad-infinitum.
		Use this parameter 'retry-timeouts:no' to move the file to 'woops'

	allow-data-with-script: (yes/no)
		normally SCRIPT does not have any data sent; use this to do so.
		The data is put into a Temporary file which the script can use as progs.FipHdr SX

	max-open-retries: (number)
		Number of retries to open a connection	- default is 5
	open-timeout:	(seconds)	default: 120
		Wait time in seconds before the remote host should allow a connection
		Reduce this to 10 or so if the remote is just across the computer room !
	timeout: (seconds)		default: 120
		Wait time in seconds before the remote host should ack a file once sent
		If sending large files over very noisy or busy or slow connections, try
		increasing this if you get lots of errors.

The following are only for use with the XML progs.FipAgent and rarely need changing :
	wrapper: xml wrapper string
		eg : wrapper:DEVICE_MESSAGE
		default is FIPAGENT
	data-length: tag containing the length of data following the wrapper
		default is DATALENGTH
	action: Action of the message
		default is REPLY
	message: progs.FipAgent Error Message
		default is MSG
	result: progs.FipAgent result code
		default is RESULT
	device-result: result code from the host process
		default is DEVICE-RESULT 
	poll-timeout: no of seconds which a reply should arrive
		default is 20	(serial running only)
	timeout-message: no of seconds before logging a slow transmission
		default is 20	(parallel running only)
	log-line: Extra logging in progs.FipSeq
		default is none

;
;	BALANCE
;
unique-name:SYSTEM
;
;

group:tables	host:fip1	port:9002
group:tables	host:fip2	port:9002	fippath:/data3/fip

group:w4	host:fip1	notjustfip:
group:w4	host:fip2	notjustfip:


Input Parameters are (all optional):
	-1 : (filename) send a single file and stop	default: spool
	-a : default action if none specified		default: STORE
	-d : default group if none specified		default: check each file
	-E : ignore any external file (SX: progs.FipHdr field). default: send contents of SX
if present
	-f : force all data to this group or client	default: check each file
	-I : (eye) increment the Port number		default: always send to the 9140 (or the
port)
		ipbalan check the IPaddress of its own system and takes the LAST digit and
adds that to 9140
		ie if the IPaddress is 195.13.83.27 then the last digit is 7 and the remote
port -> 9147
		there should be multiple ipsvrd -P914X (where X is 0->n) on each system
	-J : max increment for -I 			default: 10
	-l : (el) log each file through			default: no
	-L : log each file in /fip/log/BALAN		default: no
	-o : done queue for raw files after processing	default: deleted
	-O : Overwrite files in Offline with the same name.	default: all files are
unique
	-q : queue to poll				default: spool/2balance
	-Q : quiet mode - do not flag any missing files.default: do
	-R : run Remotebalance in parallel		default: no
		The parameter is the group in BALANCE
		** This option is ONLY for balancing ipremdata files
	-U : make files on the remote server unique and not overwrite
	-w : file wait for files arriving via NFS etc	default: 0 secs
	-W : 10 ths of a second - MUST wait if nothing processed	default: 2 for
2/10ths
	-x : do NOT use the offline folder as ALL	default: do
		traffic must go to this client.
		Use this ONLY when sending to a single remote system.
	-z : Parameter file			default: tables/sys/BALANCE
	-v : print version no and exit

Current maximums
	Groups and Hosts	400
	Unique Filenames	100

Balance can be used to get 'ipsvrd' to run a script on the remote systems :
Create a file with the following progs.FipHdr fields - remember the tilde NL at the
top:
	~
	DK:SCRIPT
	SX:/fip/local/fff	UP
	DG:scriptgroup
where
	DK is fixed as 'SCRIPT'
	SX is the full path and filename of the script with any parameters;
		it can be a progs.FipSeq string. You must be able to run the script
		or program from the logon which 'ipsvrd' has been started.
	DG is the normal group which determines which systems to run the script.

Example
	BALANCE
		group:fiptest   host:fip1	ignore-localhost nofiphdr:
		group:fiptest   host:fip2	ignore-localhost nofiphdr:
		group:fiptest   host:fip3	ignore-localhost nofiphdr:
		group:fiptest   host:fip4	ignore-localhost nofiphdr:

.. and the stub file
~
DG:fiptest
DK:STORE
XX:time.1180132986.pid.11281.seq.1
XX:/fip/bnew/ipvi
XX:at.Fri May 25 23:43:06 2007

SX:TTTT
DN:/fip/spool/xo


Version Control
;002w6	08dec05 allow zero length files to be copied over
	;b-f 24feb06 cater for Resource problems on Sparc (f for 64bit wait3)
	;g-h 13sep06 added force-queue:
	;i 23sep06 added max-open-retries and bug in offline sometimes sending to more
than one system = DONT USE
	;j-l 03oct06 new style DR with up to 10 secondaries PLUS bug in 02i PLUS bug
in single
	;m-q 07nov06 added action notes-upload and -a (; p speedy)
	;r-u 31mar07 added preserve-filename: and log-line: and timing-stats
	;v9 31may07 redid modes/permissions and added overwrite and -U
	;w4 31jul08 added -E to ignore-external-data ;1 19may09 added ZBALANCE-ID ; 2
8jun09 added ZBALANCE_STUB
		;4 parse doneque ;5-6 added ZBALANCE-ZAP-INPUT
;001z5	19may00 redid offline a bit
	;b 05jun00 added LINK and DV
	;c 14nov00 added DN for name on remote
	;d/e 25apr01 allow data-in-file as a default if no SX -this is a change!
	;f/g/h 02jan02 added failover: and bugette for progs.FipHdr on File and no SX
	;h/i 22mar02 SX and DN are now parsed so that # should be encoded as \043
	;j 25jul02 redid Timeout routine
	;k 28jul02 buggette in WINNT offline 
	;l/m/n 09aug02 added open-timeout
	;o 28aug02 better handling of 4xx errors - send to woops
	;p 28aug02 added retry-timeouts.
	;q-s 14jul03 added APPEND as an Action
	;t 17sep04 speedy
	;u-y 25feb05 remote parallel sends
	;z5 09aug05 added -I increment Ports and -O overwrite offline files

(copyright) 2011 and previous years progs.FingerPost Ltd.

Notes and Comments