iptimer

This program kicks off another script/program, waits a few seconds, does it
again... and again... and again

----- Main Parameter file -----

The syntax of the Main Parameter File - by default tables/setup/TIMER (or
TIMER.fip on Winnt)
	; comment line
	group: (groupname)
	track-status:yes/no
	stagger-starts:1
	client:(name)	day:(progs.MTuWThFSaSu)	time:20:30	mustget:
	client:(name)	date:(UTCdate 20040620)	time:20:30	mustget:
	client:(name)	day:(progs.MTuWThFSaSu)	every:30	start:07:30	end:19:00	maxtime:3600

Other (ie non client) keywords
	group - name of the group
		this is just a handle
	track-status:(yes/no)
	stagger-starts:(number)
		This is the gap in 100ths of a second between starts.
		The maximum is 100 for 1 second delay.
		Use this so all clients for a particular time are NOT started at the same
millisecond.
		default: none for zero wait
	trace-starts:(yes/no)
		Write a trace file of programs and parameters started in /fip/log/timer
		default: no
	extra-fiphdr: (progs.FipSeq)
		More progs.FipHdr information which is added to ALL starts.
		This is in addition to the information in 'client:xx ... fiphdr: ...'
	proxy-server: If using a proxy, these are the hostname (or IP address) and
port to aim at.
	proxy-port:
	proxy-logon:
	proxy-is-squid:yes/no   Is the proxy a Squid ?  default: no
		Proxy is only available for webgrab clients. Pls see the
		doc for webwire for further information.
	throttle-output:(parameter file for iproute)
		The output is to be checked for runaway repeat offenders by passing all data
thru 'iproute'
		This adds an extra progs.FipHdr field 'SR:' to the fiphdr and changes the output
folder to /fip/spool/2brouted.
	webgrab-gap: (no of seconds)
		Gap in seconds between file grabs for webwire
		default is 1 (version 2n11 and after) or 5 if previous

	minimum-every:(no of seconds)
		For those clients with 'every', override the settings with this amount if it
is lower.
		This can also be set with the -E input switch
		default: no minimum (ie 1 second)
		ie if you have a client which is being accessed every 5 secs, a second
		every 10 secs and a third every 60: if minimum-every:30 then the
		first two will be scavenged every 30 seconds and the third unaffected at 60
secs.

	bandwidth-stats:yes/no/client
		Force the worker programs - webwire, ipftp etc - to generate statistics on
duration and data valumes (if they can)
		If the parameter is 'client', individual group_client_program files are
created in log/bandwidth_stats
		Otherwise all information is placed in Hour_group_program files for all
accesses.
		If bandwidth stats are on, then logging is turned OFF for the worker
programs. So they will only log errors.
		default is NO

	skip-balance-group: name of a balance group (in tables/sys/BALANCE) to
distribute
		the skip file when changed (see doc on 'ipbalan') - for ipftp and webwire.
		This is often used where a second system could be used as a redundant server
		if the main system fails. (see also -B input switch)

	default-max-timeout: default maximum time in seconds for every process to run.
		default is 21600 for 6 hours (before version 02m3 this was ignored and there
was no max)
	check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
		Use this when an ipspool on 2 systems is accessing the same remote folder -
in a redundant way.
		that is used whether the current host should be getting the files or not.			
ie in the parameter file is
			check-primary-server:timersvr
		and in the DEST_REDUN is			
			; psuedohost	primary secondary
			timersvr	fip1	fip2
		and in the SYSTEM file for both fip1 AND fip2 there is  the same 'iptimer'
line with the same parameter file
		If fip1 is up,  'iptimer' on fip1 will always get while on fip2 it will just
check/loop.
		If fip1 is down 'iptimer' on fip2 will start processing the files.

In detail :
- 'client' or 'source' keyword (these are identical):
	client: name of a client or source
		which should match a profile defined for the times and dates
	url: 
	host: url to grab or hostname for ftp, nntp etc (url and host are identical)
	port:	port number if not default (webwire, ftp, soap etc)
	match: (progs.FipSeq string)
		(optional and possibly up to 10 may be specified)
	ignore: (progs.FipSeq string)
		(optional and possibly up to 10 may be specified)
	fiphdr: (progs.FipSeq string)
		(optional extra progs.FipHdr information)
		This will add the W3 fiphdr field. To add other fields, use a '#' to
separate:
		fiphdr:abc
			will give a value of 'abc' in the progs.FipHdr W3
		fiphdr:WY:123#WA:online
			will give a (probabaly wrong) value of 'WY:abc' in the progs.FipHdr W3
			plus 'online' in WA
		fiphdr:#WY:123#WA:online
			will give a value of nothing in the progs.FipHdr W3
			plus 'online' in WA and '123' in WY
	test: (yes/no)
		adds the progs.FipHdr field QZ:TEST
		default is NO
	type: (webwire/ipftp/...)
		Type of grabs for the Internet Grabbing  - pls see note 2 below
		default is 'webwire'
	switch: (progs.FipSeq string)
		Optional extra program input switches to use when the program is started
		default is none
	template: (name of template parameter file)
		Mandatory for webwire or ipftp
	logon:
	passwd: (webwire, soap, ftp etc where needed)
		Logon and password for any authentication
		no default
	data: (webwire)
		Do we need the data of the file - data Yes/no
		default: yes
	dest: (all)
		extra destinations (progs.FipHdr DU on output)
		default: no extra
	outque: (webwire, imap, soap and ftp)
		Different Output queue under spool - for webwire this could be for a
rpt-offenders check.
		default: spool/2go
	ignorekey: (webwire)
		add ignore-key:\W5 to the webwire parameter file.
		default: none
	spare: (all)
		Extra information which is added to the progs.FipHdr \W4
		default: none
	script: (path/script options)
		Script name and options for type:x, external script.
		default: none
	maxtime: Maximum time for the job in seconds
		default: 21600 for 6 hours (before version 02m3 - none)
	cwd: (folder name)
		for ftp only, the folder name to 'cd' to

There can be up to 1000 clients in the main parameter file.
either
	date: Actual Date to run this
		Format is UTC style : CCYYMMDD
			ie 20091225		for Christmas Day 2009
or
	day: 	Day of week to run the job :
			M	Monday
			Tu	Tuesday
			W	Wednesday
			Th	Thursday
			F	Friday
			Sa	Saturday
			Su	Sunday
			X	Every day.
			Z	Every weekday (ie Monday to Friday)
				Case is NOT important.
				Commas or dots (but NOT spaces) may be used to separate.
				Default is every day.
either
	time: 	Time of the day	on 24 hour clock.	Default is 18:00.
or
	every:	interval in minutes between grabs	Default: none
		every: (mins)	[(optional)	start:(starttime) end:(endtime)
		for seconds add 's' or 'secs' after the number - 45s for 45 secs
		eg	every:30	start:07:30	end:19:00
			every:30secs
		The minimum interval is 15 secs (every:0.15) unless modified with
'minimum-every:' (see above)

eg:
	client:UKcompanies		start:07:20	end:17:45
	client:Forex	day:MTuWThF	time:16:30
	client:Forex	day:MTuWThF	time:16:40
		Get the Page file tables/setup/FOREX every week day at 16:30 and 16:40


Input options (all optional) :
	-1 : name of a single script/program	default: none
		(same as -z - ie OldStyle; does not use the parameter file at all
	-B : name of Balance Group for skip files 	default: none
		This should exist in sys/BALANCE (see ipbalan)
	-c : Name of a single client or source to trigger an instant snap	default:
spooled
		after which the program stops
		The name MUST match an entry in the parameter file
	-E : minimum access (see minimum-every: above)	default: no minimum
	-H : alternate host name				default: none
		use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
		ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
	-i : queue to scan (-i and -q are identical)	default: none
	-l : log everything through	
	-L : log nothing (timer and worker programs)
		default: timer reports errors only but worker programs log all
	-n : parameter file in 'tables/setup'.	default: TIMER or TIMER.fip
	-q : queue to scan (-i and -q are identical)	default: none
		The DA progs.FipHdr field should contain the client to trigger
	-Q : quiet mode - message if NOT the primary system	default: do not message
	-R : do NOT resolve hostnames		default: do
		This will resolve any hostnames in the DEST_REDUN file (if there is one)
	-v : version and exit
	-V : show the run lines of each program and exit	default: no
	-w : wait time in seconds		default: 60 secs
	-x : external proxy file to use		default: no
		This should be in tables/setup and ONLY have the 4 proxy keywords in.
		This overrules any 'proxy-server' etc commands in the main parameter file
		AND any proxy commands in the program parameter files too.
		AND bandwidth-stats:yes/no/client
	-z : name of a single script/program	default: none
		(same as -1)
	-Z : parameters for the script		default: none
-- NOTES --
1. progs.FipHdr fields
	WN	client name
	WG	group
	W0	url
	W1	matchlinks
	W2	ignorelinks
	W3	extra fiphdr
	W4	spare
	W5	ignorekey
	W6	logon
	W7	passwd
	W8	data yes/no
	W9	extra destinations
	(plus any specified in the extra-fiphdr: keyword)

days:X for all.Z for weekdays (or M-F ??)

2. Types of grabs for the Internet Grabbing are (only the 1st chr is
significant)
	type:w	progs.WebGrabber	(using webwire)
	type:f	Ftp Get		(using ipftp)
	type:e	E-mail		(using ipsmtp)
	type:x	External Script	- no default
	type:m	Manual
	type:i	Instant Messenger
	type:s	Soap		(using ipsoap)
	type:n	NNTP		(using nntpwire)
	type:3	POP3		(using ipsmtp)
	type:p	IMAP		(using imapwire)

3. testing parameters
If you need to use different programs for each of the stages :
	program-webgrab: (path/filename)
	program-ftp:	 (path/filename)
	program-email:	 (path/filename)
	program-im:	 (path/filename)
	program-nntp:	 (path/filename)
	program-pop3:	 (path/filename)
	program-imap:	 (path/filename)
	program-soap:	 (path/filename)

4. iptimer can do a one-off snap of a client by just appending the clientname
to the snap file :
	/fip/fix/setup/FIPtimer.snaps.(name of progs.TimerParameter File forced uppercase)

Version Control
;002n24	13may05 bandwidth stats for ipftp, added skip-balance-group
	;c 24jun05 added minimum-every and -M
	;d-f 06sep05 added -V and -x
	;g 09jan06 external script better
	;h-i 09feb06 added throttle-output
	;j 02aug06 added -Z param switches (k 64bit wait)
	;l 07sep06 bugette with ipftp and output folders
	;m4 05may07 added snaps ;3 tweak to time and added maxtime:
	;n2 20nov08 added check-primary-server
		;3 6jan09 redid imap ;5 17feb09 added source/cwd/host ;6 allow "" in client:
lines
		;7-9 22jun09 added type:s for ipsoap
		;11 6jul09 added -c (single client)
		;12 3sep09 better reporting of errors
		;13-14 22sep09 added trace-starts:yes
		;15-16 28dec09 bugette progs.Win2K primary host
		;17-21 19apr10 tuning timings - webgrab-gap added ;20 added extra-fiphdr,
fiphdr now parsed
		;22 13oct10 added client:.. switch:.. ;23 program-soap ;24 22jun11 allow
webwire -w0
;001z	09may04 added TIMER etc
	;i-j 11jul04 status E only for failed starts plus added SU and SA
	;k 18aug04 added check if anything running before start OR new pram file
	;l-n 01oct04 added Seconds to 'every:'
	;o 01dec04 added input folder for one-off triggers rather than timers.
	;p 10dec04 WINNT - not closing Handles properly
	;r 31dec04 added use-proxy:yes/no
	;s-t 07feb05 FTP - added logon and passwd and data and dests and -q
	;u-y 18apr05 added outque,ignorekey,spare to our list
;000	22jul99 original version


(copyright) 2011 and previous years progs.FingerPost Ltd.

Links

example copy flows ...

Using iptimer to trigger ftp-gets ...

• Using Iptimer to trigger ftp-gets

Notes and Comments

Request

For use with ipftp it would be nice to be able to specify an outqueue and also a "cwd" path. Lucy

Tip ..... you can use hdr: to addd header fields which you can then call in ipftp to give a cd path ..

so: in the client entry of the timer file

fiphdr:QD:path_to_cd_to

and in the ftp parameter file

ftpbeffile: cwd /\QD

-- DotFingerPost - 08 Feb 2005

Correction

alert - the dest keyword documented above should in fact be dests. so

dests:place1+otherplace