Fip - New DataFormats Administrators Manual

this version: 003 26may06

Overview

The new version of the DataFormatting module extends and replaces the existing version by adding

• Separate Test and Production environments
  • allowing significant changes to be made without interfering with Production
• Upgrade and Rollback of jobs in Production
• Concept of Stages
  • Multiple stages rather than the old xchg->format->xchg model
  • Stages can be of any accepted program
  • Each stage can be tested separately
• Multiple Publications working (semi) independently on the same (pair of) servers
• Much better security

This guide aims to explain and detail what steps are needs to install and run.

Contents

Links to other DataFormat bits

• Users Guide - DataFormattingv2user
• Help on Syntax of data formats
• XSLT
• Others scripts or programs written in perl, csh, VB etc
• Fip Program Documenatation
  • Ipxchg
  • Iproute
  • Ipformat
  • Ipformbl
  • Ipsetter
  • Sffsort

HowTo ...

Initial Setup

Preparation

• Name of Publication(s)
  • A three, four or five letter code plus the name is required

• Will you be running multiple publications and do you need to block access between them ?
  • If so, you will need the logon lists version of the w4 logon stuff

• Do you have up-to-date binaries and scripts from FingerPost - the minimum versions are
  • the default form.setup
  • ipformat 54t
  • ipformbl 37w
  • ipformd 22h
  • ipxchg 47z
  • sfflogon 02k
  • form_job.pl 00x
  • form_admin.pl 00s
  • sffform.pl 01f
  • sffbase.pl 02h
  • admin_logon_radius.pl 06b

• Download the relevant WEBSHOW* xchgs from FingerPost

• If running Solaris or Win2K, you will also need the 'sdiff' and 'cmp' from the gnu 'diffutils' package (pre-built binaries are on the FingerPost site)

customer.setup

This is used to control the behaviour of the whole w4 system. It is placed in /fip/web/setup.

The data formatting bits are to define the publication(s) used and (for multiple publications) the logon list.

There should be at least one 'logon-pub' keyword:-

• logon-pub:DB|The Daily Bugle
• logon-pub:MEGA|Mega Entertainments Weekly

The logon list file is either FIP.INTERNAL or FIP.EXTERNAL

• logon-list-file:FIP

form.setup

This is used to control the behaviour of the system and to allow access to the various aspects. It is placed in /fip/web/setup

Do we allow Perl and/or non-standard programs/scripts

• allow-perl:yes
• allow-nonstds:yes

Restrict access to parameter files of other publications

• NONE = No restriction
• PUB = Between publications
• SECTION = Between publications and sections
• restrict-access:pub

Balance - this needs to match the entry in ~/tables/sys/BALANCE for all other fip systems running dataFormats

• balance-group:formadmin

What views of the data are required ? These map to exchanges e.g. ZWIRE requires ~/tables/xchg/WEBSHOW.ZWIRE

• view:SWIRE,SWIRE2|AP S-Wire Data
• view:SWIREBLANK|AP S-Wire Data (no formatting)
• view:ZWIRE|AP Z-Wire Data
• view:PADF|PA Data Formatted Fields
• view:INCOPY|InCopy Markup
• view:QUARK|Quark XPress Tags
• view:SETTER|Fip Setter Tags

For Solaris and Win2K, use the GNU versions of 'diff' once you have added where the programs are

• bin-cmp:/usr/bin/cmp -s
• bin-sdiff:/fip/gnu/diff -y

Logon List

Changes to other bits of Fip

• ~/tables/sys/SYSTEM file
  • If using multiple publications, add an 'ipformd' process for each publication, with its own input directory and parameter file
  • DO NOT START YET
  • Note the parameter file is always 'PROCESS.(code)' where code is the publication code.
    • abform local ipformd -Q -i 2form_ab -p process.ab
    • cdform local ipformd -Q -i 2form_cd -p process.cd
    • etc
  • If using a single publicaton
    • dpform tcform ipformd -Q -i dp_form -p process.dp
  • For balancing file over multiple systems add 'ipbalan' and 'ipsvrd' to the SYSTEM file * balan local ipbalan
    • svrd local ipsvrd -l
  • Normal Fip routing to get the data into the relevant directories

All other Fip systems

Remember to copy (manually) any files you have changed

Converting from an old Fip system

Adding a new Publication

This is currently a manual process :

Here is an example for a publication called 'piff'. Please note the case in the following : (i.e. it is important to make folders lowercase and files UPPERCASE) :

• Add the data folder
  • mkdir -p /fip/data/form/piff

• Add an entry in the setup file - /fip/web/setup/form.setup
  • pub:(short name) (full name)
  • The full name should have '\s' in place of spaces
  • pub:piff The\sDaily\sPiff

• If the publication already exists and there is an existing PROCESS file, copy it twice to
  • /fip/web/form/templates/PROCESS.EXTRA.PIFF
  • /fip/tables/form/PROCESS.PIFF

• Change the main /fip/tables/sys/SYSTEM (carefully)
  • fpiff local ipformd -Q -i form_piff -z process.piff
  • or for non-English, add the default locale
  • fpiff local ipformd -Q -i form_piff -z process.piff -E fr_FR@euro

• Add a destination in /fip/tables/sys/USERS for incoming, raw data
  • dformin_piff= DP:localhost DQ:form_piff SC:no DC:no

• Add the relevant destinations in /fip/tables/sys/USERS for output (this is site/system dependent of course)

• Route incoming data to DU:formin_piff
  • e.g. in /fip/tables/route/* files
  • ; Sport (Category R) * 1 c=r* * 2 c=rdi >w4sport * 2 c=rdr >w4racing_results+dformin_pub1+dformin_piff

• Create a logon that uses 'piff'

• Copy relevant parameter files to the correct tables folders
  • Sorry this is a manual task at the moment
  • Remember, if you copy 'xchg' files, that they may reference 'chrset' files too

• Add a logon which can access/create/manage jobs for that publication
  • Note a single logon can only access a single publication
  • Use the ZADMIN logon (or any other you have created with ADMIN access) to add or change an existing logon and select FORMAT for the Radius type and that publication

Adding a new Program for use in any DF stage

• In the main setup file, web/setup/form.setup, there should be five parameter lines added plus your comment lines

• Firstly think of a name to call it
  • p-program:(name) (full pathname to that program)
  • p-tables:(name) (full pathname to the directory where the parameter files are kept or '-' if none)
  • p-help:(name) (URL to the help page)
  • p-prod:(name) (Command line to run the program in production)
  • p-test:(name) (Command line to run the program in test)
  • e.g.
  • p-program:ipformat /fip/bin/ipformat
  • p-tables:ipformat /fip/tables/form/text
  • p-help:ipformat http://195.13.83.33/twiki/bin/view/FipDoc/Ipformat
  • p-prod:ipformat -i^i -D -S^o -p^p ^s
  • p-test:ipformat -i^i -xob -D -S^o -p^p -cformsave/^u^j^xT.^n ^s

• • Note the syntax for the command lines is :
    • ^i - Input file
    • ^o - Output file
    • ^p - Parameter file
    • ^s - Custom switches as specified in the 'Other Switches' panel on the UserInterface?
    • ^n - Stage number
    • ^j - Job name
    • ^u - Publication name
    • ^x - a single ^
    • Also ^ followed by any other character is a single ^

• • Note also, where possible, you want a copy of the output file for that section.
  • For 'ipformat' this is the '-cformsave/^u^j^xT.^n' which will create a file in spool/formsave, called 'pub^^job^T.9' which enables the user to click on 'output file' on the UI and display that file !

Cleaning out old projects

Sorry this is a manual task at the moment. See the [Under the lid] section for where files reside

Installation

Dependencies

• Fip base, in particular at least these version levels
  • sfflogon version 02f
  • ipformd version 22d
  • ipformat version 54m
  • ipformbl version 37w
  • iproute version 161u
  • ipxchg version 47l

• Web interface for a new build server
  • Apache/IIS/Domino - Please check with FingerPost if you have lost the documentation on how to setup these.
  • No changes are required at all for an existing FingerPost site, already using the W4 web interface

• Fip W4 with incoming wires/email etc. for raw data
  • Or at least some means of getting raw data into the DF modules.
  • If the W4 version is still using /fip/spool/w4data, an upgrade will be needed

Preparation

• Beforehand - Back everything up. Preferably upgrade on a spare or test system first

• form.setup pub (s) balance-group

• logons

Upgrading programs and scripts

Create a temporary folder and unzip the two files - binaries and tables under it

Copy the following files from the release media :

a. web/templates

• Templates and buttons
• cp -p web/templates/* /fip/web/templates

b. fip-cgi

• Main user interface scripts
• cp -p web/fip-cgi/* /fip/web/fip-cgi

c. pages/

• Cascading Style Sheets
• cp -p web/pages/css/* /fip/web/pages/css

d. setup

• Setup files - customer.setup and form.setup
• cp -p web/setup/* /fip/web/setup

e. xchgs

• Make sure you have the correct versions of the WEBSHOW* xchgs
• cp -p tables/xchg/WEBSHOW* /fip/tables/xchg

f. logons

• TAKE CARE If you are already running RADIUS_* versions, please take care that you do not overwrite anything here
• mkdir /fip/web/logon/lists
• mkdir /fip/web/logon/templates
• cp -p logon/list/* /fip/web/logon/lists
• cp -p logon/templates/form_* /fip/web/logon/templates
• cp -p logon/names/RADIUS* /fip/web/logon/names

g. bins

• While only the major binaries are required, you may want to upgrade all fip programs. Please turn all fip programs off beforehand!
• cp -p bin/* /fip/bin
• or
• cp -p bnew/sfflogon /fip/bin
• cp -p bnew/sfflogon /fip/bin
• cp -p bnew/ipformd /fip/bin
• cp -p bnew/ipformat /fip/bin
• cp -p bnew/ipformbl /fip/bin
• cp -p bnew/iproute /fip/bin
• cp -p bnew/ipxchg /fip/bin
• cp -p bnew/ipsgml /fip/bin

h. mkdir

• /fip/data/form
• /fip/web/form
• /fip/web/form/templates
• /fip/web/form/history
• Sort and test are ignored and can be removed

Setup

a. web/setup files

• Merge the existing customer.setup with the new one
• Please see the documentation for 'sfflogon' at http://195.13.83.33/twiki/bin/view/FipDoc/Sfflogon

• The parameter 'logon-list-file' should match the name of the file in logon/lists
• e.g. for the following, there should be a file called lists/FIPO.INTERNAL
• ; List of standard logons
• logon-list-file:fipo

b. logon

• Tune RADIUS_FORMAT for your wires
• Use the logon ZADMIN (default password hih2) to generate logons with

c. basic system - add 'ipbalan' and 'ipsvrd' if they are not already running

• So in tables/sys/SYSTEM, add 2 lines :
  • bal local ipbalan
  • svd local ipsvrd -l -P 9140

• Modify the BALANCE file in /fip/tables/sys for the format balance group
• ;
• ; 'BALANCE' - Balance files bewteen systems
• ;
• group:formadmin host:(fiphost1) ignore-localhost:
• group:formadmin host:(fiphost2) ignore-localhost:

Replace (fiphost1) with the name of the server of the first Fip box; (fiphost2) the second etc.

Note balance/svrd uses TCP port 9140 to balance data and parameter files, so if the two FingerPost servers are separated by a firewall, data through this port must be allowed.

Remember for any changes to the main SYSTEM file you can either use 'ip' and 'k all' and 'a all' to bring into operation or very carefully add the changes to the file /fip/x/SYSTEM.

d. Add the first publication - see 'Adding a new Publication' section above

Under the lid - stuff you dont need !

Where is everything is

Data is in

• /fip/data/form/(pub)/SECTION^DESK^JOB^(filename)

Parameter and history files are

• /fip/web/form/formats/(pub)/
• These files hold for each job :
  • stages - program, parameter files, extra switches
  • data files to test against
  • history
  • PROCESS file stubs

• /fip/web/form/history/(pub)/
• These files hold copies of each parameter file at each upgrade point of each job so we can use them to rollback

Folders/Naming Conventions

The carat (^) is used as a divider.

Job names can only have letter, numbers underscore and hyphens.

SECTION^DESK^JOB^P/T P-Production T-test

Next Phase

* dual/shadow running * able to pull file off w4 directly and run Test * multiple data files for test (ie be able to run same Test against several files to check) - for regression testing * locking * better conversion tools * completely isolated parameter files for separate pubs * send live test... * rollback to a test * show differences between versions of jobs - and parameter files.