The Data formatting module

Data Formatting

The FingerPost Data Formatting module is designed to take any type of input file(s) and reformat them into one or more output feeds. In addition to the formatting capabilities, it controls the processes, flags non-arrivals and can be used to initiate further processes downstream.

It can run automatically for those feeds that arrive at standard, regular times such as share prices, race cards and football league tables.

It can be run manually for one-off events such as the New Year Honours lists, London Marathon runners name lists and University course lists.

It can be run in mixed mode for events such as semi-automatically for General Election results etc.

Need for control

An important feature of the module is the control aspect. As it has been designed to run automatically, the control program tracks all incoming and outgoing files. If an incoming is late or wrongly formatted, it is flagged for the System Manager to take appropiate action. Similarly it can control the delivery process so that files that have not been sent to their destination within a certain time limit are also flagged.

Need for control

All key information is held in parameter tables allowing quick and easy setup and maintenance by qualified personnel.

Formatting features

Different input and output file types Several file types are catered for.
  • variable fields
  • comma separated files (CSV)
  • record and field delimited files
  • fixed-length record types
  • precedent character file types.
  • Input and output can be either of dissimilar or similar formats.
  • Merge and diffusion of files
  • One input feed can be split into several files each for several clients. Several input feeds can be merged and/or resplit into one or more output feeds.
  • Concept of files, blocks, records and fields
  • The Formatting module has four levels of data organisation, files, blocks, records and fields. Several different types of each are valid.
  • Reordering the sequence
  • Reordering of fields within a record.
  • Resorting records or blocks in a file.
  • Including text from other sources
  • Inclusion of fixed text from a parameter file eg for markup. Inclusion of variable text such as date, time. Inclusion of text from other files. Table lookup eg : to change IATA priority codes to IPTC.
  • Text filters
  • On files, blocks, records and fields. Standard stripping routines include:
  • Make upper/lowercase
  • Strip leading/trailing/all spaces
  • Strip leading/trailing/all punctuation
  • Strip leading/trailing/all special characters
  • Pad block/record/field
  • Before or after main processing
  • Character set and string replacement
  • Strings can be matched on a per field basis and may be matched recursively
  • Use of flags
  • Flags can be used to ignore records/fields/text or to alter the type of processing for a series records/fields.

For a more detailed overview please see the DataFormattingModuleGuide

Notes and Comments

Please leave your name and a contact email address .....
Topic revision: r1 - 15 Feb 2005 - 22:41:40 - DotFingerPost?
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback