IPSMTP

ipsmtp

This feeds and receives from Internet Mail.

It sends to an SMTP server and can receive either on a local server (Unix only)
or from a POP3 server.

We define OUTGOING as FIP -> SMTP and the reverse (SMTP or POP3 -> FIP)
INCOMING.

Routing Files Outbound
----------------------
Messages for sending are put in the input queue, usually spool/2smtp, with the
progs.FipHdr field DA containing one or more Mail Address (separated by commas if
multiple) - such as wizzo@fingerpost.co.uk.

Messages can be sent directly to the spool/2smtp queue, routed normally by Fip
or as a group client for Syndication by IPPOST.

In the latter case, the client is specified as type:unixmail and there is a
special DU called TOUNIXMAIL in USERS which has the syntax :
	TOUNIXMAIL=	DP:(hostname)	DQ:2smtp	DC:(whatever)
	where hostname is the name of the progs.UnixBox doing the transfer
	and whatever is the character set required.

The From field or sender is the header field SU or, if it exists, the SA
(Source Author) which is formatted as SA@SU. SU is normally the Source system
or type while SA is the normally the logon or sender. If the SA is an Atex
style 'LOGON ;dd/mm,hh/nn' then only the logon is used.
So if the SA:fippo and SU:liverpool-fc.co.uk, the sender becomes
		fippo@liverpool-fc.co.uk
If the SA is ALWAYS the full address - like SA:hedgehog@flea-pit.com, then use
the '-F' input switch to NOT add the SU header field.

If neither SU nor SA exist, the default sender is 'fip' unless overridden by
the 'defsender:' in the parameter file. Note that this name must be in the
passwd file as a valid logon.

The data of the outgoing messages will be converted if 8 bit or control chrs to
the normal '=XX' hex format used.

Routing Files Inbound
---------------------

Messages from the Internet are sent to the special DU, FIPMSG which is either
hardcoded to a specific destination, or more usually, directed to IPPOST to
detetmine if the recepient is on the editorial system OR is a mailbox (IPBOX)
customer.


Parameter File
--------------
The parameter file, normally tables/mail/SMTP, is read for the logon names to
scan on a regular basis plus the destination to send these to. The syntax of
the parameter file is :
	; comment
	defsender: (default sender)				default: fip
		Used ONLY if there is NO fipHdr field SU nor SA.
		This is used for the 'From:' field for outgoing messages.
	forcesender: (Force sender to this logon)		default: no
		This is the default reply-to address. It must be a valid
		logon and domain. It is also used for the 'From:' field
		for ALL outgoing messages if there is NO 'forcefrom' (see below)
		or progs.FipHdr field 'DJ' in the file to send irrespective of
		whether there is a progs.FipHdr SA or SU.
		It should be JUST the address :
			forcesender: lucy@sky-with-diamonds.com
	forcefrom: (Force the From field)
		This is the From field in the outgoing file which does not have
		to be the same as the sender's address.
		If using the '-h' host input switch it can contain a 'pretty'
		version :
		eg	forcefrom: "MP3 king Steinhoff" <dohrayme@singalong.com>
	defdest: (default Fip Destination (DU progs.FipHdr field)	default: FIPMSG
	chrset:	(Source character set ie SC header field)	default: ascii
	newDA: progs.FipHdr field to replace the DA field which	default: DA
		holds the email address of the person to sent to.
		eg	newDA:DZ
		This can also be an combination field :
			combie:QA	DA|DZ|NA
			newDA:QA
	hash-in-address: (progs.FipSeq Chr)
		Some old-style email addresses have a leading hash/pound sign. But a hash is
a field sepearator in FipHdr. So another chr is used in the DA progs.FipHdr and
mapped at the last moment.
		default is \243
	newDI: progs.FipHdr field to replace the DI field which	default: DI
		is used for the 'Subject' field.
	newDJ: progs.FipHdr field to replace the DJ field which	default: DJ
		is used for the 'From' field.
	balance-seqno:  Send the Sequence number to this Balance Group
			(see 'ipbalan')
	balance-delete: Delete the Data on remote systems once sent
			using this Balance Group (see 'ipbalan')
	forcedomain: (domain name)				default: do not
		Strip any domain in the progs.FipHdr field SA and add this one
		in its place.
			forcedomain:commies.com
		So if the SA:albert@redsunderthebed.com,
		then the from field becomes
			From: albert@commies.com
	strip-domain-on-send: Send only the logon/mail name	default: domain too
		and strip any domain name.
	defsubject: (default Subject line if nothing specified)
			default: "Message for you"
	att-message: Message to replace an attachment with.
		default:** Attachment has been Stripped, pls contact System Managers for
(name of attachment file)
	log-line: (progs.FipSeq) Additional information for the log file entry in ALL
	extra-log-fiphdr: Fipseq progs.FipHdr fields to add to any transmission log
		This is triggered on a file by file basis by the progs.FipHdr field
		ZSMTP_LOG:(dest) in the file to send - (dest) being a valid
		DU in tables/sys/USERS. Use this to pass progs.FipHdr fields from
		the incoming file to the outgoing
		default: none
		eg.	extra-log-fiphdr:\n\SI:\SI\n
	send-new-fiphdr: Fipseq progs.FipHdr fields to add to any outgoing file to act as
defaults etc
	send-extra-fiphdr: Fipseq progs.FipHdr fields to add to any outgoing file to act as
defaults etc
		The NEW variant is added BEFORE the real FipHdr, so can be used for Defaults
(and the real
		progs.FipHdr fields can override these).
		The EXTRA variant will add at the end of the file's progs.FipHdr and so will
override
		any setting,
		This is triggered on a file by file basis by the progs.FipHdr field
		default: none
		eg.	send-extra-fiphdr:\nZSMTP_TYPE:text/plain\n

	stream-connections:yes				default: no
		Do NOT drop the connection/reconnect for every file.
		Once a connection is up to a mailserver, any subsequent file
		for the same server is sent in the same transmission.
		(using SMTP only)

	fiphdr: (Map incoming mail field to a Fip Hdr field)	default: none
		fiphdr:DI	Subject
		fiphdr:AZ	Message-Id
	ignore: (Mail header field to always ignore)	default: none
		ignore:	Status
		ignore:	Apparently-To
		ignore:	X-Msmail-Priority
	decode:
	unencode: Decode any base64 sections.   default: no
		decode:	 This will NOT decode attachments
		decode:text	This will NOT decode attachments
		decode:all	This will decode body AND attachments
	maxattempts: (no of attempts before the file is sent to woops)	default: 10
		This checks the progs.FipHdr 'DO' which is managed internally. (UNIX only)
	logeachfile:(dest) Send a Success/failed msg to this destination for each
file.
			There is no default.
			This log file is just a progs.FipHdr with the following extra fields :
				DR-File Sent OK	 DR:ok or DR:error
				DG-Will Retry later	DG:retrying or DG:stopped
				DT-Some message text	DT:No connection
				DO-Number of last attempt	 DO:5
				HT-Date and Time	HT:25 Nov 98 15:35:25
			default: no log created.
			The DR and DG messages can be changed by using :
				log-dr-ok:(progs.FipSeq)
				log-dr-error:(progs.FipSeq)
				log-dg-stop:(progs.FipSeq)
				log-dg-retry:(progs.FipSeq)
			eg: log-dr-ok:File \VN sent to Remote Number \DA (\$h:\$n)
	msgeachfile:(progs.FipSeq) Additional information to add to the progs.FipHdr of the
'logeachfile'
			msg. This should be in progs.FipHdr format and be in FipSeq. It can be
			used to pass progs.FipHdr fields in the outgoing file into the log file.
			eg	msgeachfile:	DF:logdial\nSS:\SS\n
			default: nothing added
	default-smtp-host: (hostname or IPaddress)
		For sending emails out - Same as the '-h' Input Switch
	alternate-smtp-host: (hostname or IPaddress)
		a failover hostname for if the default fails
	connection-timeout: (secs)
		Max time in seconds for the connection to the SMTP server
		The call is aborted if it takes longer than this.
		If the remote is across the computer room, this time can be
		reduced to 15 or 20 secs; if over the internet, 120 secs is the norm.
		default: 120 for 120 seconds, minimum is 10

	send-single-smtp: send smtp in one file per session which is then cut and 
		reopened for the next file/session.
	hostname: (hostname to use)
		Cosmetic !
	timing-stats: yes/no
		Show timing stats if the incoming file has the HE/HR fiphdr
		default: no

	use-sx:
or	use-external-file:
		if there is an SX progs.FipHdr field with a path to the data file, use that rather
than the data in the input file.

	wrap-subject-chr: (progs.FipSeq chr)
		If the Subject field has this chr, wrap the line
		default: none
		This might have been added by ipxchg :
			z:convert-fiphdr-to-qp:DI:DJ:utf-8:\002
		wrap-subject-chr:\002
	force-subject-wrap: (No of chrs)
		Normally  subject lines at this total
		default: 76
		eg force-subject-wrap:76

	check-primary-server: pseudo-host name that is specified in
tables/sys/DEST_REDUN
		Use this when for POP3 accesses ONLY where ipsmtp on 2 systems is accessing
the same POP3 account(s) in a redundant way.
		It is used to decide whether the current host should be getting the files or
not.
		ie in the parameter file is
			check-primary-server:timersvr
		and in the DEST_REDUN is			
			; psuedohost	primary secondary
			timersvr	fip1	fip2
		and in the SYSTEM file for both fip1 AND fip2 there is  the same 'ipsmtp'
line with the same parameter file
		If fip1 is up,  'ipsmtp' on fip1 will always get while on fip2 it will just
check/loop.
		If fip1 is down 'ipsmtp' on fip2 will start processing the files.

	use-tls:yes/implicit/explicit/no
		The commends are for a ftp running over SSL/TLS on the remote server
		default is NO
		no		- normal, standard FTP on (normally) port 21 for the control
		yes or explicit - connect (normally) on port 110 in clear then use SSL for
USER, PASS and data
		implicit	- connect (normally) on port 995: use SSL for all conversations
	tls-auth: (XXX)
		AUTH type for TLS/SSL		 		 default: TLS

	ssl-method: (1,2,3,23,999)
		Version number to use for TLS/SSL		default: 999 for current default (2 or 3)
	ssl-password: (password)
	ssl-passwd: (password)					default: none
		Optional password if the handshake requires a shared secret
	ssl-cert: (name of a PEM certificate file)		default: none
	ssl-root-cert: (name of a root PEM certificate file)	defaunt: none
		Optional certificates - held in tables/ssl


Then for EACH logon you wish to track INCOMING mail from the Internet specify a
line begining :
		logon: (logon name)				default: none
	or	unix: (unixlogonname)				default: none

	These two parameters are identical.
	There is also a means, for progs.UnixBoxes only, to poll for ALL mail
	for ALL the members of a group (as in the /etc/group file) :
		group: (unixgroupname)				default: none

	Whether 'logon', 'unix' or 'group' there are other (optional)
	parameters:
		sendto:(newaddress)
		fiphdr:(FIP hdr sequence)
		striphdr:
		keephdr:
		pop3pwd: (Password for POP3)
	where	sendto allows you to specify another name for the DA field
			IPPOST will use this to route. By default
			the Fip Hdr field DA will hold the logon name.
		fiphdr allows you to add to the FIP hdr. This may be any
			- perhaps the DU field to change the destination.
		striphdr will strip the progs.SendMail header except for the
			From and Subject lines
		pop3pwd specifies the password to use IF you are accessing
			a POP3 server to receive incoming mail.  It is NOT
			required for progs.UnixBoxes scanning local mail queues.

To add a new logon :
	1. Read the 'passwd' file for the exact Unixlogon name - eg newboy
	2. Enter this name in SMTP file as :
		logon:newboy
	3. If this is the same name as in file ATXMSG, enter line
		newboy	edone
		where edone is the name of your editorial system
	4. If 'newboy' is really called 'dirtyoldman' on the 'atex'
		editorial system, the entries should be:
		SMTP file -	logon:newboy	sendto:dirtyoldman
		ATXMSG file -	dirtyoldman	atex

To bringup IPSMTP :

	1. Create logons as above
	2. Check you have destinations in the USERS file :
	FIPMSG=		DP:fipmain	DQ:post	 DC:atexeol
	MSGFORATX_EDSYS=	DP:edsys3	DQ:atexq-grp	DC:atexeol
	where 'fipmain' is the name of this UnixBox.
	Destination FIPMSG is generated by IPSMTP.
	Destination MSGFORATX_* is generated by IPPOST IF the name is in the FIPMSG
file.
	eg if the Editorial System is called ATEX (environment variable FIP_EDSYS) and
teh queue required is email-wir on system3 :
	MSGFORATX_ATEX= DP:atex3	DQ:email-wir	DC:atexeol
	3. Input Queue spool/2smtp needs to be added
	4. IPPOST needs to be running too (unless you change the FIPMSG DP/DQ to
somewhere else)
	5. Outbound Format is ATXMSG for IPGTWY/IPOUT/IP2WTR etc.
		Please check if this exists and has the correct setup

The current version can scan up to 1000 Unixmail boxes.

Where sections of progs.FipHdr fields are required or changes to the output style,
use
 keywords : fixed, partial, combie, optional, repeat, newdate and/or style.
(see The progs.SysA
dmin manual for more information).

	They are normally specified :
		fixed:QZ	1234543
		partial:QT	ST,3,2,U,<,>
		combie:QY	ep|na,(0000000)a
		option:QE	ep,11,7,s
		repeat:QK	XK,-,3
	or	repeat:QP	PK,,4,#X
		style:QS	XN,%.03d
		replace:QN	NN	abc=DEF def=GHI

For outbound traffic, progs.FipHdr fields used are :
Mandatory
	DA	holds one or more remote email addresses
		multiple addresses are separated by a comma.
		Can be overridden by 'newDA'
		Can be changed by 'forcedomain'
Optional
	SA	holds the sender's email address
		can be overridden by 'forcesender' (actual address)
		can be overridden by 'forcesender' (from field)
		can be defaulted by 'defsender'
	DI	holds the Subject: field
	DJ	holds the From: field
	ZSMTP_BINARY	for sending Binary files
	ZSMTP_ENCODE	holds the Content-Transfer-Encoding
		default to	Content-Transfer-Encoding:quotedprintable
	ZSMTP_TYPE	holds the Content-Type. no default
	ZSMTP_LOG	will send a log of the SMTP transmission to this
		destination (for SMTP outbound traffic using the '-h' input
		switch only)
	ZSMTP_HOST IPaddress or hostname of the mail server if NOT the default
			(for SMTP only).

Input Switches
--------------

Input parameters are : all optional :
	-1 : Get once only then stop for POP3 or Receive only.	default: spool
		Either -p (path) or -3 (pop3 server) must be used with this option
		This also incorporates -Q so no queu is scanned and no mail sent.
	-3 : Hostname where the pop3 daemon is running.	default: none
		Note -p and -3 are mutually exclusive.
	-a : queue for non-text attachments		default: none
		default is that attachments are passed through in text
	-A : destination for non-text attachments	default: none
		default is that attachments are passed through in text
	-C : do NOT convert unprintables to "=XX" format	default: do
	-d : do NOT delete each unix msg		default: delete
	-f : do NOT output the original filename as the first
		line of text				default: output name
	-F : do NOT output the original sending system (SU) as the From
		field					default: From: SU
	-h : Hostname where the smtp daemon is running.	default: that system
	-H : do NOT add an extra Mime Headers as the file
		is in the correct format	default: add To/From/Subject
	-i : Receive spooled queue if you are using 'sffmail'	default: using Path
	-k : alternate host name				default: none
		use this for checking primary/secondary for clustered systems where the
actual hostname differs from the cluster name
		ie if the host name may be fipcluster-a or fipcluster-b but there is a
cluster hostname of fipcluster
	-K : quiet mode - message if NOT the primary system	default: do not message
	-l : do NOT log each msg in or out		default: log
	-m : (ipchkmail folder name)			default: process inside ipsmtp
		Do not convert incoming SMTP and POP3 files immediately but pass them to
ipchkmail runing on this folder which does a better job! especially for
attachments-within-attachments

	-o : Next fip queue for incoming files		default: spool/2go
	-O : Done queue for spooled files out		default: none
	-p : Path on this system holding mailboxes 	default: depends on the OS
		Unix only. The full pathname MUST be stated.
		Note -p and -3 are mutually exclusive.
	-q : Send Queue to spool			default: spool/2smtp
	-Q : Do NOT send any mail and therefore do NOT scan any input queue
							default: scan
	-r : eolns for the data part should be CR NL	default: just NL
	-R : do NOT resolve hostnames	   default: do
		This will resolve any hostnames in the DEST_REDUN file (if there is one)
	-S : do NOT split files on the "From ..." line	default: do
	-t : scan time of queue				default: 3 secs
	-T : log the trace of each transaction		default: do not
		Normally a one line log of each file sent is stored in
		log/remote_trace with a name of 'date_Address'.
	-w : for file->Sendmail, set line length.	default: 76 + translate
		and do NOT translate unprintables to "=XX" format
		To NOT wrap text and no translations -w 0
		This can be ignored for a single file if it has
		the progs.FipHdr field ZSMTP_BINARY:
	-W : for files outbound, set JUST the line length	def: 76 + translate
		but still translate unprintables.
	-z : parameter file				default: mail/SMTP
	-v : display version number and exit.

Now a statement of sanity (?) about incoming mail from the Internet :

If like most of the civilised world you prefer to keep well away from anything
'sendmail' can throw at you, then let ipsmtp just grab its files from the
sendmail internal queue. Or even better use POP3 and forget about it all - no
setup, no nothing extra!

If you are unlucky enough to have a Sendmail specialist on your staff, before
sending for the men in white coats, point out there is a '-p path for unix
mailboxes' switch. Your guru should dictate (they always dictate - though they
claim they are working with your best interests in mind) where he is willing to
drop files for those logons you are interested in and add the -p switch to the
tables/sys/SYSTEM file. Easy huh !

Note there was a subtle change at version 15 which changed the default from
ATXMSG to FIPMSG.


Version Control
;016v10	09nov01 cleanup after failed connections better
	;a 16mar02 added remote_trace and -O done queue
	;b-c 03jun03 cleanup line-enders in mimehdr and streaming for smtp
		and added ZSMTP_HOST
	;d-i 22sep03 added send-extra-fiphdr: and stream-connections:yes
	;j-l 29jun04 added -1 and speedy
	;m-p 17sep05 added alternate-smtp-host and hostname: and log-line
	;q 10aug06 added remote_trace style timings to send file done msg as H4
	;r-t 21sep06 allow quotes in email address and wrap long subject lines
	;u 13apr07 added extra-mime-headers:
	;v4 06feb08 added hash-in-address ;2 modette for size of email list
		;3-4 added check-primary-host for POP3 and chkmail-folder:
		;5-6 bugette with send_smtp and logging long DA
		;7 3sep10 added use-sx
		;8-9 10nov10 bug in pop3 when Uidl has non-alnum
		;10 19may11 bugette when TLS does not work

(copyright) 2011 and previous years progs.FingerPost Ltd.

Links

• Smtpwire - can be used to replace sendmail for receiving SMTP mail

Examples

How to generate emails with attachments or html emails

• Ipsmtp Attachments Outbound - how to generate emails with attachments

Notes and Comments