Topal: GPG/GnuPG and Alpine/Pine integration

Introduction
Features
Important changes from previous stable versions
Installation and configuration
Topal usage
Fixing multipart emails
Notes
Author
Licence
To do
Version history
- Most recent changes

Introduction

Topal is a `glue' program that links GnuPG and Pine/Alpine. It offers facilities to encrypt, decrypt, sign and verify emails. See the list of features below.

Features

In-place decryption/verification, dealing with multiple blocks embedded in text.
Caching of output to reduce need for passphrase (at expense of storing decrypts and verification output).
Receiving of MIME RFC2015/3156 multipart/signed and multipart/encrypted messages. Top-level multipart items need some modification: see the README section `fixing multipart emails'. These features are available to any program that uses .mailcap files.
Sending of MIME RFC2015/3156 multipart/signed and multipart/encrypted messages. (Needs a patch to Pine/Alpine.)
Sending and receiving of the old application/pgp content-type (sending requires the same patch as the previous item).
Basic support for verifying S/MIME multipart/signed messages.
Offers user the opportunity to check output before sending it.
Remote sending mode for when reading email on a distant computer via ssh with secret keys on the local computer.
Rich configuration options.
Shortcuts for selecting keys, as well as general key selection routines when sending email.
Few arbitrary limits.

Important changes from previous stable versions

The previous stable releases were 0.7.2, 0.7.8, 0.7.9 and 0.7.13.6. Release numbering has changed since then. The subsequent stable releases were 55, 56, ....

Important changes in release 60

MIME sending now requires MIME-tool; mime-construct is no longer used. See compilation and installation.

Important changes in release 58

The default configuration no longer uses absolute paths.

Important changes in release 55

If you use a non-English locale, please check that Topal still works as expected (replaced code that fixed some locale problems).
The Alpine patch is based off my old Pine patches, but does a little more. You will need to set the Enable Topal hack for OpenPGP/MIME messages option in the hidden configuration list. Bug reports welcome.
The --fix-email wrapper no longer creates a multipart/alternative: it creates a multipart/misc wrapper instead. Please check that your procmail recipe includes a suitable backup in case this doesn't work for you.

Important changes in version 0.7.10

The recommended procmail recipe has been changed.

Important changes in version 0.7.8

topal-fix-email and topal-fix-folder have been replaced by the main topal binary. Change topal-fix-email in your .procmailrc to be topal --fix-email. (Or add symlinks: the binary checks what it has been called as.)

You must clear your cache otherwise the changes made for inline-separate-output (added in version 0.7.8) will break (this occurs regardless of whether the option is on or off). This new feature shows the GnuPG/Topal output separately, then hands back the decrypted or verified output without any wrappers. This makes it more suitable for dealing with attachments (but you need to set it manually via topal -config).

Finally, the send menu has a new option: `Pass through unchanged'. This does nothing to the message so, you can always have Topal invoked as a filter for sending.

Installation and configuration

Compilation and installation

To compile Topal, you need a working C compiler and the GNU Ada Compiler (GNAT). There is a makefile: simply type make. Type make install to actually install. The default location is /usr, so you'll need to be root to install. Alternatively, use make install INSTALLPATH=/usr/local to install into /usr/local. (Or use the more specific variables INSTALLPATHBIN, INSTALLPATHMAN, INSTALLPATHDOC and INSTALLPATHPATCHES.)

MIME sending requires the Topal version of mime-tool (included with the Topal sources, and compiled and installed at the same time using the Makefile). MIME viewing can be handled via metamail, run-mailcap, or by saving to a file in the ~/.topal directory and viewed with Alpine.

Pine/Alpine configuration

Assuming that the topal binary is installed in /usr/bin, set up the Pine/Alpine sending and display filters as follows:

display-filters=_BEGINNING("-----BEGIN PGP ")_ /usr/bin/topal -display _TMPFILE_ _RESULTFILE_

sending-filters=/usr/bin/topal -send _TMPFILE_ _RESULTFILE_ _RECIPIENTS_,
                /usr/bin/topal -sendmime _TMPFILE_ _RESULTFILE_ _MIMETYPE_ _RECIPIENTS_

You can choose either or both of the sending filters. The -sendmime option allows the user to choose the MIME type of the outbound email. (Legacy fixes are in place that make -decrypt and -verify behave the same as -display.) Note that _RECIPIENTS_ should be last.

For -sendmime to work, you will need to patch Pine/Alpine. There are patches for versions 4.44, 4.50, 4.53, 4.58. 4.60 and 4.64 of Pine. (They're all more-or-less the same patch.) cd into the pine4.xx directory and use the patch command.

There are patches for Alpine: versions 1.00 and 1.10. Please note that the Alpine patches also modify Alpine's configuration. There is a hidden preference `enable Topal hack' (enable-topal-hack) that you need to switch on.

It doesn't seem to have broken anything else.... It seems to work for sending via an SMTP server - it might break for sending via /usr/lib/sendmail (if it does, please send me a debug trace by invoking pine with `-d 9').

You can also add --read-from _INCLUDEALLHDRS_ before send and -sendmime. This makes Topal attempt to guess a suitable key for signing and self-encryption. If multiple possible keys match, then you'll be offered a menu of the keys.

Mailcap configuration

To decode MIME RFC2015/3156 multipart/signed and /encrypted messages requires the assistance of metamail. Add in either the user mailcap configuration (.mailcap) or the system configuration (/etc/mailcap) the lines

multipart/signed; /usr/bin/topal -mime '%s' '%t'; needsterminal
multipart/encrypted; /usr/bin/topal -mime '%s' '%t'; needsterminal
application/pgp; /usr/bin/topal -mimeapgp '%s' '%t'; needsterminal

Procmail configuration

In your procmailrc, add the recipe:

:0fw
| /usr/bin/topal --fix-email

This examines all inbound emails. Those with top-level multipart/signed or multipart/encrypted MIME types are modified to add a multipart/misc wrapper so that Pine/Alpine can hand it off to Topal. All other emails are left unchanged.

I strongly advise that you also use one of the backup recipes from the procmail manual. See also the notes in fixing multipart emails.

Topal configuration

Create a directory called `${HOME}/.topal'. This is currently hard-coded into Topal. Create the basic configuration file by running topal with the -dump or -default options. This file should be named `config'.

All .topal files are silently ignored if they cannot be found. Comments begin with a # in the first column, and run to the end of a line. They are totally ignored and are not currently preserved. Parsing errors cause an exception.

If you want to include strings with spaces, you'll need to quote them with double-quotes ("). Double-quotes themselves can be included by `stuffing' ("").

Topal usage

Help!

-help as the first argument dumps a help message.

The help message is derived from the help.txt file (included at compile time).

See help.txt for information on non-Pine use of Topal.

Send email to me if you're really stuck.

Interactive configuration

-config as the first argument brings up the configuration menu.

This menu is also available when sending (so that the signing key can be changed).

Decryption/verification

Depending on configuration, Topal will either ignore the file altogether, ask you what you want to do with it, or proceed to process the file automatically.

GPG will ask you for your passphrase when it needs it.

Caching is in place; the results of decryption and verification are (subject to configuration) saved in ~/.topal/cache. The results of caching mean that you won't be repeatedly asked for your passphrase, at the expense of storing decrypts in the clear.

Be warned: Topal often invokes less to view something. So you'll need to use q to get out of it. metamail is called for anything after MIME processing.

A new option (for version 0.7.8) called inline-separate-output concerns inlined (i.e., not MIME) messages. If the option is on, then the Topal/GnuPG output will be shown to you by less. Then the decrypted or verified output will be handed back to Pine/Alpine. This is the way to approach attachments. However, you will normally want to keep this option off, because if you're reading (for example) BugTraq mailings, then it will want you to hit q an awful lot....

Sending

If you choose the `Topal' filter when sending, you will be offered a menu

  (lkr) List/edit current recipient keys   (@) Add own key 
  (n) Pass through unchanged  (o) Configuration  (q) Abort
  Execute GPG: (e) Encrypt (s) Sign-encrypt (c) Clearsign

Above that, it will indicate picking up keys for each recipient email address. It will also add a key `for self'.

`Abort' tells Pine/Alpine you don't want Topal to process the email anymore.

`Pass through unchanged' does nothing to the message. This means that you can always have Topal invoked for sending.

`Add own key' adds an `encrypt to self' key. (It is added by default, but if you remove it, this is a quick way to restore it.)

The three `execute GPG' options use the current settings and execute GPG. You will be asked to confirm the command-line, and after processing, less is invoked to visually check that the desired result has been achieved. Again, a confirmation is asked for.

If -sendmime was used, then a menu will ask

  (p) Inline plain text  (a) application/pgp  (m) multipart/* 
  (q) Abort

offer a choice of three MIME types. Don't use (a) unless you really know what you're doing. If you are signing and encrypting, a further choice,

(e) multipart/* encapsulated

will be offered. This encapsulates a MIME signed message inside an encrypted message. Otherwise, we do both operations at once. (If you choose `clearsign' and `multipart/*', then all trailing blank lines will be deleted. Note also that Pine/Alpine appears to delete trailing whitespace in trailing blank lines.)

`Configuration' offers the same menu that is available from the -config option.

`List current recipient keys' offers a list of recipients:

  Select key, or (dq) to quit and return to main send menu
  or (s) to select a key after searching in the main keyring
  or (ak) to add keys from the main keyring (not recommended, use `s')
  Displaying choices 1 to 2 of 1 to 2    (<,) page up   (>.) page down 
   1 - 50973B91 2000-12-19 Dr Phil Brooke (at home)     
   2 - 9DAF9B5C 2005-10-21 Dr Phil Brooke

`Quit and return to main send menu' sends you back to the first menu.

`Add key from main keyring' prompts you for a search pattern. It will do a general search on your GPG keyring and add all matching keys. Beware of just pressing enter - it will select all keys on your keyring.

A better alternative is to use the `select after search' option. This also does a search on your GPG keyring, but then you must select one key to be added to your list of recipients.

Selecting a key will offer a third menu (a similar menu is offered when selecting a single key):

  Key: 50973B91 2000-12-19 Dr Phil Brooke (at home)     
  (d) Display details of key with less, (v) Verbosely
  (r) Remove key from list   (kql) Return to key list

`Return to key list' takes you back to the second menu.

`Display details of key (less)' simply uses GPG to list the key details via less. You'll need to use `q' to get out of less.

`Verbose details of key (less)' pipes verbose output from GPG for this key into gpg. You'll need to use `q' to get out of less.

`Remove key from list' removes the key from this recipient list.

Command-line usage

If you invoke Topal on the command-line with a filename as an argument, it will offer the sending functions on that file. It doesn't actually send anything: instead it allows you to encrypt, sign, etc. the message. You have a choice of overwriting or preserving the original file (this bit is case-sensitive).

The main purpose of this mode is for encrypting or signing attachments before they are attached to the message in Pine/Alpine. Beware that Pine/Alpine does not feed the attachments to a sending filter.

MIME functions are not available in this mode: it makes no sense.

Remote and server mode

Suppose you are reading your email on a remote host via ssh (as I often do). You now want to compose an email and sign it, but your secret key is only accessible on the local computer. Topal has rudimentary support for this (primarily to support my style of working). This comes in two parts: a `server' mode to run on the local computer (with access to the secret key) and a remote option in the sending menu.

The server mode (on the local host) is started by running topal -server. This is where GPG requests for signing are made.

When sending, you can choose `remote'. This prompts for the host to connect to using ssh/scp: this host should be running the `server'. The files are sent to the local server, processed by the server, then the results are copied back. ssh and scp are both used: because they're used repeatedly, you might want to use key-based authentication and have the key added to a current ssh-agent.

There isn't a remote mode for receiving: my approach is to use unison (or some other file synchroniser or a simple scp) to move the email(s) concerned, then view them on the local computer.

Fixing multipart emails

Two scripts used to be included with topal (long ago): topal-fix-email and topal-fix-folder. They have been replaced by the --fix-email and --fix-folder command-line options to the main binary.

topal --fix-email modifies any email that is (at the top level) a multipart/signed or multipart/encrypted message. It creates a multipart/misc message instead: this revised message is simply a wrapper version of the original message so that Pine/Alpine can pass the signed or encrypted part to Topal.

Usage:

topal --fix-folder <folder> ...: This fixes the old email folders you may have.
topal --fix-email: Takes no arguments; it accepts a single email on stdin. Ideally, it should be invoked by procmail (see the configuration section above).

topal --fix-email has a simpler mode (--simple) where it pretends that there are two MIME content types: `application/x-topal-encrypted' and `application/x-topal-signed'. You might prefer using this.

Why do we need this? If we just set the .mailcap file for, say, multipart/signed, then Alpine (at least version 1.00) is unable to handle a top-level multipart/signed email: an error message starting `Can't find body for requested message' is seen. But multipart/signed inside a multipart/mixed (or multipart/alternative, etc.) can be successfully handed-off to Topal.

Replying to such messages is a pain: you'll have to save off the actual message and read it in. Suggestions on fixing this are welcome....

See Workaround.Fix_Email in the sources for more details.

Notes

The Pine/Alpine patch, and sending other attachments

What does the patch to Pine/Alpine do? It removes some of the safety checking when changing the content-type (_MIMETYPE_) in a filter. Normally, if the returned content-type is not text/*, then the entire content-type is dropped.

The patch instead adds a flag, `topal_hack', and sets this if the returned content-type is not text. From time-to-time, we pretend that the body is normal text. We take a little care to check if this message is already a multipart message, so hopefully, the normal sending of attachments still works.

Key IDs and keylists

Topal internally lists keys by their fingerprint. It uses GPG to look up key fingerprints by using whatever GPG can cope with.

Duplicate keys are silently suppressed. Removing a key only removes one instance, if somehow you've coerced Topal to list duplicates (which is quite easy, since adding a key with its short key ID, and the same key with its fingerprint will add two identical keys).

The way that Topal chooses the keys is as follows:

For each recipient email address (supplied by Pine)
1. For each matching line in keylist, use the key ID to get a fingerprint, and add the key to the list.
2. In there are no matching lines in keylist, try to get a fingerprint via just that email address (but exclude `xk' configuration entries).

The keylist is a way to say, `for this particular email address, use this particular key'. In your config file, include lines such as

ake=50973B91,philb@soc.plym.ac.uk
ake=50973B91,pjb@lothlann.freeserve.co.uk

These mean `use key 50973B91 for the the given email addresses. Similarly,

xk=50973B91

means `don't use key 50973B91'. There are also similar sake and sxk options for the secret key selection (via --read-from) (although the testing of the secret key listings is less thorough so far).

Errors

Bad things happening should result in Topal setting its exit status to `failed', so Pine should detect this and not send your email.

Bug reports are welcome: send them by email to me (contact details below).

Decrypting attachments

If an attachment is a plaintext PGP ASCII-armoured message, then Topal will be invoked by Pine. You probably want to say `no' when asked here (beware of the configuration options here). Otherwise, you'll get a decrypted file with the original attachment filename, plus the various Topal headers.

Locale problems

GPG does not do any encoding of input data. This means that the encoding is dependent on Pine/Alpine and Topal. If a message is sent with one encoding and received by a user running in a different locale, then we might end up with a good message not verifying (i.e., bad signature).

I currently have no way to automatically fix this. However, the --ask-charset option will ask during inline decryption/verification if you want to change the encoding. If you know that the message was written by a UTF-8 user (and you're in a different locale), this might help. (This only happens if a bad signature is returned.)

I know it's a kludge. I'd be interested to hear success and failure reports.

Cleaning up the cache

You might want to run something like

find ${HOME}/.topal/cache -mtime +7 | xargs rm

to remove all the cache files that are a bit old (in this example, 7 days old or older).

Remote and server mode

When remote is invoked in a sending menu:

The host has to be chosen for ssh/scp.
Because topal might be outside the normal path, you'll be asked for that too.
The sender scp's the relevant files into .topal/server.
The sender calls ssh (server) -remotesend ... or ssh (server) -remotesendmime ....
The invocation of -remotesend or -remotesendmime triggers the server to run a new instance of Topal on the local computer.
When that instance is finished, the relevant files are copied back, along with the return value.

New releases

To be notified of new releases of Topal, send an email to me.

Release numbering

The old release numbering was making less sense to me. New releases are simple integers. In the event that an earlier release is modified, I'll then add extra components to the release number.

Author

Phil Brooke wrote this, partially out of boredom, but mostly because he wanted a GPG/Pine add-on to do exactly what he wants. There are many similar programs.

If you like this program, please tell me. If you'd like it better with changes, please tell me what changes you want. If particular items on the `To do' list are important to you, let me know. In particular, if you find bugs, feel free to tell me the details by email.

This package is released under the GPL: see the file COPYING.

I can be emailed on pjb@lothlann.freeserve.co.uk

My key ID is 0x50973B91; the key is available from web pages and public key servers.

If you want to send snailmail to me, email me for my (physical) address.

Licence

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

To do...

Planned releases:
- Add S/MIME support via gpgsm.
- Improve attachments code (and add some documentation).
Better error handling, particularly when missing dependencies such as mime-construct or metamail.
Add signal handlers.
Catch GPG keyboard interrupt.
Should we check that the infile matches the cache file even if the MD5 hash matches? (We'd need to store the infile in the cache as well.)
Check through code: all external calls should check return values.
Refactor code.
Add interrupt option at very beginning of execution? (which would bring up the configuration menu?)
Associate extra options with particular keys?
Configuration routine for managing keys/config/keylist?
Implement rest of configuration menu.
Make a much nicer interface all round....
Separate out all the constant strings -- so that we can have internationalization.
Context-sensitive help throughout (modify mkhelp to create multiple procedures, or do it by number?); add COPYING option?
More receiving/decrypt options: include both plaintext and ciphertext.
Add periodic cache cleanup when Topal is invoked?
Add logging for workaround mode (report time of email processing (include PID); indicate if the file was changed or not)?

Version history

Look in release for the current release number.