New Features in Exim
--------------------

This file contains full descriptions of new features that have been added to
Exim, but have not yet made it into the main manual (which is most conveniently
updated when there is a relatively large batch of changes). The doc/ChangeLog
file contains a listing of all changes, including bug fixes.


Version 4.11
------------

 1. Support has been added for ldapi:// lookups, when Exim is compiled with
    OpenLDAP. This has been tested with OpenLDAP 2.0.25. What follows here
    applies only to OpenLDAP. If Exim is compiled with a different LDAP
    library, this feature is not available.

    An ldapi lookup connects to the LDAP server via a Unix domain socket,
    instead of over TCP/IP. Instead of a host name for the server, a pathname
    for the socket is required, and the port number is not relevant. The
    pathname can be specified either as an item in ldap_default_servers, or
    inline in the query. In the former case, you can have settings such as

      ldap_default_servers = /tmp/ldap.sock : backup.ldap.your.domain

    When the pathname is given in the query, you have to escape the slashes as
    "%2F" to fit in with the LDAP URL syntax. For example:

      ${lookup ldap {ldapi://%2Ftmp%2Fldap.sock/o=...

    When Exim processes an LDAP lookup and finds that the "hostname" is really
    a pathname, it uses the ldapi code, even if the query actually specifies
    "ldap" or "ldaps". In particular, no encryption is used for a socket
    connection. This behaviour means that you can use a setting of
    ldap_default_servers such as in the example above with traditional "ldap"
    or "ldaps" queries, and it will work. First it tries an ldapi connection
    via the Unix socket; if that fails, it tries a TCP/IP connection to the
    backup host.

    If an explicit "ldapi" type is given in a query when a host name is
    specified, an error is diagnosed. However, if there are more items in
    ldap_default_servers, they are tried.

    In other words:

      (a) Using "ldap" or "ldaps" with a pathname forces the use
          of the ldapi interface.

      (b) Using "ldapi" with host name causes an error. Using "ldapi" with no
          host or path in the query, and no setting of ldap_default_servers,
          does whatever the library does by default.

    A hint about slapd: to make slapd listen on a Unix socket, start it with
    an option like this:

      -h 'ldapi://%2Ftmp%2Fldap.sock/????!x-mod=www'

    That creates the socket /tmp/ldap.sock. If you want to listen on a TCP/IP
    interface as well, you need an option such as:

      -h '"ldapi://%2Ftmp%2Fldap.sock/????!x-mod=www  192.168.34.32"'

    The double quoting is important. Without it, you get only one listener.

 2. There is a new option called smtp_return_error_details, which defaults
    false. In the default state, Exim uses bland messages such as
    "Administrative prohibition" when it rejects SMTP commands for policy
    reasons. Many sysadmins like this because it gives away little information
    to spammers. However, some other syadmins who are applying strict checking
    policies want to give out much fuller information about failures. Setting
    smtp_return_error_details true causes Exim to be more forthcoming. For
    example, instead of "Administrative prohibition", you might get:

      550-Rejected after DATA: '>' missing at end of address:
      550 failing address in "From" header is: <user@dom.ain

 3. There is a new option called check_postmaster that can be used whenever
    a callout option is set on an address verification in an ACL. For example:

      accept  verify = sender/callout/check_postmaster

    This causes Exim to do a double check. After it has verified the address in
    an SMTP connection, it also verifies that the same server will accept mail
    to "postmaster" at the same domain, within the same SMTP connection. If
    postmaster is rejected, the verify condition fails.

 4. The ${quote_ldap: operator has been reworked, because it seems that two
    different kinds of quoting are required for LDAP queries. There are now two
    different quote operators, and they work as follows:

    ${quote_ldap:....}

    This operator is designed for use on strings that are part of filter
    specifications. Conceptually, it first does the following conversions on
    the string:

      *   =>   \2A
      (   =>   \28
      )   =>   \29
      \   =>   \5C

    in accordance with RFC 2254. The resulting string is then quoted according
    to the rules for URLs, that is, all characters except

      ! $ ' - . _ ( ) * +

    are converted to their hex values, preceded by a percent sign.

    ${quote_ldap_dn:....}

    This filter is designed for use on strings that are part of base DN
    specifications in queries. Conceptually, it first converts the string by
    inserting a backslash in front of any of the following characters:

      , + " \ < > ;

    It also inserts a backslash before any leading spaces or # characters, and
    before any trailing spaces. The resulting string is then quoted according
    to the rules for URLs.

    Examples:

    1. ${quote_ldap: a(bc)*, a<yz>; } yields

       %20a%5C28bc%5C29%5C2A%2C%20a%3Cyz%3E%3B%20

       Removing the URL quoting, this is (with a leading and a trailing space):

         a\28bc\29\2A, a<yz>;

    2. ${quote_ldap_dn: a(bc)*, a<yz>; } yields

       %5C%20a(bc)*%5C%2C%20a%5C%3Cyz%5C%3E%5C%3B%5C%20

       Removing the URL quoting, this is (with a trailing space):

         \ a(bc)*\, a\<yz\>\;\

 5. Another LDAP change: When a DN is quoted in the USER= setting for LDAP
    authentication, Exim now removes any URL quoting that it may contain before
    passing it LDAP. Apparently some libraries do this for themselves, but some
    do not. This has two advantages. It makes it possible to use the same
    ${quote_ldap_dn:...} expansion (see 4 above) for USER= DNs as with DNs
    inside actual queries. It also solves any problems with spaces inside USER=
    DNs. For example, a setting such as

      USER=cn=${quote_ldap_dn:$1}

    should work even if $1 contains spaces.

 6. The new variable $ldap_dn (available when Exim is compiled with LDAP
    support) contains the DN from the last entry in the most recent successful
    LDAP lookup.

 7. Changes have been made to the way that hosts_max_try in the smtp transport
    works.

    Hosts that are past their retry limits are no longer counted for
    hosts_max_try. This means that when some hosts are in this state, a greater
    number of hosts are tried than before, but this is the only way to ensure
    that all hosts are considered before timing out an address.

    When the hosts_max_try limit is reached, Exim now looks down the host
    list to see if there is a subsequent host with a different MX. If there
    is, that host is used next, and the current host is used but not counted.
    This heuristic is added to cope with the case of a domain with a retry rule
    that hardly ever delays any hosts.

    With something like the default rule, if there is a long list of hosts with
    one MX value, and a few with a higher value, the higher MX hosts will
    eventually get used because those at the top of the list won't have reached
    their retry times. However, it is common to put a short retry time on
    domains for large ISPs. These are exactly the domains that tend to resolve
    to long lists of hosts. The short retry time means that the lowest MX hosts
    are tried every time. This will be in a different order because of random
    sorting, but the higher MX hosts never get tried at all. The new heuristic
    makes Exim try at least one address from each MX value, even when the
    hosts_max_try limit is reached.

 8. There are two additions to the building and installation facilities:

    (1) Local/Makefile has a new setting called SYSTEM_ALIASES_FILE, defaulting
        to /etc/aliases. This is used only when installing the default
        configuration, as the name of the file in the system_aliases router.
        If the file does not exist, a dummy is created (this is not a change;
        it used to happen for the fixed /etc/aliases previously).

    (2) You can specify ROOT=xxx before "make install". This has the effect of
        prepending xxxx to all file paths, which makes it possible to install
        Exim for testing (or other special purposes) in a private bit of file
        system.

 9. The output from exiqsumm now has a total line at the bottom.

10. After verify=recipient in an ACL, the value of $address_data is the last
    value that was set while routing the address. This applies even if the
    verification fails.

11. Eximstats 1.21 has a -merge option for merging reports.

12. The appendfile transport now has an option called quota_directory, which
    defines the directory to check for quota purposes when delivering into
    individual files (the default is the delivery directory, or, if
    maildirfolder exists in a maildir directory, its parent).

13. Macro expansion is now applied to physical rather than logical lines in the
    configuration file, and it is done before checking for comments and
    .include lines. Previously, macros could not be used as part of .include
    lines; now they can. However, there is one incompatible change: previously,
    if a macro that expanded to "#" was used to turn a line into a comment, the
    comment applied to any continuation lines. Now it applies only to the
    physical line on with the macro appears.

14. The -D command line option must now all be within one command line item.
    This makes it possible to use -D to set a macro to the empty string by
    commands such as

      exim -DABC  ...
      exim -DABC= ...

    Previously, these items would have moved on to the next item on the command
    line. To include spaces in a macro definition item, quotes must be used, in
    which case settings like

      exim '-D ABC = something' ...

    can be used.

15. There are two new debug selectors:

    The debug selector "timestamp" causes the current time to be put on the
    front of all debug output lines. This can be useful when trying to track
    down delays in processing.

    The debug selector "pid" causes the process id to be put on the front of
    all debug output lines (after the timestamp, if present). This selector is
    forced when debugging is turned on for a daemon, which then passes it on to
    any re-executed Exims. Exim has always added the pid to debug lines when
    several remote deliveries are run in parallel, and this has not changed.

16. There's a new utility called exiqgrep (by Matt Hubbard). It's a Perl script
    that runs "exim -bpu" and greps the output messages that match various
    criteria.

17. The ${sha1: operator has been added to compute SHA-1 digests. This works in
    the same way as the ${md5: operator. Also, the crypteq comparison now
    recognizes {sha1} as an encryption type. The encrypted password must either
    be 40 characters long if given in hex, or 28 characters long if given in
    B64 encoding. Any other length causes an expansion error.

18. There is now support for GnuTLS, as an alternative to OpenSSL. To build
    Exim to use GnuTLS, you need to set

      USE_GNUTLS=yes

    in Local/Makefile, in addition to

      SUPPORT_TLS=yes

    You must also set TLS_LIBS and TLS_INCLUDE appropriately, so that the
    include files and libraries can be found.

    There are some differences in usage when using GnuTLS instead of OpenSSL.

    (1) The tls_dhparam option is ignored, because GnuTLS has no facility for
        varying its Diffie-Hellman parameters.

    (2) GnuTLS uses RSA and D-H parameters that take a substantial amount of
        time to compute. It is unreasonable to re-compute them for every TLS
        session. Therefore, Exim keeps this data in a file in its spool
        directory called gnutls-params. The file is owned by the exim user and
        is readable only by its owner. Every Exim process that start up GnuTLS
        reads the RSA and D-H parameters from this file. If the file does not
        exist, the first Exim process that needs it computes the data and
        writes it to a temporary file which is renamed once it is complete. It
        does not matter if several Exim processes do this simultaneously (apart
        from wasting a few resources). Once a file is in place, new Exim
        processes will start using it.

        The parameters involved should be recalculated periodically, the
        frequency depending on your paranoia level. Arranging this is easy;
        just delete the file when you want new values to be computed.

    (3) OpenSSL identifies cipher suites using hyphens as separators, for
        example: DES-CBC3-SHA. GnuTLS uses underscores, for example:
        RSA_ARCFOUR_SHA. What's more, OpenSSL complains if underscores are
        present in a cipher list. To make life simpler, Exim changes
        underscores to hyhens for OpenSSL and hyphens to underscores for GnuTLS
        when processing the list of cipher suites in the require_ciphers option
        of the smtp transport.

    (4) The require_ciphers option of the smtp transport operates differently.
        In OpenSSL, you can pass the list to the library before starting the
        session, and it takes care of it. In GnuTLS, all that can be done, as
        far as I know, is to check after the session has been set up that the
        cipher is acceptable. This makes require_ciphers rather less useful, if
        not totally unusable.

19. A pipe transport may now have temp_errors = * to specify that all errors
    are to be treated as temporary.

20. The lmtp transport can now handle delivery to a Unix domain socket instead
    of to a pipe. There is a new option called "socket", which, after
    expansion, must be the name of the socket path. It is mutually exclusive
    with "command".

21. Support for flock() has been added to the appendfile transport, for use
    on those operating systems where some mailbox accessors use flock() and it
    doesn't interwork with fcntl() locking. Not all operating systems provide
    flock(). Some versions of Solaris don't have it (and some, I think, provide
    a not quite right version built on top of lockf() - i.e. fcntl()). If the
    OS doesn't have flock(), Exim will be built without the ability to use it,
    and any attempt to do so will cause a configuration error.

    Two new options are added to the appendfile transport: use_flock_lock
    (default false) and lock_flock_timeout (default zero). These work like
    their fcntl equivalents. You can use both fcntl() and flock() locking
    simultaneously if you want.

    The way that MBX locking works has been changed, to make it possible to use
    either or both of fcntl() and flock(). You can now use use_mbx_lock with
    either (or both) of use_fcntl_lock and use_flock_lock. These last two
    options control what kind of locking is used in implementing the MBX
    locking rules.

    Note that flock() locks do not work on NFS files (unless flock() is just
    being mapped onto fcntl() by the OS).

22. Support for flock() has also been added to the exim_lock utility. The
    -flock option requests flock() locking. It can be given in conjunction with
    the -mbx option to specify that flock() locks are to be used with the MBX
    locking rules.

23. There is a new expansion item for reading from a Unix domain socket. The
    minimal way of using it is:

      ${readsocket{/socket/name}{request-string}}

    Exim connects to the socket, writes the request string (unless it is an
    empty string) and reads from the socket until an end-of-file is read. A
    timeout of 5 seconds is applied.

    Additional, optional arguments extend what can be done. Firstly, you can
    vary the timeout. For example:

      ${readsocket{/socket/name}{request-string}{3s}}

    A fourth argument allows you to change any newlines that are in the data
    that is read, in the same way as for ${readfile. This example turns them
    into spaces:

      ${readsocket{/socket/name}{request-string}{3s}{ }}

    As with all expansions, the substrings are expanded before the processing
    happens. Errors in these sub-expansions cause the expansion to fail.

    The following errors can occur while interacting with the socket:

    . Failure to create a socket file descriptor;
    . Failure to connect the socket;
    . Failure to write the request-string;
    . Timeout on reading from the socket.

    By default, any of these errors causes the expansion to fail. However, if
    you supply a 5th substring, it is expanded and used when any of the above
    errors occurs. For example:

      ${readsocket{/socket/name}{request-string}{3s}{\n}{socket failure}}

    You can test for the existence of the socket by wrapping this expansion in
    ${if exists, but there is a race condition between that test and the actual
    opening of the socket, so it is safer to use the 5th argument if you want
    to be absolutely sure of avoiding an expansion error for a non-existent
    socket.

24. The redirect router now has the option forbid_filter_readsocket, to lock
    out the use of ${readsocket in expansions in filter files.

25. It is now possible to set errors_to to the empty string in routers, by
    either of these settings:

      errors_to =
      errors_to = ""

    (An expansion item that yields an empty string has the same effect.)
    If you do this, locally detected delivery errors no longer give rise to
    bounce messages; the error is discarded. In addition, if the address is
    delivered to a remote host, the return path is set to "<>" (unless
    overridden by the return_path option on the transport). If for some reason
    you want to discard local errors, but use a non-empty MAIL command for
    remote deliveries, you can preserve the return patch in $address_data in
    the router, and reinstate it in the transport by the return_path option.

26. If the expansion of an errors_to setting is forced to fail, the option is
    ignored and the return path remains unchanged.

27. There are new boolean router and transport options called disable_logging.
    If set, nothing is logged for any errors in routing or for deliveries
    carried out by the transport, respectively. You should not set either of
    these options unless you really, really know what you are doing.

28. There is now server support for the SPA (aka NTLM) authentication method. A
    new option for the spa authenticator called spa_serverpassword sets this
    up. It is expanded, and the result must be the cleartext password for the
    username which is at that stage available in $. For example:

      spa:
        driver = spa
        public_name = NTLM
        server_password = ${lookup{$1}lsearch{/etc/exim/spa_clearpass}}

29. Exim now counts the number of "non-mail" commands in an SMTP session, and
    drops the connection if there are too many. This catches some DoS attempts
    and things like repeated failing AUTHs, or a mad client looping sending
    EHLO. The new option smtp_accept_max_nonmail defines what "too many" means.
    Its default value is 10. In an authenticated, encrypted session you would
    expect there to be 4 (EHLO, STARTTLS, EHLO, AUTH), so this allows plenty of
    leeway.

    When a new message is expected, if the first command is RSET, it is not
    counted. This allows a client to send one RSET between messages (this isn't
    necessary, but some clients do it). Otherwise, all commands other than
    MAIL, RCPT, DATA, and QUIT are counted.

30. Caching is now implemented for callout verification. It uses a new hints
    database called "callout", and is enabled by default. The syntax of
    callouts in ACL statements has been extended to provide for extra
    facilities. The syntax is now:

      verify=[sender|recipient]/callout[=option,option...]

    The available options are:

      (a time)    specify the callout timeout (default 30s)
      defer_ok    assume success if callout defers (cannot be completed)
      no_cache    do not use the cache
      postmaster  check postmaster address as well
      random      check a random address as well

    For example:

      deny verify=sender/callout=10s,random,defer_ok

    The old syntax /callout_defer_ok and /check_postmaster as separate verify
    options is retained for backwards compatibility, but is now deprecated.

    The timeout is used for each response from the remote host.

    When defer_ok is specified, failure to contact any host, or any other kind
    of temporary error is treated as success by the ACL. However, the cache is
    not updated in this circumstance.

    If no_cache is specified, the callout cache is neither read not updated.
    Otherwise, the result of a callout for the address is remembered in a cache
    record. This is used to avoid re-doing the callout in subsequent
    verifications of the same address, until the callout record expires. If the
    original callout fails, a detailed SMTP error message is given about the
    failure. However, for subsequent failures as a result of the cache, this
    message is not available.

    The expiry times for negative and positive address cache records are
    independent, and can be set by the new options

      callout_negative_expire   (default 2h)
      callout_positive_expire   (default 24h)

    If a host gives a negative response to an SMTP connection, or rejects any
    commands up to and including MAIL FROM:<>, any callout attempt is bound to
    fail. Exim remembers such failures in a callout domain cache record, which
    it uses to fail callouts for the domain without making new connections,
    until the domain record times out. Again, there are two separate expiry
    times for domain cache records:

      callout_domain_negative_expire   (default 3h)
      callout_domain_positive_expire   (default 7d)

    If "postmaster" is specified as a callout option, a successful callout
    check is followed by a similar check for the local part "postmaster" at the
    same domain. If this address is rejected, the callout fails. The result of
    the postmaster check is recorded in the domain cache record; if it is a
    failure, this is used to fail subsequent callouts for the domain without a
    connection being made.

    If "random" is specified as a callout option, a successful callout check is
    followed by a similar check for a "random" local part. This isn't really
    "random" - it is defined by the expansion of the option callout_random_
    local_part, which defaults to $primary_host_name-$tod_epoch-testing. The
    idea here is to try to determine whether the remote host accepts all local
    parts without checking. If it does, there is no point in doing callouts for
    specific local parts. If the "random" check succeeds, the result is saved
    in the domain cache record, and used to force subsequent callout checks to
    succeed without a connection being made.

    Domain records expire when the negative expiry time is reached if callouts
    cannot be made for the domain, or if the postmaster check failed.
    Otherwise, they expire when the positive expiry time is reached.

    The callout caching mechanism is based entirely on the domain of the
    address that is being tested. If the domain routes to several hosts, it is
    assumed that their behaviour will be the same.

    The hints database utility programs (exim_dumpdb, exim_tidydb, and
    exim_fixdb) have been updated to handle the new callout cache database,
    identified as "callout".

31. Instead of cycling the main and reject log files by renaming them
    periodically, some sites like to use files whose names contain a datestamp,
    for example, mainlog-20021225. The datestamp is in the form yyyymmdd. Exim
    now has support for this way of working. It is enabled by setting the
    log_file_path option, and including %D at the point where the datestamp is
    required. For example:

      log_file_path = /var/spool/exim/log/%slog-%D
      log_file_path = /var/log/exim-%s-%D.log
      log_file_path = /var/spool/exim/log/%D-%slog

    As before, %s is replaced by "main" or "reject"; the following are examples
    of names generated by the above examples:

      /var/spool/exim/log/mainlog-20021225
      /var/log/exim-reject-20021225.log
      /var/spool/exim/log/20021225-mainlog

    When this form of log file is specified, Exim automatically switches to new
    files at midnight. It does not make any attempt to compress old logs; you
    will need to write your own script if you require this. You should not
    run exicyclog with this form of logging.

    The location of the panic log is also determined by log_file_path, but it
    is not datestamped, because rotation of the panic log does not make sense.
    When generating the name of the panic log, %D is removed from the string.
    In addition, if it immediately follows a /, a following non-alphanumeric
    character is removed; otherwise a preceding non-alphanumeric character is
    removed. Thus, the three examples above would give these panic log names:

      /var/spool/exim/log/paniclog
      /var/log/exim-panic.log
      /var/spool/exim/log/paniclog

32. The new expansion variable $tod_logfile gives the date in the format used
    for datestamping log files (see above).

33. Alternative versions of the hash, length, nhash, and substr expansions have
    been added which expand their parameters. For example:

      ${substr{3}{4}{abcdcdef}}   is equivalent to    ${substr_3_4:abcdef}

    but in the bracketed example, the parameters are expanded and can therefore
    vary from run to run. These new syntactic forms must always be spelt out in
    full; there are no abbreviations as there are for the fixed forms.

34. There are two new options for the scripts/exim_install script:

      -no_chown    bypasses the call to change the owner of the installed binary
                   to root, and the chmod it to be a setuid binary

      -no_symlink  bypasses the setting up of the symbolic link "exim" to the
                   installed binary (who name is, e.g. exim-4.10-2).

     In addition, the environment variable INSTALL_ARG is passed to the
     install script from "make install", which allows the usage:

       INSTALL_ARG=-no_chown make install

     for instance.

35. When the order of a host list is being randomized, either in the
    manualroute router, or in the SMTP transport, the list can now be split
    into groups whose order is separately randomized. This makes it possible to
    set up MX-like behaviour. The boundaries between groups are indicated by an
    item that is just "+" in the host list. For example, a manualroute router
    could contain this setting:

      route_list = *  host1:host2:host3:+:host4:host5

    If hosts_randomize is set, the order of the first three hosts and the order
    of the last two hosts is randomized for each use, but the first three
    always end up before the last two. If hosts_randomize is not set, a "+"
    item in the list is just ignored. If a randomized host list is passed to an
    smtp transport that also has hosts_randomize set, the list is not
    re-randomized.

36. When comparing two routed addresses to see if they can be delivered to the
    same host in a single SMTP transaction, randomized host lists from the
    manualroute router are now considered "the same" (this was always the case
    for host lists derived from MX records).

37. There are new routing options "randomize" and "no_randomize" for individual
    routes in the manualroute router. This allows you to select host
    randomization on a per-domain basis. The router option hosts_randomize now
    acts as the default. For example:

      route_list = domaina  host1:host2:host3  randomize;\
                   domainb  host4:host5

38. There's a new expansion item called hmac. It uses cryptographic hashing
    (either MD5 or SHA-1) to convert a shared secret and some text into a
    message authentication code, as specified in RFC 2104. You could produce a
    similar effect using ${md5:secret_text${data}}, but allegedly HMAC provides
    better defence against deducing the secret. The syntax is:

      ${hmac{hashname}{secret}{text}}

    where the hash name must expand to either "md5" or "sha1" at present. For
    example:

      ${hmac{md5}{somesecret}{${primary_hostname} ${tod_log}}}

    For the hostname "mail.example.com" and time "2002-10-17 11:30:59", this
    would produce:

      dd97e3ba5d1a61b5006108f8c8252953

39. By default, the timestamps on log lines are in local time without the
    timezone. This means that if your timezone changes twice a year the
    timestamps in log lines are ambiguous for an hour when the clocks go back.
    One way of avoiding this problem is to set the timezone to UTC. An
    alternative is to set the new main boolean option log_timezone. This turns
    on the addition of the timezone offset from UTC to timestamps in log lines.
    Turning on this option can add quite a lot to the size of log files because
    each line is extended by 6 characters. The $tod_log variable continues to
    yield the timestamp without the zone (as before), but there is now a new
    variable called $tod_zone that contains just the timezone offset (e.g.
    +0500).

    The exigrep and eximstats utilities have been upgraded to deal with log
    files containing timezone offsets in their timestamps. The Exim monitor has
    also been upgraded to recognize such lines, but it does not display the
    zone offsets.

40. The variable $load_average contains the system load average, multiplied by
    1000 so that it is an integer. For example, a load average of 0.21 is given
    as 210.

41. There is a new generic router option called router_home_directory. It sets
    a home directory for use while the router is running. (Compare
    transport_home_directory, which sets a home directory for later
    transporting.) In particular, this sets a value for $home while a filter is
    running. The value is expanded; forced expansion failure causes the option
    to be ignored - other failures cause the router to defer.

    The value set by router_home_directory overrides a home directory set by
    check_local_user. When a router accepts an address and routes it to a
    transport (including the cases when a redirect router generates a pipe,
    file, or autoreply delivery), the home directory setting for the transport
    is taken from the first one of these values that is set:

      . the home_directory option on the transport
      . the transport_home_directory option on the router
      . the password data if check_local_user is set on the router
      . the router_home_directory option on the router

    In other words, router_home_directory overrides the password data for the
    router, but not for the transport.

42. If the message_id_header_domain option is set, the string is expanded and
    used as the right hand side (domain) of the Message-ID: header that Exim
    creates if an incoming message does not have one. Otherwise, the primary
    host name is used. Only the characters A-Z, a-z, 0-9, dot and hyphen are
    accepted; any others are replaced by hyphens. If the expansion is forced to
    fail, or if the result is an empty string, the option is ignored.

43. If the expansion of message_id_header_text is forced to fail, the option is
    now ignored. It always was ignored if it generated an empty string.

44. The default way of looking up IP addresses for hosts in the manualroute and
    queryprogram routers has been changed. If "byname" or "bydns" is explicitly
    specified, there is no change, but if no method is specified, Exim now
    behaves as follows:

      First, a DNS lookup is done. If this yields anything other than
      HOST_NOT_FOUND, that result is used. Otherwise, Exim goes on to try a
      call to getipnodebyname() (or gethostbyname() on older systems) and the
      result of the lookup is the result of that call.

    This change has been made because it has been discovered that on some
    systems, if a DNS lookup called via getipnodebyname() times out,
    HOST_NOT_FOUND is returned instead of TRY_AGAIN. Thus, it is safest to try
    a DNS lookup directly first, and only if that gives a definite "no such
    host" to try the local function.

45. A colon-separated list of configuration files can now be given, either in
    the CONFIGURE_ FILE setting in Local/Makefile, or as an argument to the -C
    command line option. The first file that exists is used. Failure to open an
    existing file stops Exim from proceeding any further along the list. The
    output given by "exim -bV" and by "exim -bP configure_file" shows the name
    of the actual configuration file that is being used. When a list is given
    for CONFIGURE_FILE, the exim_install script (called by "make install") does
    not check for the existence of a runtime configuration in order to install
    a default.

46. There are two new variables called $rcpt_defer_count and $rcpt_fail_count.
    For an incoming SMTP message, they contain counts of the numbers of RCPT
    commands that have had temporary and permanent error responses,
    respectively. The available data about recipients is therefore as follows:

      $rcpt_count           the total number of RCPT commands
      $rcpt_defer_count     the number that had 4xx responses
      $rcpt_fail_count      the number that had 5xx responses
      $recipients_count     the number that had 2xx responses

47. There's a new log selector called "rejected_header". It is included in the
    default selection. By turning it off, you can suppress the logging of
    a message's header that happens in the reject log by default when the
    message is rejected after the DATA phase. Header logging can also be turned
    off individually for messages that are rejected by the local_scan()
    function. See 50 below.

48. There's a new generic router option called "cannot_route_message". It
    specifies a text message that is used when an address cannot be routed
    because Exim has run out of routers. The default message is "Unrouteable
    address". This option is useful only on routers that have no_more set, or
    on the very last router in a configuration, because the value that is used
    is taken from the last router that inspects an address. For example, using
    the default configuration, you could put

      cannot_route_message = Remote domain not found in DNS

    on the first (dnslookup) router, and

      cannot_route_message = Unknown local user

    on the final router that checks for local users.

49. Some new features have been added to the ACL facilities:

      (i) The new verb "drop" behaves like "deny", except that the SMTP
          connection is forcibly closed after the 550 error message has been
          sent. For example:

             drop   message   = I don't take more than 20 RCPTs
                    condition = ${if > {$rcpt_count}{20}}{yes}{no}}

     (ii) The new verb "defer" forces a deferral, that is, a temporary (4xx)
          error response. For a recipient, this is pretty much the same as
          using a redirect router and :defer: while verifying, but "defer" can
          be used in any ACL, and even for a recipient it might be a simpler
          approach.

    (iii) There are two new modifiers: "delay" and "control". Like the other
          modifiers, they appear in lists of conditions, but do not affect the
          result of the condition tests. In effect, they are always "true".

          "delay = <time>" causes Exim to wait for the time interval before
          proceeding. The time is given in the usual Exim notation. This
          modifier may appear in any ACL.

          "control = freeze" and "control = queue_only" may appear only in the
          ACLs for commands relating to an incoming message. If encountered,
          they cause the message to be frozen or just queued (without immediate
          delivery), respectively, assuming that it is accepted.

          These new modifiers can be used in two different ways:

          (a) They can form part of an "accept" or "deny" verb. For example:

                deny    <some conditions>
                        delay = 30s

                accept  <some conditions>
                        control = queue_only

              Note that if "control" is used in a RCPT ACL, it applies to the
              whole message, not just the individual recipient.

          (b) They can be used with "warn" to apply the control, leaving the
              decision about accepting or denying to a subsequent verb. For
              example:

                warn    <some conditions>
                        delay = 2m
                        control = freeze
                accept  ...

              Note that if "warn" has neither "message" nor "log_message", it
              does not add anything to the message and does not write a log
              entry.

     (iv) If a "hosts" condition is satisfied by means of a lookup, the result
          of the lookup is made available in the $host_data variable. This
          allows you, for example, to do things like this:

            deny  hosts = net-lsearch;/some/file
                  message = $host_data

      (v) ACLs can now be used at four additional times, as specified by these
          new options:

            acl_not_smtp        for non-SMTP messages
            acl_smtp_connect    run at the start of an SMTP connection
            acl_smtp_mail       run for the MAIL command
            acl_smtp_starttls   run for the STARTTLS command

          In all four cases, if no ACL is defined, the default action is to
          accept.

          The non-SMTP ACL actually applies to all non-interactive incoming
          messages, that is, it applies to batch SMTP as well as to non-SMTP
          messages. (Batch SMTP is not really SMTP.) This ACL is run just
          before the local_scan() function. Any kind of rejection is treated as
          permanent, because there is no way of sending a temporary error for
          these kinds of message. Many of the ACL conditions (e.g. host tests,
          and tests on the state of the SMTP connection such as encryption and
          authentication) are not relevant and are forbidden in this ACL.

          There is no difference between "deny" and "drop" for the connect-
          time ACL. The connection is always dropped after sending a 550
          response. The ACL test happens after the test specified by
          host_reject_connection (which is now an anomaly) and any TCP Wrappers
          testing.

          The ACL for MAIL is run after the sender address has been set up, so
          it can be checked in the ACL.


50. The variable $authentication_failed is set to 1 if a client issues an AUTH
    command which does not succeed. Otherwise it is set to 0. This makes it
    possible to distinguish between "did not try to authenticate"
    ($sender_host_authenticated is empty and $authentication_failed is set to
    0) and "tried to authenticate but failed" ($sender_host_authenticated is
    empty and $authentication_failed is set to 1). Compare $host_lookup_failed.
    Failure includes any negative response to an AUTH command, including (for
    example) an attempt to use an undefined mechanism.

51. There are a number of extensions to the API for the local_scan() function:

    * Exim's debugging function, debug_printf(), is now made available. The
      arguments are as for printf(). If this function is called when Exim is
      not in debugging mode, it does nothing. In debugging mode, it adds to
      the debugging output. If the "pid" and/or "timestamp" debugging selectors
      are set, the pid and/or timestamp are automatically added to each debug
      output line.

    * The log flag LOG_PANIC is now available for writing to the panic log.

    * The global variable debug_selector (an unsigned int) is made available.
      It is zero when no debugging is taking place. Otherwise, it is a bitmap
      of debugging selectors. Two bits are identified for use in local_scan()
      and defined as macros:

        The D_v bit is set when -v was present on the command line. This is a
        testing option that is not privileged - any caller may set it. All the
        other selector bits can be set only by admin users.

        The D_local_scan bit is provided for use by local_scan(); it is set by
        the +local_scan debug selector. It is not included in the default set
        of debugging bits.

      Thus, to write to the debugging output only when +local_scan has been
      selected, you should use code like this:

         if ((debug_selector & D_local_scan) != 0) debug_printf("xxx", ...);

    * The variable message_id is now directly accessible.

    * The return values LOCAL_SCAN_REJECT_NOLOGHDR and LOCAL_SCAN_TEMPREJECT_
      NOLOGHDR behave the same as the versions without _NOLOGHDR, except that
      the header of the rejected message is not written to the reject log. They
      have the effect of unsetting the rejected_header log selector (see 47
      above) for just this rejection. If the rejected_header selector is unset
      in the configuration file, these new return codes are no different to the
      original ones.

    * The return values LOCAL_SCAN_ACCEPT_FREEZE and LOCAL_SCAN_ACCEPT_QUEUE
      cause the message to be accepted, but frozen or just queued without
      immediate delivery, respectively.

    * Exim's normal dynamic memory used when receiving a message is recycled
      at the end of the message. This means that it cannot be used for holding
      data that must be preserved over a number of incoming messages on the
      same SMTP connection. Exim now has two pools of dynamic memory; the
      second one is not recycled. If the store_pool variable is set to
      POOL_PERM, all new allocations (e.g. by the store_get() and
      string_sprintf() functions) come from the permanent pool. There is also a
      convenience function called store_get_perm() which gets a block of store
      from the permanent pool.

    * After a failing call to expand_string() (returned value NULL), the
      variable expand_string_message contains the error message.

    * It is now possible to have option settings in the main configuration file
      which set values in static variables in the local_scan() module. If you
      want to do this, you must have the line

        LOCAL_SCAN_HAS_OPTIONS=yes

      in your Local/Makefile. (This line is in OS/Makefile-Default, commented
      out). Then, in the local_scan() source file you must define static
      variables to hold the option values, and a table to define them. The
      table must be a vector called local_scan_options, of type optionlist.
      Each entry is a triplet, consisting of a name, an option type, and a
      pointer to the variable that holds the value. The entries must appear in
      alphabetical order. Following local_scan_options you must also define a
      variable called local_scan_options_count which contains the number of
      entries in the table. Here's a short example, showing two kinds of
      option:

        static int my_integer_option = 42;
        static uschar *my_string_option = US"a default string";

        optionlist local_scan_options[] = {
          { "my_integer", opt_int,       &my_integer_option },
          { "my_string",  opt_stringptr, &my_string_option }
        };

        int local_scan_options_count =
          sizeof(local_scan_options)/sizeof(optionlist);

      The values of the variable can now be changed from Exim's runtime
      configuration file by including a section that starts with this line:

        begin local_scan

      The available types of option data are as follows:

        opt_bool

      This specifies a boolean (true/false) option. The address should point to
      a BOOL variable, which will be set to TRUE or FALSE. If you want to
      detect whether such a variable has been set at all, you can initialize it
      to TRUE_UNSET. (BOOL variables are ints underneath, so can hold more than
      2 values.)

        opt_fixed

      This specifies a fixed point number, such as is used for load averages.
      The address should point to an int, and the value is stored multiplied by
      1000, so, for example, 1.4142 is truncated and stored as 1414.

        opt_int

      This specifies an integer, and the address should point to an int. The
      value may be specified in any of the integer formats accepted by Exim.

        opt_mkint

      This is the same as opt_int, except that when such a value is output in a
      -bP listing, if it is an exact number of kilobytes or megabytes, it is
      printed with the suffix K or M.

        opt_octint

      This also specifies an integer, but the value is always interpeted as an
      octal integer, whether or not it starts with the digit zero, and it is
      always output in octal.

        opt_stringptr

      This specifies a string value; the address must be a pointer to a
      variable that points to a string (i.e. of type uschar *).

        opt_time

      This specifies a time interval value. The address must point to an int,
      and the value that is placed there is a number of seconds.

      If the -bP command line option is followed by "local_scan", it prints out
      the values of all the local_scan() options.

    * The global variable host_checking, which is true during a -bh session, is
      now accessible in the local_scan() function.

    * Some new supporting functions are provided, giving access to Exim's list-
      matching functions. They are

        int lss_match_domain(uschar *domain, uschar *list)

      This function checks for a match in a domain list. Domains are always
      matched caselessly.

        int lss_match_local_part(uschar *localpart, uschar *list, BOOL caseless)

      This function checks for a match in a local part list. The third argument
      controls case-sensitivity.

        int lss_match_address(uschar *address, uschar *list, BOOL caseless)

      This function checks for a match in an address list. The third argument
      controls the case-sensitivity of th local part match. The domain is
      always matched caselessly.

        int lss_match_host(uschar *host_name, uschar *host_address, uschar *list)

      This function checks for a match in a host list. The most common usage is
      expected to be

        lss_match_host(sender_host_name, sender_host_address, ...)

      An empty address field matches an empty item in the host list. If the
      host name is NULL, the name corresponding to sender_host_address is
      looked up.

      The return values from these four functions are one of the following
      constants, defined in local_host.h:

        OK     match succeeded
        FAIL   match failed
        DEFER  match deferred

      DEFER is usually caused by some kind of lookup defer, such as the
      inability to contact a database.

      In addition, lss_match_host() returns ERROR in the case when it had to
      look up a host name, and the lookup failed.
       
    * Exim's base 64 encode/decode functions are now available to the 
      local_scan() function. The prototypes are:
      
        uschar *lss_b64encode(uschar *cleartext, int length);
        int     lss_b64decode(uschar *codetext, uschar **cleartext);
        
      For encoding, you pass the address of the clear text and its length. The 
      text may contain bytes of any value, including zero. The result is passed 
      back in dynamic memory that is obtained by calling store_get(). It is
      zero terminated.
      
      For decoding you pass a zero-terminated string and a variable that is set 
      to point to the result, which is again in dynamic memory. The length of 
      the decoded string is the yield of the function. If the input is invalid 
      base 64 data, the yield is -1. A zero byte is added to the end of the 
      output string to make is easy to interpret as a C string (assuming it 
      contains no zeros of its own). The added zero byte is not included in the
      returned count.

52. The format of lsearch-ed files has been extended to make it possible to
    have colons and whitespace in keys. If a key begins with a quote character,
    it is terminated only by a matching quote (or end of line), and the normal
    escaping rules apply for its contents. An optional colon is permitted after
    quoted keys (exactly as for unquoted keys). There is no special handling of
    quotes for the data part of an lsearch line.

53. There's a new lookup called "wildlsearch". This searches a file linearly,
    like lsearch, but instead of being interpreted as a literal string, each
    key may be wildcarded. The following forms of wildcard are recognized:

    * The string may begin with an asterisk to mean "begins with". For example:

        *.a.b.c       data for anything.a.b.c
        *fish         data for anthingfish

    * The string may begin with a circumflex to indicate a regular expression.
      For example:

        ^\d+\.a\.b    data for <digits>.a.b

      If the regex contains white space or colon characters, you must either
      quote it (see 52 above), or represent these characters in other ways. For
      example, \s can be used for white space and \x3A for a colon. This may be
      easier than quoting, because if you quote, you have to escape all the
      backslashes inside the quotes.

    * Although I can't see it being of much use, the general matching function
      that is use means that the string may begin with a lookup name terminated
      by a semicolon and followed by lookup data. For example:

        cdb;/some/file  data for keys that match the file

      The data that is looked up from the second file is discarded.

    Keys that do not match any of these patterns are interpreted literally. The
    continuation rules for the data are the same as for lsearch, and keys may
    be followed by optional colons.

    Unlike other single-key lookup types, a file of data for wildlsearch can
    *not* be turned into a DBM or cdb file. Like lsearch, the testing is done
    case-insensitively.

54. The new variable $tls_certificate_verified is set to 1 if a TLS certificate
    was verified when the message was received, and zero otherwise.

55. There is a new lookup called "passwd". It is a query-style lookup with
    queries that are just user names. The lookup calls getpwnam() to
    interrogate the system password data, and on success, the result string is
    the same as you would get from an lsearch lookup on a traditional
    /etc/passwd file, though with "*" for the password. For example:

      *:42:42:King Rat:/home/kr:/bin/bash

    Using this lookup, it is now possible to specify a local_users precondition
    for "if local part is a local user OR matches one of these things". For
    example:

      local_parts = passwd;$local_part : lsearch;/etc/other/users

    This is in contrast to specifying check_local_user and local_parts, which
    together specify the AND of the two conditions. Note, however, that the
    side effects of check_local_user (such as setting up a home directory) do
    not occur when a passwd lookup is used in a local_parts (or any other)
    precondition.

56. The new "eval" item supports simple arithmetic in expansion strings. The
    syntax is

      ${eval:<expression>}

    The expression (after expansion) is a conventional arithmetic expression,
    but it is limited to the four basic operators (plus, minus, times, divide)
    and parentheses. All operations are carried out using integer arithmetic.
    Plus and minus have a lower priority than times and divide; operators with
    the same priority are evaluated from left to right. Numbers may be decimal,
    octal (starting with "0") or hexadecimal (starting with "0x"). A number may
    be followed by 'K' or 'M' to multiply it by 1024 or 1024*1024,
    respectively. Negative numbers are supported. The result of the computation
    is a decimal representation of the answer (without 'K' or 'M'). For
    example:

      ${eval:1+1}       yields 2
      ${eval:1+2*3}     yields 7
      ${eval:(1+2)*3}   yields 9

    As a more realistic example, in an ACL you might have

      deny   message = Too many bad recipients
             condition =                    \
               ${if and {                   \
                 {gt{$rcpt_count}{10}}      \
                 {                          \
                 lt                         \
                   {$recipients_count}      \
                   {${eval:$rcpt_count/2}}  \
                 }                          \
               }{yes}{no}}

    The condition is true if there have been more than 10 RCPT commands and
    fewer than half of them have resulted in a valid recipient.

57. The partial lookup facility for single-key lookups has been extended to
    allow for different prefixes. The motivation is to allow file formats that
    are used by other MTAs. By default, a search type such as partial-lsearch
    uses "*." as a prefix when doing the partial searching. A different prefix
    can be supplied in parentheses instead of the hyphen. For example:

      domains = partial(.)lsearch;/some/file

    In this example, if the domain is "a.b.c", the sequence of lookups would be
    "a.b.c", ".a.b.c", ".b.c", and ".c". The prefix may consist of any
    punctuation characters other than a closing parenthesis. It may be empty,
    for example:

      domains = partial()cdb;/some/file

    For this example, if the domain is "a.b.c", the sequence of lookups is
    "a.b.c", "b.c", and "c".

    There is no change to what happens if you give a number after "partial"; it
    still specifies the minimum number of non-wild elements in the key, and the
    default is still 2. However, if "partial0" is specified, what happens at
    the end (when the lookup with just one non-wild component has failed)
    depends on the prefix:

      . If the prefix has zero length, the whole lookup fails.

      . If the prefix has length 1, a lookup for just the prefix is done. For
        example, the final lookup for "partial0(.)" is for "." alone.

      . Otherwise, if the prefix ends in ".", the dot is removed, and the
        remainder is looked up. This preserves the previous action in the case
        of the default prefix "*.".

      . Otherwise, the whole prefix is looked up.

    If the lookup type has "*" on the end, the default lookup that this implies
    always happens if all other lookups fail. Previously, it was optimised out
    for types such as "partial0-lsearch*" because it would have been done
    twice. There might be a use for lookup types such as "partial0(.)lsearch*",
    for example.
    

Version 4.10
------------

The documentation is up-to-date for the 4.10 release.

****
