TLS Verification Process for Cisco Email Security

Introduction

This document describes the Transport Layer Security (TLS) server identity verification process for the Cisco Email Security Appliance (ESA)

TLS Verification Process for Cisco Email Security

The TLS Verification process is essentially a two stage validation process:

I - CERTIFICATE VALIDATION

This involves verification of:

• certificate validity period - certificate lifetime
• certificate chain issuer
• revocation list, etc ..

II - SERVER IDENTITY VALIDATION

This is a validation process of the server presented identity (contained in X.509 public key certificate) against the server reference identity.

Background

Let's keep with identity name terminology described in RFC 6125.  

Note: The presented identity is an identifier presented by a server X.509 public key certificate which can include more than one presented identifiers of different types. In case of SMTP service, it is contained either as a subjectAltName extension of type dNSName or as the CN (Common Name) derived from the subject field.

Note: The reference identity is an identifier constructed from a fully qualified DNS domain name that a client expects an application service to present in the certificate.

The verification process is mostly important for a TLS client, as in general a client initiates a TLS session and a client needs to authenticate the communication. To achieve this a client needs to verify if the presented identity matches the reference identity. The important part is to understand that the security of TLS Verification process for mail delivery is almost entirely based on the TLS client.

Step One

The first step in server identity validation is to determine the reference identity by the TLS client. It depends from the application what list of reference identifiers TLS client consider to be acceptable. Also a list of acceptable reference identifiers must be constructed independently of the identifiers presented by the service. [rfs6125#6.2.1]

The reference identity must be a fully qualified DNS domain name and can be parsed from any input (which is acceptable for a client and consider to be secure).  The reference identity need to be a DNS hostname to which the client is trying to connect. 

The recipient email domain name is reference identity which is directly expressed by the user, by the intent to send a message to a particular user in particular domain and this also met a requirement to be a FQDN to which a user is trying to connect. It is consistent only in case of self-hosted SMTP server where the SMTP server is owned and managed by the same owner and the server is not hosting too many domains. As each domain need to be listed in certificate (as one of subjectAltName: dNSName values). From an implementation perspective, most of Certificate Authorities (CA) limits the number of domain names value to as low as 25 entries (to as high as 100). This is not accepted in case of the hosted environment, let's think about Email Service Providers (ESP) where the destination SMTP servers hosts thousands and more of domains. This just does not scale.

The explicitly configured reference identity seems to be the answer but this impose some constraints, as it is required to manually associate a reference identity to source domain for each destination domain or “obtaining the data from a third-party domain mapping service in which a human user has explicitly placed trust and with which the client communicates over a connection or association that provides both mutual authentication and integrity checking”. [RFC6125#6.2.1]

Conceptually, this can be thought of a one-time “secure MX query” at the time of configuration, with the result permanently cached on the MTA to safeguard against any DNS compromise while in run state. [2]

This gives a stronger authentication only with “partner” domains but for generic domain which has not been mapped this does not pass the exam and this is also not immune against configuration changes on the side of the destination domain (like hostname or IP address changes).

Step Two

The next step in the process is to determine a presented identity. The presented identity is provided by a server X.509 public key certificate, as subjectAltName extension of type dNSName or as Common Name (CN) found in the subject field. Where it is perfectly acceptable for the subject field to be empty, as long as the certificate contains a subjectAltName extension that includes at least one subjectAltName entry.

Although the use of Common Name is still in practice it is consider to be deprecated and the current recommendation is to use subjectAltName entries. The support for the identity from Common Name stay for backward compatibility. In such a case a dNSName of subjectAltName should be used first and only when it is empty the Common Name is checked.

Note: the Common Name is not strongly typed because a Common Name might contain a human-friendly string for the service, rather than a string whose form matches that of a fully qualified DNS domain name

At the end when both type of identities have been determined, the TLS client needs to compare each of its reference identifiers against the presented identifiers for the purpose of finding a match.

ESA TLS Verification

ESA allows enabling TLS and certificate verification on delivery to specific domains (using the Destination Controls page or destconfig CLI command). When TLS certificate verification is required, you can choose one of two verification options since AsyncOS version 8.0.2. The expected verification result can vary depending on configured option. From 6 different settings for TLS, available under destination control there are two important which are  responsible for certificate verification:

1. TLS Required - Verify
2. TLS Required - Verify Hosted Domains.

CLI: destconfig 

Do you want to use TLS support?

1. No

2. Preferred

3. Required

4. Preferred - Verify

5. Required - Verify

6. Required - Verify Hosted Domains 

[6]> 

A TLS verification process for option (4) Preferred – Verify is identical to (5) Required – Verify, but the action taken based on results differs as presented in below table. The outcomes for option (6) Required – Verify Hosted Domains is identical to (5) Required – Verify but a TLS verification flow is quite different. 

TLS Settings	Meaning
4. Preferred (Verify)	TLS is negotiated from the Email Security appliance to the MTA(s) for the domain. The appliance attempts to verify the domains certificate. Three outcomes are possible: TLS is negotiated and the certificate is verified. The mail is delivered via an encrypted session. TLS is negotiated, but the certificate is not verified. The mail is delivered via an encrypted session. No TLS connection is made and, subsequently the certificate is not verified. The email message is delivered in plain text.
5. Required (Verify)	TLS is negotiated from the Email Security appliance to the MTA(s) for the domain. Verification of the domains certificate is required. Three outcomes are possible: A TLS connection is negotiated and the certificate is verified. The email message is delivered via an encrypted session. A TLS connection is negotiated but the certificate is not verified by a trusted CA. The mail is not delivered. A TLS connection is not negotiated. The mail is not delivered.

The difference between TLS Required - Verify and TLS Required - Verify Hosted Domain options lays in identity verification process. The way how the presented identity is processed and what type of reference identifiers are allowed to be used make a difference about a final result. The purpose of below description as well as the whole document is to closer this process to the end user. As the incorrect or unclear understanding of this subject can have a security impact on user network.

TLS Required Verify

The presented identity is derived first from subjectAltName - dNSName extension and if there is no match or subjectAltName extension does not exist than CN-ID - Common Name from subject field is checked.

The reference identity (REF-ID) list  is constructed from a recipient domain or recipient domain and hostname derived from a PTR DNS query run against the IP address the client is connected to.  Note: In that particular case, different reference identities are compared with different presented identity checks.

~= represents exact or wildcard match

The presented identity (dNSName or CN-ID) is compared with accepted reference identities till it is matched and in the order they are listed below. 

• If dNSName extension of subjectAltName exists:
  • exact or wildcard match is done against recipient domain only
    
    Reference identity in case of subjectAltName match is derived only from recipient domain. If recipient domain does not match any of the dNSName entries no further reference identity is checked (like hostname derived from DNS resolution MX or PTR)

• If CN of subject DN exists (CN-ID):
  • exact or wildcard match is done against recipient domain
  • exact or wildcard match is done against hostname derived from PTR query performed against an IP of the destination server
    
    Where the PTR record preserved a consistency in DNS between forwarder and resolver. What need to be mention here, that CN field is compared against a hostname from PTR only when a PTR record exist and a resolved A record (a forwarder) for this hostname (reference identity) return an IP address which match a destination server IP against which a PTR query was performed.
    
    A(PTR(IP)) == IP
    
    Reference identity in case of CN-ID is derived from recipient domain and when there is no match a DNS query is performed against a PTR record of destination IP to get a hostname. If a PTR exists an additional query is performed against an A record on a hostname derived from a PTR to confirm that a DNS consistency is preserved ! No further reference are checked (like hostname derived from MX query)

To sum up, with  'TLS Required - Verify' option  there is no MX hostname compared with dNSName or CN, a DNS PTR RR is checked only for CN and is matched only if DNS consistency is preserved A(PTR(IP)) = IP, both exact and wildcard test for dNSName and CN are performed.

TLS Required Verify - Hosted domain

The presented identity is first derived from subjectAltName extension of type  dNSName. If there is no match between the dNSName and one of accepted reference identities (REF-ID), the verification fails no matter if CN exist in subject field and could pass further  identity verification. The CN derived from subject field is validated only when the certificate does not contain any of subjectAltName extension of type dNSName.

Recall that presented identity (dNSName or CN-ID) is compared with accepted reference identities till it is matched and in the order they are listed below. 

• If dNSName extension of subjectAltName exists:
  
  If there is no matchbetween the dNSName and one of accepted reference identities listed belowthan identity validation is failed
   
  • exact or wildcard match is done against recipient domain:  One of the dNSName must match a recipient domain
  • exact or wildcard match is done against explicitly configured hostname with SMTPROUTES (*)
  • exact or wildcard match is done against MX hostname derived from (an insecure) DNS query against the recipient domain name
    
    If recipient domain does not have explicitly configured SMTP route with FQDN entries and recipient domain was not matched than a FQDN return by a MX record from (an insecure) DNS query against a recipient domain is used. If there is no match no further tests are performed, none PTR records are checked 
• If CN of subject DN exists (CN-ID):

  
  CN is validated only when dNSName does not exists in the certificate. The CN-ID is compared with below list of accepted reference identities.
  
  

  • exact or wildcard match is done against recipient domain
  • exact or wildcard match is done against explicitly configured hostname in SMTPROUTES (*)
  • exact or wildcard match is done against MX hostname derived from (an insecure) DNS query against the recipient domain name

Explicitly configured SMTPROUTES

When SMTP route is configured and the presented identity did not match email recipient domain then all FQDN routes names are compared and if they do not match there is no further checks. With explicitly configured SMTP routes no MX hostname are considered to be compared against a presented identity. The exception here makes a SMTP route which was set as an IP address.

The following rules apply in case of explicitly configured SMTP routes:

• When SMTP route exist for a recipient domain and it is a fully qualified DNS domain name (FQDN) it is considered as a reference identity. This hostname (a route name) is compared with the presented identity received from a certificate derived from a destination server to which it is pointing.

• Multiple routes for a recipient domain are allowed. If recipient domain has more than one SMTP route, the routes are processed until the presented identifiers from certificate from destination server will match the name of the route to which the connection was established. If the hosts on the list have different priorities the ones with the highest (0 is the highest and the default) are processed first. If all have the same priority the list of routes is processed in the order the routes were set by the user.

• In case when the host does not respond (is not available) or it respond but the TLS verification has failed the next host from the list is processed. When the first host is available and passes the verification the others are not used.

• If multiple routes resolves to the same IP addresses, only one connection to this IP is established and the presented identity derived from the certificate sent by the destination server must match one of these routes name.

• If SMTP route exist for recipient domains but has been configured as an IP address, the route is still use to make a connection but a presented identity from certificate is compared against recipient domain and further with the hostname derived from the DNS/MX resource record.

When we talk about TLS Required Verify option for Hosted Domains, the way how ESA has connected with a destination server is important for TLS verification process because of the explicitly configured SMTP routes which provides additional reference identity to be considered in the process.

~= represents exact or wildcard match

Example

Let's take an example from real life, but for the recipient domain: example.com. Below I tried to describe all step which are necessary to manually verify server identity. 

example.com -> IN MX mx01.subd.emailhosted.not.
example.com -> IN MX mx02.subd.emailhosted.not.

mx01.subd.emailhosted.not. -> IN A 192.0.2.1
mx02.subd.emailhosted.not. -> IN A 192.0.2.2

192.0.2.1 -> IN PTR mx0a.emailhosted.not.
192.0.2.2 -> IN PTR mx0b.emailhosted.not.

mx0a.emailhosted.not. -> IN A 192.0.2.1
mx0b.emailhosted.not. -> IN A 192.0.2.2

Note: the MX hostnames and the revDNS names do not match in this case

$ echo QUIT |openssl s_client -connect mx0a.emailhosted.not:25 -starttls smtp 2>/dev/null| openssl x509 -text | grep -iEo 'DNS:.*|CN=.*'

CN=thawte SHA256 SSL CA
CN=*.emailhosted.not
DNS:*.emailhosted.not, DNS:emailhosted.not 

echo QUIT |openssl s_client -connect mx0b.emailhosted.not:25 -starttls smtp 2>/dev/null| openssl x509 -text | grep -iEo 'DNS:.*|CN=.*'

CN=thawte SHA256 SSL CA
CN=*.emailhosted.not
DNS:*.emailhosted.not, DNS:emailhosted.not   

•   presented identity: dNSName exist (continue with comparing with allowed reference identity)
  
  
  • reference identity = recipient domain (example.com) is checked and does not match the dNSName  DNS:*.emailhosted.not, DNS:emailhosted.not
•   presented identity: CN exist (continue with next presented identiy as for the previous one there was no match)
  
  
  • reference identity = recipient domain (example.com) is checked and does not match the CN *.emailhosted.not
    
    
  • reference identity = PTR(IP) : a PTR query is performed against the IP of the server to which the TLS client (ESA) has established connection and received a certificate, and this query  returns: mx0a.emailhosted.not.
    
    
    • DNS consistency is checked to consider this hostname as valid reference identity:
      
      
      Consistency exists: A(PTR(IP)) = IP
      
      

      mx01.subd.emailhosted.not. -> IN A 192.0.2.1
      
      PTR(IP):      192.0.2.1  -> IN PTR  mx0a.emailhosted.not.
      A(PTR(IP)):   mx0a.emailhosted.not. -> IN A 192.0.2.1
      

    • The value of mx0a.emailhosted.not. is compared against CN *.emailhosted.not and there it matches.

The PTR domain name validates the identity and as the certificate is a CA signed certificate it validates the whole certificate and TLS session is established.

• presented identity: dNSName exist (so the CN will not be processed in that case)
  • reference identity = recipient domain (example.com) is checked and
    does not match the dNSName  DNS:*.emailhosted.not, DNS:emailhosted.not
  • reference identity = FQDN(smtp route) - there is no smtproutes for this recipient domain 
    
    As there are no SMTPROUTES additionally used:
    
    
  • reference identity = MX(recipient domain) - a DNS MX query is performed against the recipient domain
    and returns: mx01.subd.emailhosted.not - this does not match the dNSName  DNS:*.emailhosted.not, DNS:emailhosted.not
• presented identity: CN exist but is skipped as dNSName exist as well.