HOWTO: Solve the “Server certificate was rejected by the verifier” exception

Note:If using release 2012 R3 or later, please check out an updated version of this blogpost!

If you wrote an application using Rebex FTP/SSL or Rebex Secure Mail component and get the “Server certificate was rejected by the verifier because…” error when connecting to a server secured using the TLS/SSL protocol, this HOWTO is just what you need. In order to understand the problem, at least basic understanding of public key certificates is helpful. If this is still a gray area for you, just check out our Introduction to Public Key Certificates.

The actual exception message varies, but the reason is similar – either something is wrong with the certificate or it is not trusted by the client.

The following exceptions can occur as a result of failed certificate verification:

  1. Server certificate was rejected by the verifier because the
    certificate's common name '…' does not match the hostname '…'.
  2. Server certificate was rejected by the verifier because it has
    expired.
  3. Server certificate was rejected by the verifier because it is
    revoked.
  4. Server certificate was rejected by the verifier because of an
    unknown certificate authority.
  5. Server certificate was rejected by the verifier because it is bad.
  6. Server certificate was rejected by the verifier because of other
    problem.
  7. Server certificate was rejected by the verifier because it is
    unsupported.

(Previous releases of Rebex components used slightly different versions of these message, but they are very similar.)

In general, when any of these errors occur, you have the following options:

​a) Solve the problem. These error messages indicate that something is wrong with the certificate or certificate infrastructure. Fixing the problem is the proper thing to do, although it might not be always possible. See the instructions below for a detailed descriptions of various error conditions.

​b) If this is the “common name” mismatch problem (no. 1 in the list), see the instructions below for more information.

​c) Implement a custom certificate verifier - check out the appropriate section of our FTP/SSL tutorial or Secure Mail TLS/SSL tutorial for details. Also, check out the WinFormClient sample (in Rebex FTP/SSL) or Pop3Browser/ImapBrowser samples (in Secure Mail) - these include a custom verifier that asks the user whether to accept or reject a certificate if in doubt, and it even has a capability to automatically add the certification authority certificate into the appropriate store if the problem occurred because of an unknown certification authority (no. 4) - you can simply re-use this verifier in your application or modify it to suit your needs, because the source files that implement the verifier (Verifier.cs/.vb and VerifierForm.cs/.vb) are independent of the rest of the samples.

​d) Implement a custom certificate verifier that accepts or reject certificates based on their hash code (also known as fingerprint) – this value uniquely identifies a certificate, making it possible to be used as its ID:

C #:
// A custom certificate verifier class.
// Implements ICertificateVerifier interface.
public class FingerprintVerifier : ICertificateVerifier
{
    // This method gets called during the SSL handshake
    // process when the certificate chain is received from the server.
    public TlsCertificateAcceptance Verify(
        TlsSocket socket, string commonName, CertificateChain certChain)
    {
        // get a string representation of the certificate's fingerprint
        string fingerprint =
           BitConverter.ToString(certChain.LeafCertificate.GetCertHash());

        // check whether the fingerprint matches the desired fingerprint
        bool ok = ...

        if (ok)
            return TlsCertificateAcceptance.Accept;
        else
            return TlsCertificateAcceptance.Other;
    }
}

...

// To make your validator the current validator for an FTP session:

Ftp ftp = new Ftp();

// Create an instance of TlsParameters class and
// set the certificate verifier to an instance of CustomVerifier.
TlsParameters par = new TlsParameters();
par.CertificateVerifier = new FingerprintVerifier();

// Connect securely using explicit SSL.
// The third argument refers to the parameters class
ftp.Connect(hostname, 21, par, FtpSecurity.Explicit);

...
VB.NET:
' A custom certificate verifier class. 
' Implements ICertificateVerifier interface. 
Public Class FingerprintVerifier
Implements Rebex.Net.ICertificateVerifier

' This method gets called during the SSL handshake
' process when the certificate chain is received from the server.
Public Overloads Function Verify(ByVal socket As TlsSocket, _
    ByVal commonName As String, _
    ByVal certChain As CertificateChain) _
    As TlsCertificateAcceptance _
    Implements ICertificateVerifier.Verify

    ' get a string representation of the certificate's fingerprint
    Dim fingerprint As String = _
        BitConverter.ToString(certChain.LeafCertificate.GetCertHash())

    ' check whether the fingerprint matches the desired fingerprint
    Dim ok As Boolean = ...

    If ok Then
        Return TlsCertificateAcceptance.Accept
    Else
        Return TlsCertificateAcceptance.Other
    End If
End Function End Class

...

' To make your validator the current validator for an FTP session:

Dim ftp As New Ftp()

' Create an instance of TlsParameters class and 
' set the certificate verifier to an instance of CustomVerifier. 
Dim par As New TlsParameters() par.CertificateVerifier = New FingerprintVerifier()

' Connect securely using explicit SSL. 
' The third argument refers to the parameters class. 
ftp.Connect(hostname, 21, par, FtpSecurity.Explicit)

...

In practice, an application would have to keep a database of acceptable IDs and validate certificates according to it. However, this approach deviates from a standard public key infrastructure practices and should only be used if other solutions are unsuitable.  

e) Use the built-in AcceptAll verifier - this will undoubtedly make the certificate accepted, but doing this in production environment is highly discouraged, as it effectively disables server authentication, bypassing one of the key features of TLS/SSL. Even though this will make it possible for you to connect to the server, there will be no proof that you are actually connected to the desired server.

C #:
// create an instance of TlsParameters class
TlsParameters parameters = new TlsParameters();

// accept any certificate (don't do this in production environment!)
parameters.CertificateVerifier = CertificateVerifier.AcceptAll;

// connect to the server
ftp.Connect(hostname, 21, parameters, FtpSecurity.Explicit);

...
VB.NET:
' create an instance of TlsParameters class 
Dim parameters As New TlsParameters()

' accept any certificate (don't do this in production environment!) 
parameters.CertificateVerifier = CertificateVerifier.AcceptAll;

' connect to the server 
ftp.Connect("www.rebex.net", 21, parameters, FtpSecurity.Explicit)

... 

(Please note that Rebex.Net.SecureSocket.dll has to be referenced by your project for this to compile.)

Now, let’s look at the different error conditions.

1. Common name does not match the hostname

This one is usually easy to solve. The server hostname you used in the Connect method call’s serverName argument has to match the server name that is embedded in the certificate’s common name attribute (part of certificate subject string). If they don’t match, it means, for example, that the server you know as “www.rebex.net” is presenting a certificate that belongs to “server01.rebex.net”, and this is an equivalent of one guy presenting another’s passport at the airport – while the document may be entirely valid, the two just don’t belong together.

Possible solutions:

​a) Try using the proper server name present in the certificate to connect to the server. If the certificate says the server is “server01.rebex.net” when you connect to “www.rebex.net”, then perhaps it is in fact its proper name and you wish to connect to “server01.rebex.net” instead of “www.rebex.net” which is just an alias.

​b) If the common name in the certificate cannot be used to connect to the desired server (for example, it is a local domain-less name such as “server01”), the proper thing to do is to issue a new certificate for the server that contains the correct name. If this is not possible, try the solution below.

​c) If the common name in the certificate cannot be used to connect to the desired server, but you are unable to issue a new certificate (for example if the server is not under your control), use the variant of Connect method that accepts a TlsParameters argument and specify the expected common name:

C #:
// create an instance of TlsParameters class
TlsParameters parameters = new TlsParameters();

// specify the expected common name
parameters.CommonName = "server01";

// connect to the server
ftp.Connect("www.rebex.net", 21, parameters, FtpSecurity.Explicit);

...
VB.NET:
' create an instance of TlsParameters class 
Dim parameters As New TlsParameters()

' specify the expected common name 
parameters.CommonName = "server01"

' connect to the server 
ftp.Connect("www.rebex.net", 21, parameters, FtpSecurity.Explicit)

... 

​d) One of our users also encountered a server that presented a different certificate with different common name each time they connected. In this situation, the only workaround is to write a custom certificate verifier using the instructions in our tutorial.

2. Server certificate was rejected by the verifier because it has expired.

This happens when the validity period of the server certificate is over. When the certificate expires, the proper thing to do is to replace it with a new certificate. If this can’t be done (if you don’t have control over the server and its administrators are refusing to solve the problem), the solution is to write a custom certificate verifier using the instructions in our tutorial.

3 Server certificate was rejected by the verifier because it is revoked.

This is a problem that is worth paying attention to. It means that the certificate signature has been revoked by its signer and that the certificate should no longer be used and trusted. A custom certificate verifier can also override this problem, but doing so is strongly discouraged.

4 Server certificate was rejected by the verifier because of an unknown certificate authority.

This error message means that the certificate authority that issued the server certificate is not in the list of trusted authorities. For this reason, the identity of the server could not have been verified. This problem can be solved by adding the certificate of the root CA that signed the server certificate into the list of trusted certification authorities as described by the previous post about public key certifciates, or using Rebex CertificateStore class - check out the Verifier.cs/.vb file in the WinFormClient or Pop3Browser/ImapBrowser samples for information on how to do this in a custom certificate verifier.

5, 6 and 7…

The following three possible errors still remain:

5 Server certificate was rejected by the verifier because it is bad.

6 Server certificate was rejected by the verifier because of other problem.

7 Server certificate was rejected by the verifier because it is unsupported.

While no one ever reported any of these problems, they can happen. If you get one of these, please let us know and mail us the communication log produced using the new LogWriter functionality with LogLevel.Verbose.

That’s all for today – if you are still stuck with an unusable certificate after trying all the tricks above, please let us know!