TLS/SSL Client Certificates and Stream Decryption

I recently ran into a situation where I wanted to decrypt a TLS/SSL stream, but I only had the client certificate private key available and not the server certificate private key -- quite a common situation when client certificate authentication happens to be required! I was under the incorrect assumption that a key exchange protocol had been used during the transaction where the client used its private key as a source for the resulting master secret. I have come to learn that this was not the case, and it is here that I will describe why.

For those unfamiliar, TLS/SSL is an application layer protocol that uses both asymmetric and symmetric encryption to secure the data being transmitted. For the majority of ciphers, asymmetric encryption is briefly used to generate a shared master secret (and/or authenticate the endpoints if appropriate), which is then used in the symmetric encryption portion of the client/server exchanges to exchange application data back and forth. There are many libraries available for application developers to use, but more often on one simply provided by the operating system. To be able to decrypt a stream and view the contents of the exchange, one must have the shared master secret used in the symmetric encryption of that data. To get that shared master secret after an exchange has completed, and all your left with is a pcap file of the transactions after the session has terminated, you need to recalculate it based on the information exchanged during key agreement of the handshake, which includes the pre-master secret that the master secret is derived from. In the case of simple RSA key agreement, since the server public and private key pairs of the endpoints are used as part of this process to generate the master secret, they are necessary in order to recalculate it.

I only had the client certificate private key and at first, my investigation started with exploring the problem on why Wireshark was attempting to associate the client certificate private key I provided within the SSL protocol preferences to only the server certificate in the handshake, and did not attempt to apply it to the client certificate even though I used the IP address of the client when making the private key available to Wireshark. Some further investigation led me to realize that the client certificate public or private keys are not used in the generation of the master secret used for the transaction.

So this lead me to the following two questions:

• Are there any key exchange protocols that use the client certificate as part of the key generation material? 
• For plain RSA, what's the process used to generate the pre-master secret?

It's best to step back and walk through the negotiation phase of the SSL handshake to understand how the client and server determine the final master secret. The full handshake is depicted as such in the TLS RFC:

      Client                                             Server
      ------                                             ------

      ClientHello             --------> 
                                                    ServerHello
                                                   Certificate*
                                             ServerKeyExchange*
                                            CertificateRequest*
                              <--------         ServerHelloDone
      Certificate*
      ClientKeyExchange
      CertificateVerify*
      [ChangeCipherSpec]
      Finished                -------->
                                             [ChangeCipherSpec]
                              <--------                Finished
      Application Data        <------->        Application Data

             Figure 1.  Message flow for a full handshake

   * Indicates optional or situation-dependent messages that are not
   always sent.

During the Client Hello, the endpoint proposes all the ciphers it supports to the server. The server in turn then selects only one of those and advertises it in its Server Hello. At this point, both client and server know which cipher suite to use. If the server did not like any of the proposed ciphers by the client, it could immediately send an alert and end the connection, leaving the client out of luck. With that though, we have to make an addendum to my earlier comment on how TLS/SSL uses both asymmetric encryption and symmetric encryption to protect data. TLS can be broken into three components:

• Key Exchange 
• Privacy 
• Integrity

These three components can be seen in the offered cipher suites by the client, which again can be easily seen using Wireshark or by taking looking a look at the TLS/SSL RFCS or OpenSSL's supported cipher list. Let's use one particular, common example: TLS_RSA_WITH_AES_128_CBC_SHA. This tells us multiple things:

1. This cipher suite was introduced in TLS 1.0 or later. 
2. It specifies to use RSA for asymmetric encryption, and in this case the key exchange as well. 
3. It specifies to uses the chain-block cipher AES for symmetric encryption, with a 128-bit key. 
4. Lastly, it cites SHA-1 for MAC calculation.

We care about the Key Exchange portion here. The handshake I described above takes place when using RSA, which in this context is considered an authenticated key exchange protocol using public key cryptography. There are other key exchange protocols, including the unauthenticated and authenticated Diffie-Hellman protocol, but we'll focus on plain RSA here. Since this example cipher does not utilize an anonymous key exchange protocol, the server must send the client a certificate after the Server Hello whose type is appropriate for the determined cipher. The server will not send over a Server Key Exchange message as the client already has the public key of the server for use in exchanging the determined pre-master secret, and confirming that the server owns the private key to that server certificate. A Server Key Exchange message is necessary for key exchange protocols that are not using the public keying material in the described behavior, which includes almost all iterations of Diffie-Hellman (except when the material is included in the certificates, as is the case for Fixed Diffie-Hellman).

After the Server Hello, the server may send a Certificate Request message to the client, instructing it to send the server the client's own certificate if it has one present. Just like the server certificate, this client certificate has an appropriate private and public key, which will be used shortly by the server to ensure the client actually has the private key associated with that certificate. If the client does not have an available certificate, it will not send anything (in TLS 1.2 the client will send a Certificate message with an empty certificate list), at which point the server can treat this as a fatal error or continue with the connection. Its fairly rare to find a deployment of HTTPS (HTTP over SSL/TLS) to require a client certificate (if the cipher does not require it), but the client certificate requirement is much more common in applications that implement SSL/TLS to conduct secure, dual-authenticated client-server transactions. The server finishes its first part with a Server Done message.

As mentioned, if the client has a certificate, it will now send it over to the server. The next expected message sent by the client is the important Client Key Exchange message, which is always going to be sent, even though the contents of it may not be necessary. In the case of Fixed Diffie-Hellman, this information could be contained in the certificate the client just sent to the server so the Client Key Exchange message may be left blank. For all others, this message is either going to contain the RSA-encrypted pre-master secret or the client's Diffie-Hellman's parameters.

The RSA pre-master secret is a 48 byte value, generated by client, composed of the 2 byte client version found within the Client Hello and 46 (pseudo)random bytes. So how is the remaining 46 byte stream created? Well, it's entirely up to the SSL/TSL library the client application is using, unless the client implemented the SSL/TLS protocol itself. Here is the pre-master secret generation code snippet from OpenSSL 1.0.1c:

       unsigned char tmp_buf[SSL_MAX_MASTER_KEY_LENGTH];
       ...
       tmp_buf[0]=s->client_version>>8;
       tmp_buf[1]=s->client_version&0xff;
       if (RAND_bytes(&(tmp_buf[2]),sizeof tmp_buf-2) <= 0)
           goto err;

OpenSSL relies on its own RAND_bytes function to generate a stream of 46 bytes if its using its provided engine. If best practices are followed correctly, the library used (or the entity that implemented SSL/TLS) is to delete the pre-master secret from memory as soon as the master secret is calculated. By the time this occurs, the pre-master secret has been encrypted using the server's public key and sent out on the wire, which means at this point only the server can decrypt the contents of the Client Key Exchange using its private key due to the properties of asymmetric encryption.

If the client sent a certificate to the server with a digitalSignature bit enabled (which is required for all cases except Fixed Diffie-Hellman), it will also send a Certificate Verify message that will be a signed hash of all messages exchanged between the client and server prior to the forthcoming message and including the Client Hello. A Change Cipher Spec message follows from the client, as well as a Finished message that acts as a test of the negotiated master secret determined during the handshake. Much like the Certificate Verify message, this is a hash of all exchanges between the client and server excluding the current message and the required prior Change Cipher Spec message, which is run through the pseudo-random function defined by the cipher using the negotiated master secret. The client or server may begin sending encrypted data back and forth once they have received and validated the Finished message from the other endpoint.

The master secret is derived the same way for RSA and Diffie-Hellman key exchanges. The only difference between those two is how the pre-master secret is calculated. For all master secrets, as specified in the TLS 1.0 and 1.2 RFC:

       master_secret = PRF(pre_master_secret, "master secret",
                           ClientHello.random + ServerHello.random)
                           [0..47];

The last integer refers to a random number that was included in both the Client and Server Hello messages, with the bracketed material stating that it will be 48 bytes. The PRF cited is outlined in the TLS 1.0 and TLS1.2 RFCs.

With this new information, let's revisit the earlier questions:

• Are there any key exchange protocols that use the client certificate as part of the key generation material?

Yes, but only one type: Fixed Diffie-Hellman. However, a client certificate is not a necessary required part of the key exchange (the client will send its public value in a Client Key Exchange message if it does not have a certificate), so the server may not even ask for it depending on the implementation.

• For plain RSA, what's the process used to generate the pre-master secret? 

This is dependent upon the application itself, and what library it wants to use. It could rely on an operating system provided library, a third-party library it includes in the application, or it could attempt to implement SSL/TLS itself. Time has shown that mistakes are common in the field of cryptography, between researchers finding flaws in ciphers considered cryptographically secure when they were drafted, to over-confident software engineers implementing a layer of cryptography in their application.

This is why you must have the server certificate private key, including the entire handshake in a capture, to decrypt a particular SSL/TLS stream if the session has already completed. A forthcoming article will discuss how to pull a master secret from memory and allow you to decrypt the stream before the session terminates.

More information about the above topics can be found from their associated links contained within the article, in addition the following documents were referenced:

• RFC2246 - TLS Protocol - Version 1.0 
• RFC5246 - TLS Protocol - Version 1.2 
• OpenSSL 1.0.1c source