How to Set Up X.509 Support for SunSSH on Oracle Solaris 11

by Hai-May Chao

This article describes how to set up X.509 authentication support for SunSSH on Oracle Solaris 11.


Published April 2013

About SSH

If you'd like to download software, participate in forums, and get access to other technical how-to goodies in addition to content like this, become an OTN member. No spam!

There are various programs for remote copy (ftp, rcp), remote logins (rlogin, telnet), and remote execution (rsh). However, many of these programs do not provide security to protect data during transmission. Fortunately, the Secure Shell (SSH) can overcome these security issues. SunSSH is a forked version of SSH that is available in Oracle Solaris.

SSH provides encrypted communication between a client and a server. It prevents traffic sniffing and password theft. SSH also provides a secure method for the server and client to authenticate themselves to the other side:

  • Server authentication: The user verifies the identity of the server while connecting.
  • User authentication: The server verifies the identity of the user while connecting.

SSH supports the public key authentication method used for server and user authentication. Public key authentication is based on asymmetric cryptographic algorithms. The client and server have a pair of public and private keys, and they exchange public keys during the handshake.

For server public key authentication, the server is authenticated using the server's host key. The client checks the database file .ssh/known_hosts in the user's home directory during login. If the host public key is not found in the known_hosts file, a warning message is displayed to the user. The client will continue connecting if the user answers yes to accept the authenticity of the host.

For user public key authentication, the user first generates a public and private key pair and keeps the private key secret. The public key is installed in the file ~/.ssh/authorized_keys on the server where the user will log in. Then, the server will authenticate the user using the user's public key instead of a password. The server will verify a challenge that was signed by the user's private key against the user's public key on server side.

Motivation for X.509 Authentication

Server public key authentication lacks a good method for verifying that the provided host public key really belongs to the server. For user public key authentication, this method first requires the distribution of user public keys in the user's home directory on the server.

When using X.509 certificates to do authentication, the user prompt for accepting the host identity in order to connect is removed. Using X.509 also eliminates the need to populate user public keys on the remote server.

Overview of X.509 Authentication

On the SSH server side, the user first needs to generate a public and private key pair. The private key should be stored in a safe location. The user fills in the Certificate Signing Request (CSR) with the public key along with other information, such as a distinguished name to identify the user as an applicant. (A CSR is a message request from an applicant who sends it to a certificate authority in order to apply for a digitally signed certificate.)The user then submits the CSR to a trusted Certificate Authority (CA)—an entity that signs and issues digital certificates—for signing.

The CA uses its private key to sign the certificate based on the CSR, and then appends the digital signature to the signed certificate. The CA sends the signed host certificate and its own certificate back to the user.

Upon receiving these certificates, the user installs the host certificate in the keystore of the SSH server system. The CA certificate should also be passed to the clients for subsequent server authentication.

When the SSH client tries to connect to the SSH server, the server passes the host certificate to the client. Using the CA's public key in the CA certificate, the client verifies the host certificate against the digital signature associated with the host certificate.

To simplify the CSR signing task, the user can generate a self-signed trusted anchor (TA) certificate locally via the pktool gencert command. (A self-signed certificate is a digital certificate that is signed by the entity that created the certificate, and a TA certificate is the digital certificate of the root certificate authority.) The pktool command allows users to manage the PKI objects, such as certificates and keys, on multiple keystores including PKCS#11 tokens. Refer to the pktool(1) man page for more information.

Then the user can use the pktool gencsr command to generate a CSR to be signed by the TA's private key. Similarly, the user can generate a user certificate on the SSH client side for X.509 user authentication using the same approach.

Example of Using the X.509 Certificates

The following procedure describes how to use X.509 certificates to do user authentication:

  1. On the client machine, the PKCS#11 softtoken keystore is used to store the certificates and keys. Initialize and set the PIN (passphrase) to the PKCS#11 softtoken keystore using the pktool setpin command. The default passphrase for the softtoken keystore is changeme.

    pktool setpin
     Enter token passphrase: changeme
     Create new passphrase:
     Re-enter new passphrase:
     Passphrase changed.
    
  2. On the client, generate a self-signed certificate and a key pair in the softtoken keystore using the pktool gencert command. The gencert subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore. The resulting certificate will be used as a trusted anchored certificate in this procedure.

    pktool gencert keystore=pkcs11 label=authority subject="CN=authority" serial=0x01
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the type of the keystore. The pkcs11 type is specified for the softtoken keystore in the example.
    • The label option specifies the label of the key pair and the trusted anchor certificate.
    • The subject option specifies the distinguished name for the trusted anchor certificate.
    • The serial option specifies a unique serial number for the trusted anchor certificate.
  3. On the client, export the trusted anchor certificate from the softtoken keystore using the pktool export command. The export subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool export keystore=pkcs11 outfile=ta.cert objtype=cert label=authority outformat=pem
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The outfile option specifies the output file name for the trusted anchor certificate, which is ta.cert in this example.
    • The objtype option specifies the class of the trusted anchor certificate.
    • The label option specifies the label of the trusted anchor certificate.
    • The outformat option specifies the output format for the trusted anchor certificate.
  4. As root on the server, create the directory /etc/ssh/cert. Then place ta.cert in the /etc/ssh/cert directory. This is done to install the trusted anchor certificate so it will be used to validate the user certificate from the client side.
  5. On the client, generate a certificate signing request (CSR) using the pktool gencsr command. The gencsr subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool gencsr keystore=pkcs11 label=user outcsr=user.csr subject="CN=<USERNAME>"
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The label option specifies the label of the private key created in the softtoken keystore.
    • The outcsr option specifies the output CSR file name to write to.
    • The subject option specifies the distinguished name of the certificate request.
  6. On the client, use the pktool signcsr command to sign the CSR with the trusted anchor private key to create a user certificate. The signcsr subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool signcsr keystore=pkcs11 signkey=authority csr=user.csr outcert=user.cert serial=0x02 issuer="CN=authority"
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The signkey option specifies the label of the trusted anchor private key.
    • The csr option specifies the input CSR to be signed.
    • The outcert option specifies the output file name for the user certificate, which is user.cert in this example.
    • The serial option specifies a unique serial number for the user certificate.
    • The issuer option specifies the issuer name with the trusted anchor's distinguished name in the user certificate.
  7. On the client, import the user certificate into the softtoken keystore using the pktool import command.

    pktool import keystore=pkcs11 infile=user.cert label=user
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The infile option specifies the file name of the user certificate to be imported.
    • The label option specifies the label for the user certificate.
  8. As root on the server, create a Key Management Framework (KMF) policy database file using the kmfcfg create command. The KMF policy database can constrain the use of keys and certificates that are managed through the KMF framework. Refer to the kmfcfg(1) man page for more information.

    kmfcfg create dbfile=/etc/ssh/policy.xml policy=ssh ta-name=search mapper-name=cn
    

    • The dbfile option specifies the name of policy database file.
    • The policy option specifies the name of policy record to be created in the policy database file.
    • The ta-name option specifies the value search in this example, so the KMF policy will locate a certificate that matches the issuer name of the certificate to be validated and it will use that for the validation.
    • The mapper option specifies mapping a certificate to a name with the cn mapper, which maps a certificate to its value from the Common Name attribute.
  9. As root on the server, put the TrustedAnchorKeystore, KMFPolicyDatabase, and KMFPolicyName options in the sshd_config configuration file. These are used to specify the attributes associated with the trusted anchor certificates. See sshd_config(4) for details.

    The keyword-value pairs in sshd_config are as follows:

    cat /etc/ssh/sshd_config
     ...
     TrustedAnchorKeystore /etc/ssh/cert
     KMFPolicyDatabase /etc/ssh/policy.xml
     KMFPolicyName ssh
    

    • The TrustedAnchorKeystore keyword specifies a directory where certificates of trusted anchors are located.
    • The KMFPolicyDatabase keyword specifies the file name for the policy database file.
    • The KMFPolicyName keyword specifies the name of the policy to be used.
  10. On the server, re-enable the sshd service instance using svcadm. The svcadm command allows users to manipulate service instances. Refer to the svcadm(1M) man page for more information. The SSH daemon will validate the user certificate of the connecting client.

    svcadm disable svc:/network/ssh
    svcadm enable svc:/network/ssh
    
  11. On the client, the user logs in to the server using the user certificate as the user identify in the softtoken keystore:

    ssh -o IdentityFile="pkcs11:object=user;token=Sun Software PKCS#11 softtoken" <USERNAME>@<HOSTNAME>
     Enter PIN for 'Sun Software PKCS#11 softtoken': 
    

    • The IdentityFile keyword specifies the user certificate in the softtoken keystore by the PKCS#11 URI scheme. Refer to ssh(1) man page.
    • The object attribute provides the label of the certificate object in the softtoken keystore.
    • The token attribute identifies a PKCS#11 token by its token label.

The following procedure describes how to use X.509 certificates to do server authentication:

  1. On the client, make sure the known_hosts file in the ~username/.ssh directory does not contain the host public key entry that the client is connecting to.
  2. On the server, the PKCS#11 softtoken keystore is used to store the certificates and keys. Initialize and set the PIN (passphrase) to the PKCS#11 softtoken keystore using the pktool setpin command. The default passphrase for the softtoken keystore is changeme.

    pktool setpin
     Enter token passphrase: changeme
     Create new passphrase:
     Re-enter new passphrase:
     Passphrase changed.
    
  3. On the server, generate a self-signed certificate and a key pair in the softtoken keystore using the pktool gencert command. The resulting certificate will be used as a trusted anchored certificate in this procedure. The gencert subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool gencert keystore=pkcs11 label=authority subject="CN=authority" serial=0x01
     Enter PIN for Sun Software PKCS#11 softtoken: 
    

    • The keystore option specifies the type of the keystore. The pkcs11 type is specified for the softtoken keystore in the example.
    • The label option specifies the label of the key pair and the trusted anchor certificate.
    • The subject option specifies the distinguished name for the trusted anchor certificate.
    • The serial option specifies a unique serial number for the trusted anchor certificate.
  4. On the server, export the trusted anchor certificate from the softtoken keystore using the pktool export command. The export subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool export keystore=pkcs11 outfile=ta.cert objtype=cert label=authority outformat=pem
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The outfile option specifies the output file name for the trusted anchor certificate, which is ta.cert in this example.
    • The objtype option specifies the class of the trusted anchor certificate.
    • The label option specifies the label of the trusted anchor certificate.
    • The outformat option specifies the output format for the trusted anchor certificate.
  5. As root on the client, create the directory /etc/ssh/cert, and place ta.cert in the /etc/ssh/cert directory. This is done to install the trusted anchor certificate so it will be used to validate the host certificate from the server side.
  6. On the server, generate a CSR using the pktool gencsr command. The gencsr subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool gencsr keystore=pkcs11 label=host outcsr=host.csr subject="CN=<HOSTNAME>"
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The label option specifies the label of the private key created in the softtoken keystore.
    • The outcsr option specifies the output CSR file name to write to.
    • The subject option specifies the distinguished name of the certificate request.
  7. On the server, use the pktool signcsr command to sign the CSR with the trusted anchor private key to create a host certificate. The signcsr subcommand of pktool will prompt the user to enter a PIN for the softtoken keystore.

    pktool signcsr keystore=pkcs11 signkey=authority csr=host.csr outcert=host.cert serial=0x03 issuer="CN=authority"
     Enter PIN for Sun Software PKCS#11 softtoken:
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The signkey option specifies the label of the trusted anchor private key.
    • The csr option specifies the input CSR to be signed.
    • The outcert option specifies the output file name for the host certificate, which is host.cert in this example.
    • The serial option specifies a unique serial number for the host certificate.
    • The issuer option specifies the issuer name with the trusted anchor's distinguished name in the host certificate.
  8. On the server, import the host certificate into the softtoken keystore using the pktool import command.

    pktool import keystore=pkcs11 infile=host.cert label=host
    

    • The keystore option specifies the keystore type for the softtoken keystore.
    • The infile option specifies the file name of the host certificate to be imported.
    • The label option specifies the label for the host certificate.
  9. As root on the client, create a Key Management Framework (KMF) policy database file using the kmfcfg create command.

    kmfcfg create dbfile=/etc/ssh/policy.xml policy=ssh ta-name=search mapper-name=cn
    

    • The dbfile option specifies the name of policy database file.
    • The policy option specifies the name of policy record to be created in the policy database file.
    • The ta-name option specifies the value, which is search in this example. Then the KMF policy will locate a certificate that matches the issuer name of the certificate to be validated and it will use that for the validation.
    • The mapper option specifies mapping a certificate to a name with the cn mapper, which maps a certificate to its value from the Common Name attribute.
  10. On the server, as root, create a PIN file containing the passphrase for the softtoken keystore.

    printf "PIN" > /etc/ssh/pinfile
    
  11. As root on the server, put the HostKey option for the host certificate in the sshd_config file. The host certificate is specified using the PKCS#11 URI scheme. Refer to sshd(1M).

    The keyword-value pair in sshd_config is as follows:

    cat /etc/ssh/sshd_config
     ...
     HostKey       pkcs11:object=host;token=Sun Metaslot;pinfile=/etc/ssh/pinfile
    

    • The object attribute in the PKCS#11 URI provides the label of the certificate object in the softtoken keystore.
    • The token attribute identifies a PKCS#11 token by its token label.
    • The pinfile attribute provides the passphrase of the softtoken keystore.
  12. On the server, re-enable the sshd service instance using svcadm. The SSH daemon will use the host certificate as the host key.

    svcadm disable svc:/network/ssh
    svcadm enable svc:/network/ssh
    
  13. On the client, the user logs in to a server using the TrustedAnchorKeystore, KMFPolicyDatabase, and KMFPolicyName options to specify a trusted anchor certificate, which is used to validate the host certificate.

    ssh -o TrustedAnchorKeystore=/etc/ssh/cert -o KMFPolicyDatabase=/etc/ssh/policy.xml 
    -o KMFPolicyName=ssh <USERNAME>@<HOSTNAME>
    

    • The TrustedAnchorKeystore option specifies a directory where certificates of trusted anchors are located.
    • The KMFPolicyDatabase option specifies the file name for the policy database file.
    • The KMFPolicyName option specifies the name of the policy to be used.

See Also

About the Author

Hai-May Chao is a principal software engineer in the Oracle Solaris Security Technologies group. She is currently working on technology areas that include the Oracle Solaris Cryptographic Framework and Key Management Framework.

Revision 1.0, 04/02/2013

Follow us:
Blog | Facebook | Twitter | YouTube