Certificate expire issue in Websphere Application server

RECOMMENDATION: Servers self-signed certificate will get replaced 60 days before they expire. That means about 10 months after the self-signed certificate gets created. This will cause a server outage on services like WebServer where the managing of the client signer certificate is a manual step. So this change will extend the life span of the default self-signed certificate to 15 years and provide addition warning time before certificates are automatically replaced.

 In WAS 6.1 the default certificate expires in one year. Just before the expiration, the cert is renewed automatically. After this automatic cert renewal, dmgr cannot talk to nodeagents, resulting in “JSSL0080E SSL HandShake Execption”. If the renewal is done while WAS is up and running, the user has to update dmgr/trust.p12 and appsrv/trust.p12 when prompted during the next WAS shutdown. This does not work If WAS is running as a service on Windows platforms. If the cert is expired while WAS is NOT running, WAS has to be started with expired cert. Automatic renewal runs during the next start-up of dmgr. The user has to run sync node.

As a work around, the user currently has to add manually the renewed certs to the trust stores. Add the cert of Cell to Node, and the other one of Node to Cell. The error is produced as a direct result of automatice cert renewal. The renewed cert should be added to Cell and Node trust stores automatically.
Additionally, the certificate expiration monitor has been modified to properly handle this condition; this fix has been shipped in APAR PK48659. Local fix As a work around, the user currently has to add manually the renewed certs to the trust stores. Add the cert of Cell to Node, and the other one of Node to Cell.

How to create & add a new Signer certificates for existing profile.
Create new key.p12 keystore
SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates
Create Self Signed Cert
Alias:
Common Name:
Validity Period: 3650
Organization: xyz

Click OK
Extract certificate
SSL certificate and key management > Key stores and certificates > CellDefaultKeyStore > Personal certificates > Extract certificate

Certificate file name: -key.arm ex: Server Name-key.arm
Data type: Base64-encoded ASCII data

Default location for file to be created is:
/profiles/dmgr/etc/ ex: /opt/was61/profiles/dmgr/etc/

Import certificate created trust.p12
SSL certificate and key management > Key stores and certificates > CellDefaultTrustStore > Signer certificates

Alias: -key
File Name: /profiles/dmgr/etc/-key.arm
Data type: Base64-encoded ASCII data

Copy trust.p12 and key.p12 to all the nodes
FROM:
/profiles/dmgr/config/cells//trust.p12
/profiles/dmgr/config/cells//key.p12

TO: /profiles/dmgr/config/cells//nodes//trust.p12

/profiles/dmgr/config/cells/ /nodes/ /key.p12

Restart nodes and dmgr from command line. When prompted to accept certificate, accept the certificate.
Start dmgr
Sync nodes manually to dmgr. When prompted to accept certificate, accept the certificate.

8. SSL certificate and key management > SSL configurations > CellDefaultSSLSettings

Select the certificate that you created in the following drop downs:
Default server certificate alias
Default client certificate alias
Click ->Get Certificate Aliases
Click -> OK

9. SSL certificate and key management > Manage endpoint security configurations
NOTE: This is a similar process the needs to be completed for all nodes and cells, both inbound and outbound
Select Node Level:
Change
Certificate alias in key store: Certificate that you imported

Click -> Update Certificate Alias List

Click -> OK

Repeat for Node Level – Inbound and Outbound
Repeat for Cell Level – Inbound and Outbound

SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore

If you are a WebSphere Administrator or any Application Server (WebLogic, Tomcat) administrator you might already know that managing the ssl certificates in a large complex environments becomes hectic and troublesome because of the different expiration dates of the certificates that websphere uses and also the ssl certificates of the external systems (like SAP , Siebel) that websphere applicaton server might connect to using a secure connection, multiple administrators in an organization renewing it and not keeping tracking of the expiration dates. The problem is , SSL certificate might expire resulting in servers will not initialize and your running servers will stop operating, becomes unresposive if not properly renewed on time. Hence this article will explain how you can monitor the expiration of the ssl certificates using a simple command and proactively monitor the expiration dates, setup your calendar and renew it on time to prevent any downtimes.


If you are using the latest version (as on Dec 2007) of WebSphere Application Server 6.1 and using default self signed there are provisions in the application server itself to effectively monitor the certificate expiration, notifying you and renewing it automatically. If you are using a perosnal self signed certificate or a personal certificate signed by a certificate authority (CA) like VeriSign or Thawte , which is most likely in all of your production environments then you might want to use the below command to find the expiration date and renew it.

If you are using a java keystore ( DummyServerKeyFile.jks , DummyServerTrustFile.jks , jssecerts , cacerts) use the below find command along with keytool or websphere's ikeyman command line option to find the expiration of the keys.

LINUX:


Using Ikeyman cmdline utility (ikeyman.sh):

find PATH -name *.jks xargs -i bash -c '$WAS_HOME/java/bin/java -classpath $WAS_HOME/java/jre/lib/ext/ibmjceprovider.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmjcefw.jar:$WAS_HOME/AppServer/java/jre/lib/ext/US_export_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/local_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmpkcs.jar:$WAS_HOME/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw
grep "Not After:.*yyy[y,y]" -B 3 && echo {}'

(e.g)


find /usr/IBM/WebSphere/AppServer/profiles/ -name *.jks xargs -i bash -c '/usr/IBM/WebSphere/AppServer/java/bin/java -classpath /usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/AppServer//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw WebAS grep "Not After:.*200[7,8]" -B 3 && echo {}'


Using Java Keytool utility:


find $WAS_HOME -name '*.jks' xargs -i bash -c "echo {}; $WAS_HOME/java/jre/bin/keytool -list -v -keystore {} -storepass password grep 'until: .*/.*/Y[Y,Y]' -A 3 -B 7"

(e.g)

find /usr/IBM/WebSphere/AppServer -name '*.jks' xargs -i bash -c "echo{};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS grep 'until: .*/.*/0[7,8]' -A 3 -B 7"

Output:


/usr/IBM/WebSphere/AppServer/profiles/etc//WWWStageClientKey.jksAlias name: verisign class 1 ca individual subscriber-persona not validatedCreation date: Dec 18, 2006Entry type: trustedCertEntryOwner: CN=VeriSign Class 1 CA Individual Subscriber-Persona Not Validated, OU="www.verisign.com/repository/RPA Incorp. By Ref.,LIAB.LTD(c)98", OU=VeriSign Trust Network, O="VeriSign, Inc."Issuer: OU=Class 1 Public Primary Certification Authority, O="VeriSign, Inc.", C=USSerial number: d8b4feeaad218df5bf4756a9d29e17ffbValid from: 5/12/98 12:00 AM until: 5/12/08 11:59 PMCertificate fingerprints: MD5: CA:66:3C:FC:71:2B:BA:41:92:71:DD:72:AD:E5:65:65 SHA1: 12:51:9A:E9:CD:77:7A:56:01:84:F1:FB:D5:42:15:22:2E:95:E7:1F--Creation date: Dec 18, 2006Entry type: keyEntryCertificate chain length: 1Certificate[1]:Owner: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=USIssuer: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=USSerial number: 458709efValid from: 12/18/06 9:36 PM until: 12/18/07 9:36 PMCertificate fingerprints: MD5: C2:4F:CC:SS:19:DC:E4:88:B0:2A:78:76:69:4D:DC:EF:47 SHA1: 53:AB:D0:13:45:45:12:59:64:F6:C9:38:41:F2:C8:E3:37:05:73:95:F3:83/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jksAlias name: www stage websphere server caCreation date: Dec 8, 2006Entry type: keyEntryCertificate chain length: 1Certificate[1]:Owner: CN=wadm.stage.www.com, OU=IT, O="www, Inc.", L=Fremont , ST=California, C=USIssuer: OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=USSerial number: 1ea5f21b33ca65e551440d662399dffdfga382Valid from: 12/5/06 12:00 AM until: 12/15/07 11:59 PMCertificate fingerprints: MD5: C2:22:E4:12:12:6D:3F:22:67:4E:37:48:12:3F:F3:1B:FE:26 SHA1: CA:81:37:A7:E9:12:22:19:27:D9:3F:C5:79:E9:25:C8:1A:3C:1E:5C:17:02

As you can see in the above output the command will list the file name (/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks) , certificate label (www stage websphere server ca) , Entry Type (keyEntry - personal cert or trustedCertEntry - Signer Cert), and the expiration date (12/5/06 12:00 AM until: 12/15/07 11:59 PM) of


all the java keystore (*.jks) in the given directory that expired in 2007 (07) or 2008 (08). By which you shoule be able to indentify the certs, setup your calendat atleast 10 days before the expiration and renew it to proactively to preveny any problems.

Note you can also change the above command by passing differnt filename like cacerts and jssecerts and also pass different year infomration to find expirations during those years.

Sometimes you might get no output which means either there are no expiring ceritificates for the years that you passed or if you are expecting any expiring certificate to show and if not then the password for the keystore might be wrong, so run without the grep portion in the above command which will list all the certificates irrespective of the expiration date or it will complain if the password is wrong as below.


find /usr/IBM/WebSphere/AppServer -name '*.jks' xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS



/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks

java.io.IOException: Keystore was tampered with, or password was incorrect

Also you can use the same command by modifying the -type parameter to "-type cms" for the webserver such as IBM HttpServer where it's using cms database ( kdb file).



find /usr/IBM/HttpServer -name *.kdb xargs -i bash -c '/usr/IBM/WebSphere/Plugins/java/bin/java -classpath /usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/Plugins//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type cms -pw WebAS grep "Not After:.*200[7,8]" -B 3 && echo {}'

Enable SSL between WebServer (plugin-in) and the WebSphere Application Server

Enable SSL between WebServer (plugin-in) and the WebSphere Application Server


To setup a new SSL-Connection between a IBM HTTP Server (IHS) and your WebSphere Application Server (6.1) a (self-signed) CA SSL certificate has to be propageted to all involved servers.

When setting up a IHS via the administrative console SSL between IHS and WebSphere AppServer should be enabled by default.

1. First take a look at your plugin-cfg.xml of your IHS installation and search for the entry Property Name="keyring"...




ServerCluster CloneSeparatorChange="false" GetDWLMTable="false" IgnoreAffinityRequests="true" LoadBalance="Round Robin" Name="server1_testNodeoglxanclatest32Bit_Cluster" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60"

Server ConnectTimeout="0" ExtendedHandshake="false" MaxConnections="-1" Name="testNodeoglxanclatest32Bit_server1" ServerIOTimeout="0" WaitForContinue="false"

Transport Hostname="oglxanclatest" Port="9080" Protocol="http"/

Transport Hostname="oglxanclatest" Port="9443" Protocol="https"

Property Name="keyring" Value="/opt/HTTP/Plugins/config/test-webserver/plugin-key.kdb"/

Property Name="stashfile" Value="/opt/HTTP/Plugins/config/test-webserver/plugin-key.sth"/

/Transport

/Server

/ServerCluster

This tags defines the location of the Key-Database for the secure connection between your IHS and your AppServer.

2. In administrative console of WebSphere go to Servers > WebServers > "your webserver" > Plugin-in properties

On this page all necessary entries should be done automatically. To re-copy the default plugin-in.key to your IHS press "Copy to Webserver key store directory".


3. Restart your IHS

###############################################################



To manually setup SSL between IHS and WebSphere first locate the plugin-key.kdb on your AppServer. Than copy over the plugin-key.kdb to the IHS into the specified location (get location form picture above). Then edit the plugin-cfg.xml of the IHS (see tag from step 1). Then restart your IHS and your WebSphere instance

Generate a (new) SSL Certificate for https [IBM HTTP Server]

Generate a (new) SSL Certificate for https [IBM HTTP Server]
[FOR HTTPS/SSL BETWEEN CLIENT AND WEBSERVER]
To generate a new CA-Signed SSL-Certificate for use with the IBM HTTP Server you need to start the iKeyman-Utility first. The iKeyman is the Key Management Tool from IBM.

1. Navigate to the /bin-directory of your IHS-Installation
2. execute
./ikeyman
to open the Key Management Tool
3. Use "Key Database File > Open" to open your password-protected Key-Database

4. After the Key-Database is loaded switch to "Personal Certificate Requests" (under "Key database content").
5. Click New and fill out the certificate request dialog. Depending on your CA-Provider (VeriSign,...) you may need to fill out the dialog in a special way (VeriSign demands the common name to be the domain)

6. Click "OK" to save the certificate request in a file.
7. now you need to provide the content of the certificate request file to your Ceritifcate Authority (e.g.: VeriSign). You will receiving a new certificate file from them.
8. If you received the certificate switch back to "Personal Certificates" (under "Key database content").
9. Click Receive and navigate to the certificate file. Click Ok to import the certificate file.

10. Open the httdp.conf-File of your IHS and replace the SSL-Cert-Name (new one will be displayed after the import of the new certificate in iKeyman). Usally the SSL-Cert is definded within a virtual host:
Example:

ServerName www.test.com
SSLEnable
SSLClientAuth 0
SSLServerCert ihssslcert
AllowEncodedSlashes On

Options Indexes MultiViews
Order allow,deny
Allow from all



DocumentRoot /usr/IBM/HTTPServer/www-doc-root/


11. Restart the IHS-Server (/bin/apachectl stop --> /bin/apachectl start)