Steps involved in moving WebSphere dmgr from one host (machine1) to another host (machine2) with the hostname change

1) As a caution make please make a backup of the following so that it's easy to restore when something goes wrong,


a) Backup the configuration of all the profiles (DMGR) and (AppSrv) in machine1 that is involved in the cell .

(eg) basically run WAS_ROOT/bin/backupconfig.sh which will create WebSphereConfig_2007-11-16.zip

b) (Optional) Also do a filesystem backup of the directory if possible to avoid any surprises.

2) Install WebSphere ND in the new box and create a new Dmgr profile with machine2_dmgr_profile .


3) Extract the WebSphereConfig_2007-11-16.zip to /config directory.

(eg) jar -xvf WebSphereConfig_2007-11-16.zip


4) if is different than than change USER_INSTALL_ROOT "value" in /config/cells//nodes//variables.xml to point to the new dmgr profile location.

(eg)

5) Change the following properties in /bin/setupcmdLine.sh to point to the machine1 dmgr cell name and node name.

(eg) WAS_CELL=machine1Cell01

WAS_NODE=machine1CellManager01

6) Copy the custom keyfiles (*.jks) from /etc/ to /etc; or skip this step if dmgr is using the default keys.

7) Follow the instructions mentioned in http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&q1=best+practices&uid=swg27007419&loc=en_US&cs=utf-8&lang=en from page 4-6 on section 2 and 2.1


8) check if syncs works for all the nodes and you were able to see all the configurations from the previous dmgr.

SSL Certificates expiration monitoring for WebSphere or any java based application server using java keystore

If you are a WebSphere Administrator or any Application Server (WebLogic, Tomcat) administrator you might already know that managing the ssl certificates in a large complex environments becomes hectic and troublesome because of the different expiration dates of the certificates that websphere uses and also the ssl certificates of the external systems (like SAP , Siebel) that websphere applicaton server might connect to using a secure connection, multiple administrators in an organization renewing it and not keeping tracking of the expiration dates. The problem is , SSL certificate might expire resulting in servers will not initialize and your running servers will stop operating, becomes unresposive if not properly renewed on time. Hence this article will explain how you can monitor the expiration of the ssl certificates using a simple command and proactively monitor the expiration dates, setup your calendar and renew it on time to prevent any downtimes.


If you are using the latest version (as on Dec 2007) of WebSphere Application Server 6.1 and using default self signed there are provisions in the application server itself to effectively monitor the certificate expiration, notifying you and renewing it automatically. If you are using a perosnal self signed certificate or a personal certificate signed by a certificate authority (CA) like VeriSign or Thawte , which is most likely in all of your production environments then you might want to use the below command to find the expiration date and renew it.

If you are using a java keystore ( DummyServerKeyFile.jks , DummyServerTrustFile.jks , jssecerts , cacerts) use the below find command along with keytool or websphere's ikeyman command line option to find the expiration of the keys.

LINUX:


Using Ikeyman cmdline utility (ikeyman.sh):

find PATH -name *.jks xargs -i bash -c '$WAS_HOME/java/bin/java -classpath $WAS_HOME/java/jre/lib/ext/ibmjceprovider.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmjcefw.jar:$WAS_HOME/AppServer/java/jre/lib/ext/US_export_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/local_policy.jar:$WAS_HOME/AppServer/java/jre/lib/ext/ibmpkcs.jar:$WAS_HOME/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw
grep "Not After:.*yyy[y,y]" -B 3 && echo {}'

(e.g)


find /usr/IBM/WebSphere/AppServer/profiles/ -name *.jks xargs -i bash -c '/usr/IBM/WebSphere/AppServer/java/bin/java -classpath /usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/AppServer//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/AppServer/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type jks -pw WebAS grep "Not After:.*200[7,8]" -B 3 && echo {}'


Using Java Keytool utility:


find $WAS_HOME -name '*.jks' xargs -i bash -c "echo {}; $WAS_HOME/java/jre/bin/keytool -list -v -keystore {} -storepass password grep 'until: .*/.*/Y[Y,Y]' -A 3 -B 7"

(e.g)

find /usr/IBM/WebSphere/AppServer -name '*.jks' xargs -i bash -c "echo{};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS grep 'until: .*/.*/0[7,8]' -A 3 -B 7"

Output:


/usr/IBM/WebSphere/AppServer/profiles/etc//WWWStageClientKey.jksAlias name: verisign class 1 ca individual subscriber-persona not validatedCreation date: Dec 18, 2006Entry type: trustedCertEntryOwner: CN=VeriSign Class 1 CA Individual Subscriber-Persona Not Validated, OU="www.verisign.com/repository/RPA Incorp. By Ref.,LIAB.LTD(c)98", OU=VeriSign Trust Network, O="VeriSign, Inc."Issuer: OU=Class 1 Public Primary Certification Authority, O="VeriSign, Inc.", C=USSerial number: d8b4feeaad218df5bf4756a9d29e17ffbValid from: 5/12/98 12:00 AM until: 5/12/08 11:59 PMCertificate fingerprints: MD5: CA:66:3C:FC:71:2B:BA:41:92:71:DD:72:AD:E5:65:65 SHA1: 12:51:9A:E9:CD:77:7A:56:01:84:F1:FB:D5:42:15:22:2E:95:E7:1F--Creation date: Dec 18, 2006Entry type: keyEntryCertificate chain length: 1Certificate[1]:Owner: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=USIssuer: CN=wadm.stage.WWW.com, OU=IT, O=WWW, L=Fremont, ST=CA, POSTALCODE=94089, C=USSerial number: 458709efValid from: 12/18/06 9:36 PM until: 12/18/07 9:36 PMCertificate fingerprints: MD5: C2:4F:CC:SS:19:DC:E4:88:B0:2A:78:76:69:4D:DC:EF:47 SHA1: 53:AB:D0:13:45:45:12:59:64:F6:C9:38:41:F2:C8:E3:37:05:73:95:F3:83/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jksAlias name: www stage websphere server caCreation date: Dec 8, 2006Entry type: keyEntryCertificate chain length: 1Certificate[1]:Owner: CN=wadm.stage.www.com, OU=IT, O="www, Inc.", L=Fremont , ST=California, C=USIssuer: OU=Secure Server Certification Authority, O="RSA Data Security, Inc.", C=USSerial number: 1ea5f21b33ca65e551440d662399dffdfga382Valid from: 12/5/06 12:00 AM until: 12/15/07 11:59 PMCertificate fingerprints: MD5: C2:22:E4:12:12:6D:3F:22:67:4E:37:48:12:3F:F3:1B:FE:26 SHA1: CA:81:37:A7:E9:12:22:19:27:D9:3F:C5:79:E9:25:C8:1A:3C:1E:5C:17:02

As you can see in the above output the command will list the file name (/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks) , certificate label (www stage websphere server ca) , Entry Type (keyEntry - personal cert or trustedCertEntry - Signer Cert), and the expiration date (12/5/06 12:00 AM until: 12/15/07 11:59 PM) of


all the java keystore (*.jks) in the given directory that expired in 2007 (07) or 2008 (08). By which you shoule be able to indentify the certs, setup your calendat atleast 10 days before the expiration and renew it to proactively to preveny any problems.

Note you can also change the above command by passing differnt filename like cacerts and jssecerts and also pass different year infomration to find expirations during those years.

Sometimes you might get no output which means either there are no expiring ceritificates for the years that you passed or if you are expecting any expiring certificate to show and if not then the password for the keystore might be wrong, so run without the grep portion in the above command which will list all the certificates irrespective of the expiration date or it will complain if the password is wrong as below.


find /usr/IBM/WebSphere/AppServer -name '*.jks' xargs -i bash -c "echo {};/usr/IBM/WebSphere/AppServer/java/jre/bin/keytool -list -v -keystore {} -storepass WebAS



/usr/IBM/WebSphere/AppServer/profiles/etc/wwwStageServerKey.jks

java.io.IOException: Keystore was tampered with, or password was incorrect

Also you can use the same command by modifying the -type parameter to "-type cms" for the webserver such as IBM HttpServer where it's using cms database ( kdb file).



find /usr/IBM/HttpServer -name *.kdb xargs -i bash -c '/usr/IBM/WebSphere/Plugins/java/bin/java -classpath /usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjceprovider.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmjcefw.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/US_export_policy.jar:/usr/IBM/WebSphere/Plugins//java/jre/lib/ext/local_policy.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext/ibmpkcs.jar:/usr/IBM/WebSphere/Plugins/java/jre/lib/ext com.ibm.gsk.ikeyman.ikeycmd -cert -list all -expiry -db {} -type cms -pw WebAS grep "Not After:.*200[7,8]" -B 3 && echo {}'

Useful UNIX commands for WebSphere Application server...

These are some useful commands that i came across when administrating WebSphere Application Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.


Performance related Commands to monitor system performance


-------------------------------------------------------------------------

1) Find the process that uses most CPU

ps -eo pcpu,pid,user,args
sort -k 1 -r
head -2

(eg) output of the above command when websphere server process is utilizing the most of the CPU


%CPU     PID    USER       COMMAND
71.7       31237   root          /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=54859 -classpath
 /cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

Java Class finder with a UNIX command

Java Class finder with a UNIX command

Being a Java developer or Java based Application server administrator like WebSphere or WebLogic , i am sure you should have come across this exception " Error: java.lang.ClassNotFoundException " which basically implies that the JVM is unable to find a definition for your class. It is usually resolved by adding the jar file that contains the class to the classpath or packaging the jar file along with your EAR or WAR file or placing the jar file in the right locations so that your Application Server classloader can find and load the class . So inorder to indentify the .jar file where the class file is present , the following simple UNIX command will help us to indentify the jar file and take appropiate actions to resolve the issue .


LINUX:

find PATH -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f  xargs -i bash -c "jar -tvf {} tr / . grep CLASSNAME && echo {}"

(e.g)

find /usr/IBM/WebSphere/AppServer/ -name '*.jar' -o -name '*.war' -o -name '*.ear' -type f  xargs -i bash -c "jar -tvf {} tr / . grep org.apache.xerces.dom.DocumentImpl.class && echo {}"

output:

11160 Fri Feb 20 14:00:34 PST 2004org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/archive_lib/xerces-2.6.2.jar
11160 Fri Oct 06 10:12:24 PDT 2006 org.apache.xerces.dom.DocumentImpl.class
/usr/IBM/WebSphere/AppServer/updateinstaller/update.jar

The above command will recursively search all the jar , EAR and WAR files under the /usr/IBM/WebSphere/AppServer/ for the class name org.apache.xerces.dom.DocumentImpl.class and output the location of the .jar file that contains the class.

Although this command is useful only in unix environment but there are other java based utilities to find the same like Jar Class Finder , Java class finder, etc , but i find this command simple to use without having to copy addtional jar files to your systems.