WebSphere Core Groups

What is a core group?

A core group encapsulates processes in a Network Deployment cell to create high availability domains.
A core group is a grouping of WebSphere Application Server cell processes. A core group can contain standalone servers, cluster members, node agents, and the deployment manager. A core group must contain at least one node agent or the deployment manager.
DefaultCoreGroup is a core group that is created by default at installation time and can be used out-of-the-box; that is, all processes will know about each other.
Note:
1. A core group cannot extend beyond a cell
2. All JVMs in a core group must able to communicate (they use heartbeat messages to know each other)

Core group coordinator

Once the core group stabilizes at runtime, one of the member will be elected to act as an coordinator. That member called as Coregroup coordinator is responsible for managing the high availability with in that core group.
1. It maintains all group information like group name, members and policy of the group
2. It maintains a record state of the group members as they start, stop or fail
3. Assigning singleton services to group members and handling failover based on policy specified.
We can change the default core group coordinator by going to:
servers –>coregroups->coregroup settings->Default Coregroup ->preferred coordinator servers.
When a member becomes active coordinator, you can see the following messages in the SystemOut:

[3/3/10 18:00:37:758 CET] 00000013 CoordinatorIm I HMGR0206I: The Coordinator is an Active Coordinator for core group DefaultCoreGroup.

If a member was failed/stopped in the core group:

[3/3/10 18:00:37:758 CET] 00000026 RoleMember    W   DCSV8104W: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: Removing member [Test-Cell\node02\server02] because the member was requested to be removed  by member Test-Cell\node02\server01. Internal details VL suspects others: CC-Situation Normal
[3/3/10 18:00:38:176 CET] 00000023 VSyncAlgo1    I   DCSV2004I: DCS Stack DefaultCoreGroup at Member Test-Cell\node01\server01: View synchronization completed successfully. The View Identifier is (22898:0.Test-Cell\node02\server01). The internal details are None.
[3/3/10 18:00:38:207 CET] 00000023 VSyncAlgo1    I   DCSV2004I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: View synchronization completed successfully. The View Identifier is (331:0.Test-Cell\node02\server01). The internal details are None.
[3/3/10 18:00:38:537 CET] 00000024 CoordinatorIm I   HMGR0218I: A new core group view has been installed. The core group is DefaultCoreGroup.
[3/3/10 18:00:39:228 CET] 00000026 DataStackMemb I   DCSV8050I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: New view installed, identifier (332:0.Test-Cell\node02\server01), view size is 11 (AV=11, CD=12, CN=12, DF=12)
[3/3/10 18:00:39:343 CET] 00000021 DRSBuddyManag A   CWWDR0006I:  Replication instance terminated : Test-Cell\node02\server02

If a new member joins the core group, you can see the following message

[3/3/10 18:17:13:245 CET] 00000026 RoleMember    I   DCSV8051I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: Core group membership set changed. Added: [Test-Cell\node02\server02].
[3/3/10 18:17:13:315 CET] 00000023 MbuRmmAdapter I   DCSV1032I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: Connected a defined member Test-Cell\node02\server02.
[3/3/10 18:17:30:337 CET] 00000023 VSyncAlgo1    I   DCSV2004I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: View synchronization completed successfully. The View Identifier is (333:0.Test-Cell\node02\server01). The internal details are None.
[3/3/10 18:17:30:353 CET] 00000026 DataStackMemb I   DCSV8050I: DCS Stack DefaultCoreGroup.TestRepln at Member Test-Cell\node01\server01: New view installed, identifier (334:0.Test-Cell\node02\server01), view size is 12 (AV=12, CD=12, CN=12, DF=12)

What happens when coordinator went down?
When the active coordinator is not available (stopped/crashed), the HA manager will elect the first inactive server in the preferred coordinator servers list. If preferred list is not specified, it will select lexically lowest named server.
The newly selected coordinator initiates a state rebuild by sending a message to all core group members to report their states.

Core group settings

1. Number of coordinators

Specifies the number of coordinators for this core group. The default value is one coordinator, although multiple coordinators are advisable for large core groups. All of the group data must fit in the memory of the allocated coordinators. One coordinator can run out of memory in a system with a large core group, which can cause the system to work improperly.

2. Transport type

Specifies the transport mechanism to use for communication between members of a core group.

Channel framework

Channel framework is the default transport type. It uses the channel framework service to incorporate port reusability and shared port technology into the communication system.

Unicast

Unicast is a targeted network model that focuses on a direct recipient for communication. This type of communication is most suitable when the intended message is sent to a specific set of recipients.

Multicast

Multicast consists of a broadcast network model. This model broadcasts communication across the defined network, depending upon the values that are provided for the multicast settings. Multicast settings are suitable when there are many recipients for the intended message; otherwise broadcast communication tends to overload the network with traffic, and can impact performance goals.

3. Channel chain name

Specifies the name of the channel chain if you select channel framework for the transport type.
If you select Multicast transport type

• Multicast port
  The port setting tells the coordinator where to scan for transmissions. When setting this value, verify that you are specifying a port that is not used by another network communication device. Setting a port value that has conflicts causes problems with your high availability manager infrastructure.
• Multicast group IP start
  Specify the starting Internet Protocol (IP) address of the intended communication area.
• Multicast group IP end
  Specify the ending IP address of the intended communication area. Plan the network to accommodate scalability.

4. Additional Properties
Core group servers
Specifies the server processes that belong to the core group. Server processes include the deployment manager, node agents, application servers, and cluster members. You can use the panel that displays to move server processes to a different core group.

Policies

Use to define the policies that determine which members of a high availability group are made active.

Preferred coordinator servers

Specifies which core group servers are preferred coordinator servers.

Core Group policies:

Servers > Core groups > Core group settings > New or existing core group > Policies.

Policy types

All active	The All active policy indicates that the high availability manager keeps all of the application components that are running on all of the servers in the high availability group active at all times
M of N	The M of N policy is similar to the One of N policy. However, it enables you to specify the number (M) of high availability group members that you want to keep active if it is possible to do so. The number of active members must be greater than one and less than or equal to the number of servers in the high availability group. If the number of active servers is set to one, this policy is a match for the One of N policy
No Operation	The No operation policy indicates that no high availability group members are made active
One of N	The One of N policy keeps one member of the high availability group active at all times. This is used by groups that desire singleton failover. If a failure occurs, the high availability manager starts the singleton on another server
Static	The Static policy allows you to statically define or configure the active members of the high availability group

Match Criteria
Specifies one or more name-value pairs that are used to associate this policy with a high availability group. These pairs must match attributes that are contained in the name of a high availability group before this policy is associated with that group.

Is alive timer
In seconds, the interval of time at which the high availability manager will check the health of the active group members that are governed by this policy. If a group member has failed, the server on which the group member resides is restarted.
Quorum
Specifies whether quorum checking is enabled for a group governed by this policy. Quorum is a mechanism that can be used to protect resources that are shared across members of the group in the event of a failure. The quorum mechanism is designed to work in conjunction with a hardware control facility that allows application servers to be shut down if a failure causes the group to be partitioned.
note: The Quorum setting in the policy will only have an effect if the following items are true:
* The group members are also cluster members.
* GroupName.WAS_CLUSTER=clustername must be specified as a property in the group name of any high availability group matching this policy.
Fail back
Specifies whether work items assigned to the failing server are moved to the server that is designated as the most preferred server for the group if a failure occurs. This field only applies for M of N and One of N policies.
Preferred servers only
Specifies whether group members are only activated on servers that are on the list of preferred servers for this group. This field only applies for M of N and One of N policies.

Core group servers:

 

Use this to move servers into a different core group. All members of a cluster must be in the same core group. If you select one or more members of a cluster, all of the members of that cluster must be moved.

Preferred coordinator servers:

 

 

Use Add and Remove to move servers into and out of the list of preferred servers. Use Move up and Move down to adjust the order within the list of preferred servers. Make sure that the most preferred server is at the top of the list and the least preferred server is at the bottom.

Core group member Failure detection

HA manager monitors all the core group members. It uses 2 settings to detect the failure
1. Active failure detection
If the heartbeat from a JVM is failing for specified interval of time, then it will be marked as failed. When using default settings, heartbeats are sent every 10sec and 20times (200sec) should be failed before marking the JVM as failed. When a JVM is marked as failed, a new view is installed and you can see that in the SystemOut log.
2. TCP Keep Alive
If one member is not able to contact other member, and if gets closed socket error, it will signal the other members to treat that member as failed. Say, if one jvm is panics or network issue etc, as soon as the TCP settings allow, the failure will be detected.
Note: TCP Keep alive setting is of the operating system.
About DCS and finding which core group member crashed/stopped. Here


############################################################################################

 

 

Application Server type

Application Server type

• A stand-alone application server is an application server created through the use of an application server profile and is not federated to a cell. This is the only option in the Base and Express environments. You can also create a stand-alone application server in the Network Deployment package. However, the expectation is that you will federate the application server to a cell for centralized management in the future.



• A managed application server is one that is managed from a deployment manager. This is only possible with the Network Deployment package. A managed server can either be an application server that was created using an application server profile and subsequently federated to the cell, or it can be created directly from the deployment manager’s administrative console

following options to stop an application server:

Stopping application server

From the administrative console, you have the following options to stop an
application server:

• Stop: quiesces the application server and stops it.


• Immediate Stop: stops the server, but bypasses the normal server quiesce process that supports in-flight requests to complete before shutting down the entire server process. This shutdown mode is faster than the normal server stop processing, but some application clients can receive exceptions.


• Terminate deletes the application server process. Use this if immediate stop
  fails to stop the server.

Restarting servers in cluster

Restarting servers in cluster

You can restart all the servers in a cluster through WAS Admin COnsole by selecting the server and clicking on ripple start

or you can execute the following wasadmin command


AdminControl.invoke('WebSphere:name=<clustername>,process=dmgr,platform=common,node=dmgrCellManager01,version=6.1.0.25,type=Cluster,mbeanIdentifier=cluster1,cell=dmgrCell01,spec=1.0', 'rippleStart')

Restarting all the servers on particular node

Restarting all the servers on particular node

You can restart all the servers on particular node through WAS admin console by going to System Administration

-< node agents. Selecting the node agent of your choice and clicking on Restart all Servers on Nodes like this

you can achieve same functionality using this wsadmin script

AdminControl.invoke('WebSphere:name=NodeAgent,process=nodeagent,platform=common,node=<nodeagentname>,diagnosticProvider=true,version=6.1.0.25,type=NodeAgent,mbeanIdentifier=NodeAgent,cell=<cellname>,spec=1.0', 'restart', '[false true]', '[java.lang.Boolean java.lang.Boolean]')

PMT

Profile Management Tool(PMT)

Profile Management tool is a GUI tool that you can use to create profile, You can launch it in following ways

• By going to was_home/bin/ProfileManagement/pmt.bat command


• In case of windows environment by going to Start - programs - IBM WebSphere - Application Server Network deployment v6.1 - profile management


• As a part of installation program select the launch Profile Management tool on the last page

Irrespective of how you launch the PMT tool this is the second screen that you get and rest of flow depends on type of profile that you want to create

What is node federation

What is node federation

In distributed environment there is one deployment manager and it is used to manage application servers on multiple machines. The way it works is you create a node agent on each of these machines and then the deployment manager makes use of this node agent to manage servers on different machines. It makes use of node agent for making configuration changes, installing file as well as getting PMI data.

The federation process is used for creating node agent on a machine. There are multiple ways in which you can create node agent

• Using the addNode command


• You can federate custom profile as part of profile creation wizard

Irrespecitive of how you federate the node, WAS will first create a node agent on target machine and then start it.

What addNode does

What addNode does

The addNode command adds a new node to an existing administrative cell.
The actions the command performs are:

1. Connects to the deployment manager process. This is necessary for the file transfers performed to and from the deployment manager in order to add the node to the cell.


2. Attempts to stop all running application servers on the node.


3. Backs up the current stand-alone node configuration to the


4. <profile_home>/config/backup/base/ directory


5. Copies the stand-alone node configuration to a new cell structure that
   matches the deployment manager structure at the cell level.


6. Creates a new local config directory and definition (server.xml) for the node
   agent.


7. Creates entries (directories and files) in the master repository for the new
   node’s managed servers, node agent, and application servers.


8. Uses the FileTransfer service to copy files from the new node to the master
   repository.


9. Uploads applications to the cell only if the -includeapps option is specified


10. Performs the first file synchronization for the new node. This pulls everything
    down from the cell to the new node.


11. Fixes the node’s setupCmdLine and wsadmin scripts to reflect the new cell
    environment settings.


12. Launches the node agent.

What removeNode does

What removeNode does

You can remove a Node using either the WAS Admin Console or using command line .
The command performs the following operations:

1. Connects to the deployment manager process to read the configuration data.


2. Stops all of the running server processes of the node, including the node agent process.


3. Removes servers in the node from clusters
   
   .


4. Restores the original stand-alone node configuration. This original configuration was backed up when the node was originally added to the cell. When a node is removed, it is restored to its original configuration,except when it was added to the cell.


5. Removes the node’s configuration from the master repository of the cell. The local copy of the repository held on each node will get updated at the next synchronization point for each node agent. Although the complete set of configuration files are not pushed out to other nodes, some directories and files are pushed out to all nodes.


6. Removes installed applications from application servers in the cell that are part of the node being removed.


7. Copies the original application server cell configuration into the active configuration.

Granting Access to the WebSphere Application Server 6.1 Administration Console

Granting Access to the WebSphere Application Server 6.1 Administration Console


how to get Access to the WebSphere Application Server 6.1 Administration Console. To Grant Access to the WebSphere Application Server 6.1 Administration Console


1. From the WebSphere Administrator console, expand the Users and Groups node.

2. Click “Administrative Group Roles”. A new page opens.

3. Click the Add button. A new page opens.

a. In the field “Group name” field, enter wasadminrole, which was defined earlier.

b. In the “Role(s)” field, select “Administrator”.

c. Click OK and return to the “Administrative Group Roles” page.

4. Click “Save directly to the master configuration”.

Steps involved in moving WebSphere dmgr from one host (machine1) to another host (machine2) with the hostname change

1) As a caution make please make a backup of the following so that it's easy to restore when something goes wrong,


a) Backup the configuration of all the profiles (DMGR) and (AppSrv) in machine1 that is involved in the cell .

(eg) basically run WAS_ROOT/bin/backupconfig.sh which will create WebSphereConfig_2007-11-16.zip

b) (Optional) Also do a filesystem backup of the directory if possible to avoid any surprises.

2) Install WebSphere ND in the new box and create a new Dmgr profile with machine2_dmgr_profile .


3) Extract the WebSphereConfig_2007-11-16.zip to /config directory.

(eg) jar -xvf WebSphereConfig_2007-11-16.zip


4) if is different than than change USER_INSTALL_ROOT "value" in /config/cells//nodes//variables.xml to point to the new dmgr profile location.

(eg)

5) Change the following properties in /bin/setupcmdLine.sh to point to the machine1 dmgr cell name and node name.

(eg) WAS_CELL=machine1Cell01

WAS_NODE=machine1CellManager01

6) Copy the custom keyfiles (*.jks) from /etc/ to /etc; or skip this step if dmgr is using the default keys.

7) Follow the instructions mentioned in http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&q1=best+practices&uid=swg27007419&loc=en_US&cs=utf-8&lang=en from page 4-6 on section 2 and 2.1


8) check if syncs works for all the nodes and you were able to see all the configurations from the previous dmgr.

Administrative user roles

Administrative user roles

The WAS Administration console application uses the J2EE role-based authorization concept. It has following roles

•Monitor: Least privileged. Allows a user to view the WebSphere

configuration and current application server state.A monitor can complete the following tasks:

◦View the WebSphere Application Server configuration.

◦View the current state of the Application Server.

•Configurator: An individual or group that uses the configurator role has the monitor privilege plus the ability to change the WebSphere Application Server configuration. The configurator can perform all the day-to-day configuration tasks. For example, a configurator can complete the following tasks:

◦Create a resource.
◦Map an application server
◦Install and uninstall an application.
◦Deploy an application.
◦Assign users and groups-to-role mapping for applications.
◦Set up Java 2 security permissions for applications.
◦Customize the Common Secure Interoperability Version 2 (CSIv2), Secure Authentication Service (SAS), and Secure Sockets Layer (SSL) configurations..

•Operator: Monitor privilege in addition to the ability to change runtime state, such as starting or stopping server, also check server status.

•Deployer: Only available for wsadmin users (not administration console). Allows a user to change configuration and runtime state on applications using wsadmin.

•Admin Security Manager: Allows a user to map users and groups to administrative roles through the administrative console, or through wsadmin for fine-grained security. Also, when fine grained administrative security is used, users granted this role can manage authorization groups

•iscadmins: Only available for administration console users. Allows a user to manage users and groups in the Federated repositories.

•Administrator: Operator, configurator, and iscadmins privilege, in addition to additional privileges granted solely to the administrator role, such as:

1.Modifying the primary administrative user and password

2.Create, update, and delete users and groups

3.Enabling or disabling administrative and Java 2 security

The primary administrative user specified when enabling administrative security is automatically mapped to the Administrator and AdminSecurityManager roles. Therefore, it is not necessary to manually add this identity to either of these administrative roles.