HPUX :HP Serviceguard Software - Procedure to Update Cluster When Changing LAN IP

HP Serviceguard Software - Procedure to Update Cluster When Changing LAN IP Addresses

Information

What is the procedure to update Serviceguard with new IP addresses?

Details

Serviceguard identifies IP addresses in the cluster configuration ASCII file, possibly one or more package configuration ASCII files and each legacy package control scripts where a relocatable IP is assigned. The cluster binary file distributed to all nodes is loaded with IP addresses identified in the configuration files when cmapplyconf completes successfully.

The cluster configuration file references fixed IPs, and the package configuration file references relocatable IPs that may be co-assigned to the same NICs as the fixed IPs. When changing an IP or set of IPs, determine which configuration file(s) must be updated and applied using cmapplyconf . Package control script modifications do not require cmapplyconf .

Note that cmapplyconf will validate the assignment of the new IPs to a NIC on each node, so it must be assigned to a NIC before cmapplyconf will succeed.

Every IP address defined in the cluster ASCII file that was used to create and distribute the cluster binary file (via cmapplyconf ) is stored in the cluster binary file (/etc/cmcluster/cmclconfig ). It is necessary to update the cluster binary when its IP addresses are changed.

When a cluster is created, each subnet has at least one fixed IP assigned to each node. Example content of a cluster configuration file:

NODE_NAME      eon   NETWORK_INTERFACE lan0     HEARTBEAT_IP    16.113.9.254 ---snipped--- NODE_NAME      ion   NETWORK_INTERFACE lan0     HEARTBEAT_IP    16.113.9.255

In this case, subnet 16.113.0.0 has an IP representative on eon (16.113.9.254 ) and a representative on ion (16.113.9.255 ). This document refers to this relationship as an IP set (*).

Changing IP addresses on a LAN NIC on the fly can interrupt transmission.

Although Serviceguard A.11.18-A.11.20 permits online IP reconfiguration, it is necessary to halt the cluster and perform IP changes for older versions of Serviceguard.

Unless the cluster operates as a cross-subnet cluster, changing one subnet/IP usually means changing the IP set*. Therefore, when changing IPs, expect to edit each node section in the cluster configuration ASCII file.

Also, changing a fixed IP may also require a change to a relocatable IP. Any package that relies on a relocatable IP related to a changed fixed IP must be updated with the new relocatable IP after the package is halted.

Updating a single subset of IPs across nodes:

Beginning with Serviceguard version A.11.19, it is possible to update an IP set while the cluster is running. (See limitations identified in the "Managing Serviceguard" manual.) Edit the cluster ASCII configuration file with the changes and perform cmapplyconf on the file. Note that any package that has a dependency on the IP/subnet must be modified at the same time, and the package cannot be running when the cmapplyconf is performed.

Beginning with Serviceguard version A.11.18, it is possible to remove an IP set (via editing the cluster ASCII file, and then cmapplyconf) and subsequently add an updated IP set while the cluster is running, assuming related relocatable IPs are not in use (packages are halted).

Earlier versions of Serviceguard require the cluster be halted if the cluster binary will be modified as a result of the change. The process requires 2 cmapplyconf commands. The first to remove the OLD NIC/IP set, and the 2nd to re-add the NETWORK_INTERFACE with its new IP.

If using a Serviceguard version earlier than A.11.18, determine whether downtime is necessary by using the following sort of command:

# cmviewconf | grep 10.X.X.17 (OLD IP)         ip address:                     10.X.X.17

If the ip address: line appears, then the IP is embedded in the cluster binary, requiring the cluster binary be updated with the new IP. If the subnet changes along with the IP, packages configuration files and control scripts that use that subnet must be updated as well.

Before proceeding, ensure /etc/cmcluster/cmclnodelist exists on each node, and lists every node in the cluster, much like a .rhosts file. Example:

nodeA   root nodeB   root

1. Edit the cluster configuration file replacing the old IP with the new IP.
2.   # cd /etc/cmcluster    # cmgetconf CLUSTER
3. Edit the file, locate the lines containing the old IP(s) and update the IPs.
4. The lines will look something like this:
5.   NODE1      NETWORK_INTERFACE    lan0        HEARTBEAT_IP        10.X.X.17      NETWORK_INTERFACE    lan1      NETWORK_INTERFACE    lan2        HEARTBEAT_IP        192.X.X.1  
6. Do the same for the other node's section.
7. If the subnet for that LAN adapter changed, it is necessary to determine if package-related subnet parameters must be updated as well.
8. # cmviewconf | grep -e "package name" -e "package subnet"       package name:                     clk       package subnet:                   10.X.X.0  <--- HERE    A.11.20: cmviewconf no longer exists.     Use 'cmviewcl -v -f line | grep package | grep subnet' to identify whether    packages have a dependency on subnets.   If the subnet will be modified, locate the package configuration file that   references the subnet.  In /etc/cmcluster:   # grep 10.X.X.0 */*   PKG_1/pkg.config:SUBNET                  10.X.X.0   PKG_1/pkg.ctrl:SUBNET[0]="10.X.X.0"  
9. In the above case, only one package was affected and its files will have to be updated to reflect the new subnet. It may also require replacing old relocatable IPs with new ones and registering them on the name server.
10. After updating the affected files, apply the updates to the cluster binary:
11. # cmapplyconf -C CONF [ -P  ]
12. When completed successfully, start the new cluster:
13. # cmruncl
14. Finally, test the package startup and shutdown on each server to be confident the change was performed properly.

Procedure to update multiple/all IP addresses in a cluster:

On one node:

1. Use cmgetconf to create an up-to-date cluster configuration ASCII file and package configuration ASCII files.
2. Verify the current configuration is viable:
3. # cmcheckconf -C -P -P ...
4. Halt the cluster: cmhaltcl .
5. In the event that a new cluster must be built, decluster any VG used by the cluster:
6. # vgchange -c n (repeat for each clustered VG).
7. On each cluster node:
8. Update /etc/cmcluster/cmclnodelist to identify each node.
9. Edit /etc/rc.config.d/cmcluster , set AUTOSTART_CMCLD=0
10. mv /etc/cmcluster/cmclconfig /etc/cmcluster/cmclconfig.old
11. Update /etc/rc.config.d/netconf with the new IP/subnet mask for each NIC that will change.
12. Edit/update /etc/hosts with the new IP addresses.
13. On one node:
14. Edit/update the cluster ASCII file with the new IP addresses assigned to each node.
15. Edit/update any package configuration file that identifies a relocatable IP or SUBNET value with new values.
16. Edit/update any package control script (legacy format) that identifies an old relocatable IP/SUBNET value with new values.
17. Reboot both systems.
18. On one node:
19. cmapplyconf the cluster and package configuration ASCII files (using the
20. custom command/script).
21. Start the cluster: cmruncl .
22. On each cluster node:
23. Edit /etc/rc.config.d/cmcluster , set AUTOSTART_CMCLD=1 .

Changing a relocatable IP:

Altering a relocatable IP is a simple process but an IP must be selected that matches the new subnet of the LAN card it will be configured on.

Find and edit the legacy package control script or modular package configuration file under /etc/cmcluster that references the old relocatable IP.

An easy way to locate the script is to "grep" the last few digits of the IP in all files beneath /etc/cmcluster . Use cmviewcl -v -f line | grep to identify the package to update.

With the package down (cmhaltpkg ), edit the file with the replacement IP. When saved, copy this script to the other servers which also operate the package when necessary.

NOTE:

Online alteration of the relocatable IP is not advisable. Also, the relocatable IP must be propagated to the name server, whether /etc/hosts on each server, or DNS.

Again, test the package startup and shutdown on each server to be confident the change was performed properly.