Explore topic-wise InterviewSolutions in .

SLAPD and slurpd

slapd and slurpd

/var/VRTSvcs/log/engine_A.log14

/var/VRTSvcs/log/engine_A.log14

A split BRAIN occurs when two independent systems configured in a cluster assume they have exclusive ACCESS to resource. this scenario can be CAUSED when all cluster heartbeat links are SIMULTANEOUSLY lost. Each cluster node will then MARK the other cluster node as FAULTED. , usually resulting in data corruption.

A split brain occurs when two independent systems configured in a cluster assume they have exclusive access to resource. this scenario can be caused when all cluster heartbeat links are simultaneously lost. Each cluster node will then mark the other cluster node as FAULTED. , usually resulting in data corruption.

The STATE in which a NODE is missing ONE of the two required heartbeat connections. When a node is running with one heartbeat only (in jeopardy), VCS does not restart the applications on a new node. This action of disabling failover is a safety MECHANISM that PREVENTS data corruption.

The state in which a node is missing one of the two required heartbeat connections. When a node is running with one heartbeat only (in jeopardy), VCS does not restart the applications on a new node. This action of disabling failover is a safety mechanism that prevents data corruption.

haconf -makerw

hagrp -unfreeze <SERVICE Group> -PERSISTENT

haconf -dump -makero

haconf -makerw

hagrp -unfreeze <Service Group> -persistent

haconf -dump -makero

lsmod | GREP LLT

rmmod LTT

lsmod | grep llt

rmmod ltt

lltconfig -U

lltconfig -U

lsmod | GREP GAB

rmmod gab

lsmod | grep gab

rmmod gab

• gabconfig -U
• gabconfig -a

gabconfig -a

gabconfig -a

hastop -all -FORCE

hastop -all -force

HAGRP –online -SYS NODENAME

hagrp –online -sys nodename

HAGRP –switch -to NODENAME

hagrp –switch -to nodename

hagrp is used for doing administrative ACTIONS on service groups like online, OFFLINE, SWITCH ETC.

hagrp is used for doing administrative actions on service groups like online, offline, switch etc.

Switchover is an manual task where as FAILOVER is AUTOMATIC. You can switchover service group from online cluster node to offline cluster node in CASE of POWER outage, hardware failure, schedule shutdown and reboot. But the failover will failover the service group to the other node when VCS heartbeat link down, damaged, broken because of some disaster or SYSTEM hung.

Switchover is an manual task where as failover is automatic. You can switchover service group from online cluster node to offline cluster node in case of power outage, hardware failure, schedule shutdown and reboot. But the failover will failover the service group to the other node when VCS heartbeat link down, damaged, broken because of some disaster or system hung.

HALINK is USED to LINK the DEPENDENCIES of the RESOURCES.

halink is used to link the dependencies of the resources.

Service GROUP is made up of RESOURCES and their links which you NORMALLY requires to maintain the HA of APPLICATION.

Service group is made up of resources and their links which you normally requires to maintain the HA of application.

HARES –state

hares –state

hacf -VERIFY /etc/VRTSvcs/conf/config

hacf -verify /etc/VRTSvcs/conf/config

MAIN.cf is the main CONFIGURATION FILE for VCS and it is LOCATED in /etc/VRTSvcs/conf/config/.

main.cf is the main configuration file for VCS and it is located in /etc/VRTSvcs/conf/config/.

hastatus –sum

hastatus –sum

If the ManageFaults ATTRIBUTE of a service group is set to NONE, VCS does not take any automatic action when it DETECTS a resource fault. VCS places the resource into the ADMIN_WAIT STATE and waits for administrative intervention.

To clear the resource in ADMIN_WAIT state WITHOUT faulting service group :

# hares -probe [resource] -SYS node01

To clear the resource in ADMIN_WAIT state by changing the status to OFFLINE|FAULTED :

# hagrp -clearadminwait -fault [SG] -sys node01

If the ManageFaults attribute of a service group is set to NONE, VCS does not take any automatic action when it detects a resource fault. VCS places the resource into the ADMIN_WAIT state and waits for administrative intervention.

To clear the resource in ADMIN_WAIT state without faulting service group :

# hares -probe [resource] -sys node01

To clear the resource in ADMIN_WAIT state by changing the status to OFFLINE|FAULTED :

# hagrp -clearadminwait -fault [SG] -sys node01

To CLEAR a resource FAULT, we FIRST have to fix the underlying problem.

• For persistent resources : Do not do anything and wait for the next OfflineMonitorInterval (default – 300 SECONDS) for the resource to become online.
• For non-persistent resources : Clear the fault and probe the resource on node01 :

# hares -clear [resource_name] -sys node01
# hares -probe [resource_name] -sys node01

To clear a resource fault, we first have to fix the underlying problem.

# hares -clear [resource_name] -sys node01
# hares -probe [resource_name] -sys node01

FLUSHING of a service GROUP is required when, agents for the RESOURCES in the service group seems suspended waiting for resources to be taken online/offline. Flushing a service group clears any internal wait states and STOPS VCS from attempting to bring resources online.

To flush the service group SG on the cluster node, node01 :

# hagrp -flush [SG] -sys node01

Flushing of a service group is required when, agents for the resources in the service group seems suspended waiting for resources to be taken online/offline. Flushing a service group clears any internal wait states and stops VCS from attempting to bring resources online.

To flush the service group SG on the cluster node, node01 :

# hagrp -flush [SG] -sys node01

MULTINICA and IPMultiNIC:

• supports active/passive configuration.
• Requires only 1 BASE IP (test IP).
• Does not require to have all IPs in the same subnet.

MultiNICB and IPMultiNICB:

• supports active/active configuration.
• Faster FAILOVER than the MultiNICA.
• Requires IP address for each interface.
• TROUBLESHOOTING

MultiNICA and IPMultiNIC:

MultiNICB and IPMultiNICB:

SPLIT brain occurs when all the LLT links fails simultaneously. Here systems in the cluster fail to identify whether it is a SYSTEM failure or an interconnect failure. Each mini-cluster thus formed thinks that it is the only cluster thats ACTIVE at the moment and tries to start the service groups on the other mini-cluster which he think is down. Similar thing happens to the other mini-cluster and this may lead to a SIMULTANEOUS access to the storage and can cause data CORRUPTION.

Split brain occurs when all the LLT links fails simultaneously. Here systems in the cluster fail to identify whether it is a system failure or an interconnect failure. Each mini-cluster thus formed thinks that it is the only cluster thats active at the moment and tries to start the service groups on the other mini-cluster which he think is down. Similar thing happens to the other mini-cluster and this may lead to a simultaneous access to the storage and can cause data corruption.

To RECOVER from JEOPARDY, just fix the FAILED link(s) and GAB automatically DETECTS the new link(s) and the jeopardy membership is removed from NODE.

To recover from jeopardy, just fix the failed link(s) and GAB automatically detects the new link(s) and the jeopardy membership is removed from node.

When a node in the cluster has only the last LLT LINK INTACT, the node forms a regular membership with other nodes with which it has more than one LLT link ACTIVE and a JEOPARDY membership with the node with which it has only one LLT link active.

When a node in the cluster has only the last LLT link intact, the node forms a regular membership with other nodes with which it has more than one LLT link active and a Jeopardy membership with the node with which it has only one LLT link active.

When you freeze a service group, VCS continues to monitor the service group, but does not allow it or the resources under it to be TAKEN offline or brought online. Failover is also disable even when a resource faults. When you unfreeze the SG, it start BEHAVING in the normal way.

To freeze/unfreeze a Service Group temporarily :

# hagrp -freeze [service-group]
# hagrp -unfreeze [service-group]

To freeze/unfreeze a Service Group persistently (across REBOOTS) :

# hagrp -freeze -persistent[service-group]
# hagrp -unfreeze [service-group] -persistent 
Communication FAILURES : Jeopardy, split brain

When you freeze a service group, VCS continues to monitor the service group, but does not allow it or the resources under it to be taken offline or brought online. Failover is also disable even when a resource faults. When you unfreeze the SG, it start behaving in the normal way.

To freeze/unfreeze a Service Group temporarily :

# hagrp -freeze [service-group]
# hagrp -unfreeze [service-group]

To freeze/unfreeze a Service Group persistently (across reboots) :

# hagrp -freeze -persistent[service-group]
# hagrp -unfreeze [service-group] -persistent 
Communication failures : Jeopardy, split brain

The COMMAND to SWITCH the SERVICE group to target node :

# HAGRP -switch [service-group] -to [target-node]

The command to switch the service group to target node :

# hagrp -switch [service-group] -to [target-node]

To online/offline the service group on a particular node :

# hagrp -online [service-group] -sys [node] (Online the SG on a particular node)

# hagrp -offline [service-group] -sys [node] (Offline the SG on particular node)

The -any OPTION when used instead of the node name, brings the SG online/offline based on SG’s failover POLICY.

# hagrp -online [service-group] -any

# hagrp -offline [service-group] -any

To online/offline the service group on a particular node :

# hagrp -online [service-group] -sys [node] (Online the SG on a particular node)

# hagrp -offline [service-group] -sys [node] (Offline the SG on particular node)

The -any option when used instead of the node name, brings the SG online/offline based on SG’s failover policy.

# hagrp -online [service-group] -any

# hagrp -offline [service-group] -any

To SEE the service GROUP configuration :

# HAGRP -display SG

To see the service group configuration :

# hagrp -display SG

In GENERAL, to ADD a service GROUP NAMED SG with 2 nodes (node01 and node02) :

• haconf –makerw
• hagrp –add SG
• hagrp –modify SG SystemList node01 0 node02 1
• hagrp –modify SG AutoStartList node02
• haconf –dump -makero

In general, to add a service group named SG with 2 nodes (node01 and node02) :

To start HAD or VCS on all NODES in the CLUSTER, the hastart command need to be run on all nodes individually.

# hastart

To start HAD or VCS on all nodes in the cluster, the hastart command need to be run on all nodes individually.

# hastart

The GAB configuration file /etc/gabtab defines the minimum number of nodes that must be communicating for the cluster to start. This is called as GAB seeding.

In case we don’t have sufficient number of nodes to start VCS [ may be due to a maintenance ACTIVITY ], but have to do it anyways, then we have do what is called as MANUAL seeding by FIRING below command on each of the nodes.

# gabconfig -C -x

The GAB configuration file /etc/gabtab defines the minimum number of nodes that must be communicating for the cluster to start. This is called as GAB seeding.

In case we don’t have sufficient number of nodes to start VCS [ may be due to a maintenance activity ], but have to do it anyways, then we have do what is called as manual seeding by firing below command on each of the nodes.

# gabconfig -c -x

The commands to stop and start LLT are :

# lltconfig -c -&GT; start LLT
# lltconfig -U -> stop LLT (GAB NEEDS to stopped FIRST)

The commands to stop and start LLT are :

# lltconfig -c -> start LLT
# lltconfig -U -> stop LLT (GAB needs to stopped first)

The commands to start and STOP GAB are :

# gabconfig -C (start GAB)
# gabconfig -U (stop GAB)

The commands to start and stop GAB are :

# gabconfig -c (start GAB)
# gabconfig -U (stop GAB)

The file /etc/gabtab contains the COMMAND to start the GAB.

# cat /etc/gabtab
/sbin/gabconfig -c -n 4

here -n 4 –&GT; number of nodes that MUST be communicating in ORDER to start VCS.

The file /etc/gabtab contains the command to start the GAB.

# cat /etc/gabtab
/sbin/gabconfig -c -n 4

here -n 4 –> number of nodes that must be communicating in order to start VCS.

LLT uses /etc/llttab to set the configuration of the LLT interconnects.
# cat /etc/llttab
set-node node01
set-cluster 02
LINK nxge1 /dev/nxge1 - ether - -
link nxge2 /dev/nxge2 - ether - -
link-lowpri /dev/nxge0 – ether - -

Here, set-cluster -> unique cluster number ASSIGNED to the entire cluster [ can have a value ranging between 0 to (64k – 1) ]. It should be unique across the organization.

set-node -> a unique number assigned to each node in the cluster. Here the name node01 has a corresponding unique node number in the FILE /etc/llthosts. It can range from 0 to 31.

Another configuration file used by LLT is – /etc/llthosts. It has the cluster-wide unique node number and nodename as FOLLOWS:

# cat /etc/llthosts
0 node01
1 node02

LLT has an another optional configuration file : /etc/VRTSvcs/conf/sysname. It contains short names for VCS to refer. It can be used by VCS to remove the dependency on OS hostnames.

LLT uses /etc/llttab to set the configuration of the LLT interconnects.
# cat /etc/llttab
set-node node01
set-cluster 02
link nxge1 /dev/nxge1 - ether - -
link nxge2 /dev/nxge2 - ether - -
link-lowpri /dev/nxge0 – ether - -

Here, set-cluster -> unique cluster number assigned to the entire cluster [ can have a value ranging between 0 to (64k – 1) ]. It should be unique across the organization.

set-node -> a unique number assigned to each node in the cluster. Here the name node01 has a corresponding unique node number in the file /etc/llthosts. It can range from 0 to 31.

Another configuration file used by LLT is – /etc/llthosts. It has the cluster-wide unique node number and nodename as follows:

# cat /etc/llthosts
0 node01
1 node02

LLT has an another optional configuration file : /etc/VRTSvcs/conf/sysname. It contains short names for VCS to refer. It can be used by VCS to remove the dependency on OS hostnames.

The COMMAND to check DETAILED LLT STATUS is :

# lltstat -nvv

The command to check detailed LLT status is :

# lltstat -nvv

A cluster can have a MAXIMUM of 8 LLT LINKS including high and LOW priority LLT links.

A cluster can have a maximum of 8 LLT links including high and low priority LLT links.

To CHECK the STATUS of GAB PORTS on VARIOUS NODES :

# gabconfig -a

To check the status of GAB ports on various nodes :

# gabconfig -a

• a : gab driver
• b : I/O fencing (to ensure data integrity)
• d : ODM (ORACLE DISK Manager)
• F : CFS (Cluster File System)
• h : VCS (VERITAS Cluster Server: high availability daemon, HAD)
• o : VCSMM driver (kernel MODULE needed for Oracle and VCS INTERFACE)
• q : QuickLog daemon
• v : CVM (Cluster Volume Manager)
• w : vxconfigd (module for cvm)

GAB, LLT and HAD forms the basic building blocks of vcs functionality:

• LLT (low latency transport protocol) – LLT transmits the HEARTBEATS over the interconnects. It is also used to distribute the inter system communication traffic EQUALLY among all the interconnects.
• GAB (Group membership services and atomic broadcast) – The group membership service part of GAB maintains the overall cluster membership information by TRACKING the heartbeats sent over LLT interconnects. The atomic broadcast of cluster membership ensures that every node in the cluster has same information about every resource and service group in the cluster.
• HAD (High Availability daemon) – the main VCS engine which manages the AGENTS and service group. It is in turn monitored by a daemon named hashadow.

GAB, LLT and HAD forms the basic building blocks of vcs functionality:

• Save and close the configuration :
  # haconf -dump -makero
• Stop VCS on all NODES in the cluster :
  # hastop -all
• Edit the configuration file after taking the backup and do the CHANGES :
  # cp -p /etc/VRTSvcs/conf/config/main.cf /etc/VRTSvcs/conf/config/main.cf_17march
  # vi /etc/VRTSvcs/conf/config/main.cf
• Verify the configuration file syntax :
  # hacf -verify /etc/VRTSvcs/conf/config/
• start the VCS on the system with modified main.cf file :
  # hastart
•  start VCS on other nodes in the cluster.

Note : This can be DONE in another way by just stopping VCS and LEAVING services running to minimize the downtime. (hastop -all -force GAB, LLT and HAD)

Note : This can be done in another way by just stopping VCS and leaving services running to minimize the downtime. (hastop -all -force GAB, LLT and HAD)

• PERSISTENT : VCS can only monitor these resources but can not offline or online them.
• On-Off : VCS can start and stop On-Off resource type. Most resources fall in this category.
• On-Only : VCS starts On-Only resources but does not stop them. An example WOULD be NFS DAEMON. VCS can start the NFS daemon if required, but can not take it offline if the associated service GROUP is take offline.

To verify the SYNTAX of the main.cf FILE just mention the ABSOLUTE directory PATH to the main.cf file :

# hacf -verify /etc/VRTSvcs/conf/config

To verify the syntax of the main.cf file just mention the absolute directory path to the main.cf file :

# hacf -verify /etc/VRTSvcs/conf/config

To CHECK the status of the ENTIRE CLUSTER :

# hastatus -sum

To check the status of the entire cluster :

# hastatus -sum

The VCS CLUSTER ENGINE logs is located at /var/VRTSvcs/log/engine_A.log. We can either DIRECTLY view this file or USE COMMAND line to view it:

# hamsg engine_A

The VCS cluster engine logs is located at /var/VRTSvcs/log/engine_A.log. We can either directly view this file or use command line to view it:

# hamsg engine_A

To SET the configuration file in read-only/read-write :

• # haconf -DUMP -makero (Dumps in memory configuration to main.cf and MAKES it read-only)
• # haconf -makerw (Makes configuration writable)

To set the configuration file in read-only/read-write :