Veritas Cluster Server 6.

0 for UNIX:
Install and Configure
Lesson 13: Cluster Communications

Lesson introduction

Lesson 1: High Availability Concepts

Lesson 2: VCS Building Blocks
Lesson 3: Preparing a Site for VCS
Lesson 4: Installing VCS
Lesson 5: VCS Operations
Lesson 6: VCS Configuration Methods
Lesson 7: Preparing Services for VCS
Lesson 8: Online Configuration
Lesson 9: Offline Configuration
Lesson 10: Configuring Notification
Lesson 11: Handling Resource Faults
Lesson 12: Intelligent Monitoring Framework
Lesson 13: Cluster Communications
Lesson 14: Protecting Data Using SCSI 3-Based Fencing
Lesson 15: Coordination Point Server

Lesson objectives
Topic

Objectives

VCS communications Describe how components communicate in

review
a VCS environment.
Cluster membership

Describe how VCS determines cluster

membership.

Cluster interconnect
configuration

Describe the files that specify the cluster

interconnect configuration.

Joining the cluster

membership

Describe how systems join the cluster

membership.

Changing the
interconnect
configuration

Change the cluster interconnect

configuration.

VCS communications
review
After completing this topic, you
will be able to describe how
components communicate in a
VCS environment.

On-node and off-node communication

Agents

Agents

Agents

HAD

HAD

HAD

GAB

GAB

GAB

LLT

LLT

LLT

Each LLT
module tracks
the status of
heartbeats from
each peer on
each interface.

LLT forwards the

heartbeat status
of each node to
GAB.

LLT sends a
heartbeat on
each interface
every second.

Cluster interconnect specifications

Link limit
Up to eight links per node
High priority
Heartbeats every half-second
Cluster status information carried over links
Usually configured for dedicated cluster
network links
Low priority
Heartbeats every second
No cluster status sent
Automatically promoted to high priority if
there are no high-priority links functioning
Can be configured on public network
interfaces

Cluster membership
After completing this topic, you
will be able to describe how VCS
determines cluster membership.

GAB status and membership notation

20s placeholder

Cluster with four nodes: 0, 1, 21, 22 Nodes 0 and 1

# gabconfig -a
GAB Port Memberships
===============================================
Port a gen f7c001
Port b gen f7c004
Port h gen f7c002

HAD is
communicating
Fencing
is communicating
GAB is
communicating
Cluster with 22 nodes

membership 01
membership 01
membership 01

;
;
;

;12
;12
;12

10s placeholder
(0 displayed if node 10
is a member)
Nodes 21 and 22

Port a gen a4e095 membership 0123456789012345678901

Port b gen a4e098 membership 0123456789012345678901
Port h gen a4e096 membership 0123456789012345678901

LLT link status

# lltconfig
llt is running

lltconfig: Status of LLT protocol

lltstat n[vv] [active | configured]
-nvv: Status of nodes, with very verbose output
active: Status of only active nodes
configured: Status of all nodes
s1# lltstat nvv active
LLT node information:
Node
* 0 s1

1 s2

State

Link

Status

Address

OPEN
dev1

UP

00:0C:29:97:FB:5D

dev2

UP

00:0C:29:97:FB:67

dev1

UP

00:0C:29:C4:1D:0A

dev2

UP

00:0C:29:C4:1D:14

OPEN

* Shows which system runs the command

Cluster interconnect
configuration
After completing this topic, you
will be able to describe the files
that specify the cluster
interconnect configuration.

The llttab file

Assigns node numbers to systems
Sets the cluster ID number to identify a system with a cluster
Specifies the network devices used for the cluster interconnect
Modifies default LLT behavior, such as heartbeat frequency

/etc/llttab

set-cluster
10
set-node
s1
link nxge0 /dev/nxge:0 - ether - link nxge4 /dev/nxge:4 - ether - -

The llttab file on AIX

Range (all)

SAP

# cat /etc/llttab
set-node
S1
set-cluster
10
link en1 /dev/en:1 - ether - link en2 /dev/en:2 - ether - Tag Name

Device:Unit Link Type

MTU

The llttab file on HP-UX

Range (all)

SAP

# cat /etc/llttab
set-node
S1
set-cluster
10
link lan1 /dev/lan:1 - ether - link lan2 /dev/lan:2 - ether - Tag Name

Device:Unit

Link Type

MTU

The llttab file on Linux

SAP

# cat /etc/llttab
set-node
S1
set-cluster
10
link eth1 eth1 ether -
link eth2 eth-00:05:56:3f:02:2f ether 0xCAFD
Tag Name

Device

Link Type

Range (all)

MTU

The llthosts file

Associates a system name with a VCS cluster
node ID
Has the same entries on all systems
Maps unique node numbers to system
names
Matches system names with llttab and
main.cf
Matches system names with sysname

/etc/llthosts

0 s1
1 s2

How node and cluster numbers are

specified
/etc/llttab
set-node s1
0 to 64k -1

set-cluster 10
link qfe0 /dev/qfe:0 - ether - link qfe4 /dev/qfe:4 - ether - -

/etc/llthosts
0 to 63

0 s1
1 s2

The sysname file

Contains a short-form system name used by VCS
Removes VCS dependency on UNIX uname
Is optional, but created by default during cluster
configuration using CPI
Can be specified in llttab so the same file can be copied
to multiple nodes
/etc/VRTSvcs/conf/sysname

s1
s1
s1
s1
s1
s1 s1

s1
s1
s1
s1
s1

s1
s1
s1
s1
s1
s1 s12

s1
s1
s1
s1
s12

set-node /etc/VRTSvcs/conf/sysname
set-cluster 10
link dev1 /dev/dev:1 - ether - link dev2 /dev/dev:2 - ether - -

The GAB configuration file

The /etc/gabtab file:
Contains the command to start GAB:
/sbin/gabconfig -c -n number_of_systems
Specifies the number of systems that must be
communicating to allow VCS to start.

/etc/gabtab

/sbin/gabconfig c n 4

Cluster startup
After completing this topic, you
will be able to describe how
systems join the cluster
membership and services come
online.

Seeding during startup

I am
alive

HAD

2 I am
1 alive

HAD

I am
alive

HAD

GAB

GAB

GAB

LLT

LLT

LLT

s1

s2

s3

LLT starts on each system.

GAB starts on each system with a seed value equal to the number of
systems in the cluster: gabconfig c n 3
When GAB sees three members, the cluster is seeded.

HAD starts only after GAB is communicating on all systems.

How LLT and GAB are started

automatically on AIX
/etc/rc.d/rc2.d

S70llt

/etc/llttab?

sbin/lltconfig -c

S92gab

/etc/gabtab?

sh /etc/gabtab
/sbin/gabconfig c n #

S99vcs

hastart

How LLT and GAB are started

automatically on HP-UX
/sbin/rc2.d/S680llt /sbin/rc2.d/S920gab /sbin/rc2.d/S990vcs

/etc/llttab?

sbin/lltconfig -c

/etc/gabtab?

sh /etc/gabtab
/sbin/gabconfig c n #

hastart

How LLT and GAB are started

automatically on Linux
/etc/rc[2345].d

/etc/rc3.d/S66llt

/etc/llttab?

sbin/lltconfig -c

/etc/rc3.d/S67gab

/etc/rc3.d/S99vcs

/etc/gabtab?

sh /etc/gabtab
/sbin/gabconfig c n #

hastart

How LLT and GAB are started

automatically on Solaris 10
/lib/svc/method/llt

/etc/llttab?

/sbin/lltconfig -c

/lib/svc/method/gab

/etc/gabtab?

sh /etc/gabtab
/sbin/gabconfig c n #

/lib/svc/method/vcs

hastart

Probing resources during normal startup

1

A, B autodisabled for s1, s2

B
2

Monitor
3

HAD

HAD

During startup, HAD autodisables service groups.

HAD directs agents to probe (monitor) all resources on all systems

in the SystemList to determine their status.

If agents successfully probe resources, HAD brings service groups online

according to AutoStart and AutoStartList attributes.

System and cluster

interconnect failures
After completing this topic, you
will be able to describe how VCS
responds to common failures.

VCS response to system failure

A

s1

s2

s3

s3 faults; C started on s1 or s2
Regular membership: s1, s2

No membership: s3

Failover duration on a system failure

+Detect the system failure21 seconds

for heartbeat timeouts.
+Select a failover targetless than one
second.
+Bring the service group online on
another system in the cluster.
= Failover duration

Manual seeding
5
Seede
d
3

HAD

Seeds
4

GAB
2

LLT

HAD
1

GAB
LLT

s1

s2

s3 is down for maintenance. s1 and s2 are rebooted.

LLT starts on s1 and s2. GAB starts but cannot seed with s3
down.
Manually force GAB to seed on s1: gabconfig -x.

GAB on s2 now seeds because it can detect another seeded system

(s1).
Warning: Manual seeding
Start HAD on s1 and s2.
can cause split-brain
condition.

s3

Interconnect failure and potential split

brain condition
A

1
1

s1

s2

1 s1 and s2 determine that s3 is faulted.

1 No jeopardy occurs; no groups are
autodisabled.
If all systems are in all groups'
2
1 SystemList, VCS tries to bring them
online.

s3

s3 determines that s1
and s2 are faulted.

Interconnect failures with a low-priority

public link
A

1
1

s1

No change in membership

Regular membership: s1, s2, s3

s2

s3

Jeopardy membership: s3
2 Public network is now
1
used for heartbeat and
status

Changing the
interconnect
configuration

After completing this topic, you

will be able to change the cluster
interconnect configuration.

Example reconfiguration scenarios

Manual configuration:

s1

s3

s4

s2

Merging clusters
Changing parameters persistent
across reboots, such as the peer
inactive timeout

Automatic changes by CPI:

Adding or removing cluster nodes

The lltconfig command:

Changes parameters temporarily

set-node
set-node s1
s1
set-cluster
set-cluster 10
10
.. .. ..
set-timer
set-timer peerinact:1200
peerinact:1200

Requires running on each node to

affect cluster-wide

s6
s5

s7

Manually modifying the interconnect

/etc/gabtab

/etc/llttab
set-node s1
set-cluster 10

/etc/llthosts

path/gabconfig c n #

0 s1
1 s2

One system

Back up and
edit files
Stop VCS
Stop
fencing

All systems
haconf dump -makero
hastop all -force
Start VCS
vxfenconfig -U

hastart

Start
fencing

vxfenconfig c
sh /etc/gabtab

Stop GAB

gabconfig -U

Start GAB

Stop LLT

lltconfig -U

Start LLT

lltconfig -c

Example LLT link specification

Range (all)

SAP

/etc/llttab

set-node
s1
set-cluster
10
link nxge0 /dev/nxge:0 - ether - link nxge4 /dev/nxge:4 - ether - #
link-lowpri e1000g0 /dev/e1000g:0 - ether - Tag name

Device:Unit

Link type MTU

Lesson summary
Key points
The cluster interconnect is used for cluster membership and
status information.
The cluster interconnect configuration may never require
modification, but can be altered for site-specific requirements.

Reference materials
Veritas Cluster Server Installation Guide
Veritas Cluster Server Users Guide

End of Presentation