GPFS3.

1 for AIX

IBM AIX
AIX , IBM psystem
8 AIX CATE
: AIX5.3 GPFS3.1
GPFS


1. GPFS
GPFS IBM IBM SP
( VSD ) GPFS

GPFS
NFS GPFS SAN

2.
1

CD

pN SD
ser v er

CD

sNSD
ser v er
J S2 1

N SD1
JS 21

CD

c lie n t
J S2 1

c lie n t
J S2 1

N SD2
JS 21

CD

A pp 1
JS 21

CD

c lie n t
J S2 1

A pp2
JS 21

c lie n t
J S2 1

A pp 3
JS 21

To ta lS to ra g e

SAN

64G B

D S 8100

CD

J S2 1

A pp4
JS 21

ip

NSD1

NSD

10.66.3.98

2CPU/4G/AIX5305

NSD2

NSD

10.66.3.99

2CPU/4G/AIX5305

App1

10.66.5.51

user1501 2CPU/4G/AIX5305

App2

10.66.5.52

user2502 2CPU/4G/AIX5305

App3

10.66.5.53

user3503 2CPU/4G/AIX5305

App4

10.66.5.54

user4

504)

2CPU/4G/AIX5305

/share

GPFS id

/share/user1work App1 user1 App1

1./share/user1work
user1 700 id

/share/user1temp userx
x user1
2/share/user1temp
750 userx bea bea
userxx


SAN

1.

1 gpfs volume group

2 host zone
3 DSCLI volume group hostconnect
mkgroup volume group v3 chvolgrp fbvol volume
group hostconnect wwn IOPORT
group

Date/Time: August 13, 2007 7:40:05 PM GMT+08:00 IBM DSCLI Version: 5.2.400.426 DS:
IBM.2107-75L3351
Name

ID

WWPN

volgrpID
App1

HostType Profile

portgrp

ESSIOport
IBM.2107-75L3351/0020 210000E08B9B4D48 pSeries

IBM pSeries - AIX

IBM pSeries - AIX

IBM.2107-75L3351/V6 I0232,I0032,I0302,I0102
App1

IBM.2107-75L3351/0021 210100E08BBB4D48 pSeries

IBM.2107-75L3351/V6 I0232,I0032,I0302,I0102

4 sddpcm
sddpcm IBM AIX MPIO
failover
sddpcm
devices.fcp.disk.ibm.mpio.rte 1.0.0.7
# IBM MPIO FCP Disk Device
devices.sddpcm.53.rte 2.1.2.5
# IBM SDD PCM for AIX V53
4
pcmpath query essmap
[App1][root][/]>pcmpath
Disk

Path

Rank C/A
-------

fffb

17

path0
Y

hdisk3
fffb

17

hdisk3
fffb

17

hdisk3
fffb

17

32

--------

LUN SN

Type

Size

LSS

Vol

----

---

---

----

---

75L33510125

IBM 2107-900 65.5GB

37

75L33510125

IBM 2107-900 65.5GB

37

75L33510125

IBM 2107-900 65.5GB

37

75L33510125

IBM 2107-900 65.5GB

37

RAID5

01-11-01[FC] fscsi1
RAID5

01-11-01[FC] fscsi1

R1-B4-H1-ZC 302

------------

RAID5

01-10-01[FC] fscsi0

R1-B3-H3-ZC 232
path3

adapter

01-10-01[FC] fscsi0

R1-B2-H1-ZC 102
path2

---------- --------

R1-B1-H3-ZC
path1

Location

Connection port RaidMode

----- -

hdisk3

RAID5

5 cfgmgr

 App3
lspv
[App3][root][/home/scripts]>lspv
hdisk2

none

None

pcmpath query essmap

[App3][root][/home/scripts]>pcmpath
Disk

Path

Rank C/A

-------

----- -

Location

adapter

LUN SN

Type

Size

LSS

Vol

Connection port RaidMode

----------

--------

--------

------------

----

---

---

----

---

-----------

---- -------hdisk3
0e

hdisk3
0e

hdisk3
0e

hdisk3

path0
R1-B1-H3-ZC
path1

01-10-01[FC] fscsi0
32

path2

path3

0e

2.

37

fffc

75L33510025

IBM 2107-900 65.5GB

37

fffc

75L33510025

IBM 2107-900 65.5GB

37

fffc

75L33510025

IBM 2107-900 65.5GB

37

fffc

RAID5

01-11-01[FC] fscsi1

R1-B4-H1-ZC 302

RAID5

01-11-01[FC] fscsi1

R1-B3-H3-ZC 232

IBM 2107-900 65.5GB

RAID5

01-10-01[FC] fscsi0

R1-B2-H1-ZC 102

75L33510025

RAID5

1 /etc/hosts,
/etc/hosts
#gpfs start
10.66.3.98
NSD1
10.66.3.99
NSD2
10.66.5.51
App1
10.66.5.52
App2
10.66.5.53
App3
10.66.5.54
App4
#gpfs end
2 /.rhosts, 2 NSD
/.rhosts
NSD1 root
NSD2 root
3 shell
a. run_cmd.sh #
cat machines.list|awk '{print "rsh "$1" \"hostname;$1 $2 $3 $4 $5\""}'>/tmp/test.sh
cat /tmp/test.sh
sh /tmp/test.sh $1 $2 $3
$4
$5
rm /tmp/test.sh
b. machines.list #
10.66.5.51
App1
10.66.5.52
App2
10.66.5.53
App3
10.66.5.54
App4

10.66.3.98
10.66.3.99

NSD2
NSD2

c. rcp_file.sh # NSD
cat machines.list|awk '{print "rcp $1 "$1":$1"}'>/tmp/test.sh
cat /tmp/test.sh
sh /tmp/test.sh $1
rm /tmp/test.sh
d. install_gpfs.sh # gpfs gpfs /soft_ins/gpfs3.1
cd /soft_ins/gpfs3.1
installp -acYd . all
e. change_profile.sh .proifle
echo "export PATH=\$PATH:/usr/lpp/mmfs/bin:." >> /.profile
4 2 NSD rsh,rcp ok
run_cmd.sh date
[NSD1][root][/home/scripts/gpfs]>run_cmd.sh
App1
Thu Aug 2 19:37:00 BEIST 2007
App2
Thu Aug 2 19:37:00 BEIST 2007 NSD1
........
run _cmd.sh mkdir /home/scripts/gpfs
[NSD1][root][/home/scripts/gpfs]>run
rcp_file.sh /home/scripts/gpfs
/change_profile.sh
[NSD1][root][ /home/scripts/gpfs]>rcp_file.sh
/home/scripts/gpfs/change_profile.sh
run _cmd.sh ls -l /home/scripts/gpfs
[NSD1][root][/home/scripts/gpfs]>run
App1
-rw-r--r-1 root
system
37 Jun 28 14:08 change_profile.sh

 GPFS
1. install_gpfs.sh gpfs
/soft
_ins NFS
/soft
/soft_ins
rcp_file.sh /home/scripts/gpfs/install_gpfs.sh
[NSD1][root][/home/scripts/gpfs]>rcp_file.sh
[NSD1][root][/home/scripts/gpfs]> run _cmd.sh /home/scripts/gpfs/install_gpfs.sh
ok,
.profile
2. .profile
run _cmd.sh /home/scripts/gpfs/chang_profile.sh
[NSD1][root][/home/scripts/gpfs]>run
3.
run _cmd.sh mkdir /share
[NSD1][root][/home/scripts/gpfs]>run
run _cmd.sh ln -s /share /tmp/
mmfs
[NSD1][root][/home/scripts/gpfs]>run
/tmp/mmfs
run _cmd.sh mkdir /tmp/gpfs
[NSD1][root][/home/scripts/gpfs]>run

 GPFS
1.
vi /tmp/gpfs/nodefile
[NSD1][root][/]>vi
NSD1:quorum
NSD2:quorum
App1:client
App2:client
App3:client
App4:client
2. GPFS
GPFS
mmdelnode -f
[NSD1][root][/tmp/gpfs]>mmdelnode

mmcrcluster -C bgbcrun -U bgbc -N /tmp/gpfs/nodefi

le -p NSD1 -s
[NSD1][root][/tmp/gpfs]>mmcrcluster
/tmp/gpfs/nodefile
NSD2
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD1
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD2
..
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

-C bgbcrun
-U bgbc

-N /tmp/gpfs/nodefile
-p NSD1 NSD NSD1
-s NSD1 NSD NSD1
3.
mmlscluster
[NSD1][root][/tmp/gpfs]>mmlscluster
GPFS cluster information
========================
GPFS cluster name:
bgbcrun.NSD1
GPFS cluster id:
739157013761844865
GPFS UID domain:
bgbc
Remote shell command:
/usr/bin/rsh
Remote file copy command: /usr/bin/rcp
GPFS cluster configuration servers:

----------------------------------Primary server:
NSD1
Secondary server: NSD2
Node
Daemon node name
IP address
Admin node name
Designation
----------------------------------------------------------------------------------------------1
NSD1
10.66.3.98
NSD1
quorum
2
NSD2
10.66.3.99
NSD2
quorum
3 App1
10.66.5.51
App1
4 App2
10.66.5.52
App2
5 App3
10.66.5.54
App3
6 App4
10.66.5.55
App4

 GPFS
Network Shared Disk
1. NSD
NSDNetwork
Disk
vi /tmp/gpfs/nsdfile
[NSD1][root][/tmp/gpfs]>vi

hdisk2:NSD1:NSD2: dataAndMetadata:4
2. NSD
mmcrnsd -F /tmp/gpfs/nsdfile
[NSD1][root][/tmp/gpfs]>mmcrnsd
mmcrnsd: Processing disk hdisk2
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

cat nsdfile
[NSD1][root][/tmp/gpfs]>cat
# hdisk2:NSD2:NSD1: dataAndMetadata:4
gpfs1nsd:::dataAndMetadata:4:
3.
lspv
[NSD1][root][/tmp/gpfs]>lspv
hdisk2
00003e846ffa7a6e

gpfs1nsd

4. GPFS
mmstartup -a
[NSD2][root][/tmp/gpfs]>mmstartup
Thu Jun 28 15:52:12 BEIST 2007: 6027-1642 mmstartup: Starting GPFS ...
NSD2: 6027-2114 The GPFS subsystem is already active.

App4: 6027-2114 The GPFS subsystem is already active.

 GPFS
1. GPFS
mmcrfs /share sharelv -F /tmp/gpfs/nsdfile -A yes -B 64K -n 30 -v no
[NSD2][root][/]>mmcrfs
GPFS: 6027-531 The following disks of sharelv will be formatted on node NSD1:
gpfs1nsd: size 67108864 KB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 140 GB can be added to storage pool 'system'.
Creating Inode File
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
GPFS: 6027-572 Completed creation of file system /dev/sharelv.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

/share mount
sharelv lv
-F
NSD
-A mount yes
-B
64K
-n
30
-v

2.
mmlsfs sharelv
[NSD1][root][/tmp/gpfs]>mmlsfs
flag value
description
---- -------------- -----------------------------------------------------s roundRobin
Stripe method

-a 1048576
Estimated average file size
-n 30
Estimated number of nodes that will mount file system
-B 65536
Block size
.
-A automount
Automatic mount option
-o none
Additional mount options
-T /share
Default mount point
mmchfs

3.
run_cmd.sh mount /share
[NSD2][root][/home/scripts/gpfs]>run_cmd.sh

4.
mkdir /share/user1work
[App1][root][/]>mkdir
chown user1:bea /share/user1work
[App1][root][/]>chown
[App1][root][/]>chmod 700 /share/user1work
mkdir /share/user1temp
[App1][root][/]>mkdir
chown user1:bea /share/user1temp
[App1][root][/]>chown
[App1][root][/]> chmod 750 /share/user1temp
3 work
temp

 GPFS
1. GPFS
gpfs
mmchconfig autoload=yes
[NSD1][root][/]>mmchconfig
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Quorum
no
[NSD1][root][/]> mmchconfig singleNodeQuorum=
singleNodeQuorum=no
mmchconfig: 6027-1119 Obsolete option: singleNodeQuorum
2. GPFS
mmlsconfig
[NSD1][root][/]>mmlsconfig
Configuration data for cluster bgbcrun.NSD2:
----------------------------------------------clusterName bgbcrun.NSD2
clusterId 739157013761844865
clusterType lc
autoload no
useDiskLease yes
uidDomain bgbc
maxFeatureLevelAllowed 912
GPFS
mmgetstate -a
[NSD1][root][/]>mmgetstate
Node number Node name
GPFS state
-----------------------------------------1
NSD1
active
2
NSD2
active
3
App1
active
4
App2
active
5
App4
active
10
App3
active


userid App1 /share user1
temp work
ls -latr
[App1][/share]>ls
drwx-----2 user1
bea
2048 Jun 28 17:21 user1work
drwxr--x-- 2 user1
bea
2048 Jun 28 17:21 user1temp
drwx-----2 502
bea
2048 Jun 28 17:22 user2work
drwxr--x--2 502
bea
2048 Jun 28 17:22 user2temp
drwx-----2 503
bea
2048 Jun 28 17:22 user3work
drwxr-x-2 503
bea
2048 Jun 28 17:22 user3temp
drwxr---2 504
bea
2048 Jun 28 17:22 user4work
drwxr-x--2 504
bea
2048 Jun 28 17:23 user4temp
cd user2work
[App1][user1][/share]>cd
ksh: user2temp : Permission denied.
cd user2
temp
[App1][user1][/share]>cd
user2temp
>a
[App1][user1][/share]>>a
The file access permissions do not allow the specified action.
ksh[2]: a: 0403-005 Cannot create the specified file.
cd /user1
temp
[App1][user1][/share]>cd
user1temp
>a
[App1][user1][/share/user1temp]>>a
ls -l
[App1][user1][/share/user1temp]>ls
-rw-r----1 user1
bea
0 Aug 13 18:47 a
rm a
[App1][user1][/share/xhtemp]>rm
ls -l
[App1][user1][/share/xhtemp]>ls
ls -latr
[App1][/share]>ls
drwxr-----2 user1
bea
2048 Jun 28 17:21 user1work
drwxr-----2 user1
bea
2048 Jun 28 17:21 user1temp
drwxr-----2 502
bea
2048 Jun 28 17:22 user2work
drwxr-----2 502
bea
2048 Jun 28 17:22 user2temp
drwxr-----2 503
bea
2048 Jun 28 17:22 user3work
drwxr-----2 503
bea
2048 Jun 28 17:22 user3temp
drwxr-----2 504
bea
2048 Jun 28 17:22 user4work
drwxr-----2 504
bea
2048 Jun 28 17:23 user4temp
3

 GPFS
1
1.

mmaddnode -N bgbc
w14:client
[NSD1][root][/home/scripts/gpfs]>mmaddnode
bgbcw
Thu Jun 28 16:28:21 BEIST 2007: 6027-1664 mmaddnode: Processing node App3
mmaddnode: Command successfully completed
mmaddnode: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
2.
vi /tmp/gpfs/nsdfile
[NSD1][root][/tmp/gpfs]>vi

hdisk2:NSD1:NSD2: dataAndMetadata:4
NSD
mmcrnsd -F /tmp/gpfs/nsdfile
[NSD1][root][/tmp/gpfs]>mmcrnsd
mmcrnsd: Processing disk hdisk2
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

cat nsdfile
[NSD1][root][/tmp/gpfs]>cat
# hdisk2:NSD2:NSD1: dataAndMetadata:4
gpfs1nsd:::dataAndMetadata:4:

lspv
[NSD1][root][/tmp/gpfs]>lspv
hdisk2
00003e846ffa7a6e
gpfs2nsd

mm
add
disk sharelv gpfs2nsd
[NSD1][root][/tmp/gpfs]>mm
mmadd
adddisk

[NSD1][root][/tmp/gpfs]>
del
disk sharelv gpfs1nsd
mm
mmdel
deldisk

3.

mmshutdown -a
[NSD2][root][/home/scripts/gpfs]>mmshutdown
Mon Jul 30 09:56:02 BEIST 2007: 6027-1341 mmshutdown: Starting force unmount of GPFS file
systems
NSD1: forced unmount of /share

App4: forced unmount of /share

Mon Jul 30 09:56:07 BEIST 2007: 6027-1344 mmshutdown: Shutting down GPFS daemons

NSD1: Shutting down!

App3: Shutting down!

NSD1: 'shutdown' command about to kill process 368890
.
App4: 'shutdown' command about to kill process 474040
Mon Jul 30 09:56:13 BEIST 2007: 6027-1345 mmshutdown: Finished
mmshutdown -N
4. inode
mmchfs /dev/sharelv -F 1024000
[bgbcc13][root][/]>mmchfs
mmchfs /dev/shareplv -F 1024000
[bgbcc13][root][/]>mmchfs
5.
1.
2.
3.
4.
5.
6.
7.
8.

GPFS
fuser kcu /share
unmount /share #
mmdelfs sharelv,
mmlsfs sharelv #
mmdelnsd F /tmp/gpfs/nsdfile
mmshutdown a
mmdelnode n /tmp/gpfs/nodefile
mdelnode f #

GPFS
GPFS userid
50 GPFS GPFS
GPFS