<Insert Picture Here>

Infiniband TOI Session 3 & 4

Muppalla Sridhar
SAND-NETWORK
muppalla.sridhar@oracle.com
Agenda

Infiniband Switch Firmware versions

Configuration
Troubleshooting

2
NM2-36P Firmware Versions

1.0.1
1.1.3
1.1.4
1.3.3-2

3
 Exadata: NM2-36p Infiniband Switch firmware
history
v1.0.1
original release for switches shipped in V2
v1.1.3-2b
patch release for switches used in V2. Original version
shipped in X2-2/X2-8
required for updating to the v1.3.3-2 release
v1.3.3-1
withdrawn due to memory leak issue
v1.3.3-2
patch release with new OFED stack; resolves issues with
Subnet Manager that can hang the fabric. Highly
recommended update for all customers. Check script and
instructions are in patch 12373676 and firmware package in
patch 11891229 (you need both)

4
NM2-36P firmware version

On the management controller, enter:

versio(newer version) or nm2version(older command)
root@gmpasw-ib2 is4_0]# version
SUN DCS 36p version: 1.1.3-2
Build time: Mar 25 2010 10:00:23
SP board info:
Manufacturing Date: 2009.06.22
Serial Number: "NCD3R0335"
Hardware Revision: 0x0006
Firmware Revision: 0x0102
BIOS version: NOW1R112
BIOS date: 04/24/2009

5
NM2-GW Firmware version

1.0.1
1.1.2 MOS patch 11732400

6
Exalogic: NM2-GW Firmware

Only 1.1.2 (MOS patch 11732400) is currently

supported

7
NM2-GW firmware version

On the management controller, enter

Version
# version
SUN DCS gw version: 1.1.2-2
Build time: Sep 24 2010 10:32:29
FPGA version: 0x33
SP board info:
Manufacturing Date: 2010.04.26
Serial Number: "NCD2T0518"
Hardware Revision: 0x0005
Firmware Revision: 0x0000
BIOS version: SUN0R100
BIOS date: 06/22/2010

8
 IB Switch Firmware Upgrade/Downngrade
Login as root
Shutdown and disable SM using disablesm. If the
switch was running the master SM, the master SM will
now relocate to another switch on the fabric
#disablesm
From the IB switchprompt, launch spsh
# spsh
This will give the ILOM prompt. Now execute the load
command
-> load -source tftp://10.7.1.70/sundcs_36p_repository_1.3.3_2.pkg
After FW upgrade is complete, exit from ILOM. This
takes you back to linux prompt and reboot switch
-> exit
#reboot

9
NM2-36p & NM2-GW
CONFIGURATION

10
NM2-36 Port Infiniband Switch

11
NM2-GW Infiniband Switch

12
13
14
15
16
17
18
Infiniband Switch Node description

<NM-Switch># cat /sys/class/infiniband/is4_0/node_desc

Sun DCS 36 QDR switch gmpasw-ib2.uk.sun.com

19
 Starting the Subnet Manager Manually
Start the SM by running the following command
#enablesm
Verify by executing below command
# ps -ef | grep opensm | grep -v grep
root 16434 1 0 May13 ? 00:01:23 /usr/sbin/opensm --daemon
Set the SM priority
#setsmpriority priority
ex # setsmpriority 5 ( sets the SM priority to 5)

The following output is displayed:

-------------------------------------------------

OpenSM 3.2.6_20090717

Reading Cached Option File: /etc/opensm/opensm.conf

Loading Cached Option:routing_engine = ftree

Loading Cached Option:sminfo_polling_timeout = 1000

Loading Cached Option:polling_retry_number = 3

Command Line Arguments:

Priority = 5

Creating config file template '/tmp/osm.conf'.

Log File: /var/log/opensm.log

-------------------------------------------------

20
Starting the Subnet Manager Manually(Contd..)

For the changes to take effect, restart the SM as

follows
#disablesm
#enablesm

21
 NM2-GW
Ethernet over Infiniband Configuration

22
Setting up Ethernet over Infiniband

Use an SSH client to log in to NM2-GW switch

Determine the HCA port GUIDs, MAC addresses, the
partition keys and the connectors that will be assigned
to the VNICs
Create the VNIC on the management controller
<NM2-GW>#createvnic connector -guid guid -mac mac -vlan vlan_ID
-pkey p_key
<NM2-GW>#createvnic 0A-ETH-3 -guid 00:21:28:00:01:a0:d7:0d -mac
a0:d7:0d:01:10:04 -pkey default
In Exalogic, unique mac address is created using last 3 octets from
host ib guid followed by last 3 octets created using admin ip.
Connector is the name of the connector(0A-ETH-1 to 0A-ETH-4, 0A-
ETH, 1A-ETH-1 to 1A-ETH-4 and 1A-ETH)

23
Setting up Ethernet over Infiniband(Contd..)

24
 Setting up Ethernet over Infiniband(Contd..)
Linux running on Exalogic Compute node
Create interfaces eth4 & eth5 by using the unique MAC
address created for the vNICs as HWADDR on each ethX file
Reboot the compute node to create the interfaces

25
Setting up Ethernet over Infiniband(Contd..)
Oracle Solaris running on Exalogic Compute node

26
Solaris IPoIB Configuration &
Troubleshooting

27
IpoIB Configuration

28
IpoIB Configuration(contd..)

29
IpoIB Configuration(Contd..)

Verify that the devices are plumbed

#ifconfig -a
Check that data can be send over the Infiniband link
#ping host-ip-address

30
IpoIB Troubleshooting

31
IpoIB Troubleshooting(Contd..)

32
IpoIB Troubleshooting(Contd..)

Snoop
The snoop program captures and inspects network
packets
#snoop -d ibd1
Netstat
Netstat shows network status
#netstat -l ibd 1
Kstat
Kstat displays kernel statistics
#kstat ibd:1

33
TROUBLESHOOTING

34
 NM2-36p & NM2-GW :
Administrative Commands

35
Hardware Commands

chassis_led
checkboot
checkpower
checkvoltages
connector
dcsport
disablecablelog
disablelinklog
disablesm
disableswitchport

36
Hardware Commands(Contd..)

enablecablelog
enablelinklog
enablesm
enableswitchport
env_test
getfanspeed
getmaster
getportstatus
ibdevreset
listlinkup

37
Hardware commands(contd..)

managementreset
setcontrolledhandover
setloghost
setsmpriority
setsubnetprefix
showfruinfo
showpsufru
showtemps
showunhealthy
version

38
Infiniband commands

ibaddr
ibcheckerrors
ibchecknet
ibchecknode
ibcheckport
ibcheckportstate
ibcheckportwidth
ibcheckstate
ibcheckwidth
ibclearcounters

39
Infiniband Commands (Contd..)

ibdatacounters
ibdatacounts
ibdiagnet
ibdiagpath
ibhosts
ibnetdiscover
ibnodes
ibportstate
ibroute
ibstat

40
Infiniband Commands (Contd..)
ibstatus
ibswitches
ibsysstat
ibtracert
opensm
opensmd
osmtest
perfquery
saquery
sminfo
smpdump
smpquery

41
NM2-GW Commands

# help gwresource
Ethernet Gateway resource administration commands:
showvnics - Show the virtual NIC resources on the gateway
showvlan - Show the virtual LAN resources for the gateway
createvnic - Create a virtual NIC resource on a gateway port
deletevnic - Delete a virtual NIC resource on a gateway port
enablevnic - Enable a virtual NIC resource on a gateway port
disablevnic - Disable a virtual NIC resource on a gateway port
createvlan - Create a virtual LAN resource on a gateway port
deletevlan - Delete a virtual LAN resource on a gateway port

42
 NM2-GW Commands(Contd..)
# help gwconfig
Ethernet Gateway configuration commands:
showgwports - Show status for internal and external GW ports
showgwconfig - Show the GW configuration parameters
enablegwport - Enable an external gateway port
disablegwport - Disable an external gateway port
setgwsl - Set the EoIB and CTRL SL used by the GW
setgwinstance - Set the instance number of the GW
setgwethport - Configure an Ethernet port on the GW
setgwsystemname - Set the system name of the GW
createlag - Create LAG consisting of a number of external GW ports
deletelag - Delete an existing LAG
addlagport - Add a new external GW port to a LAG
dellagport - Removes an external GW port from LAG
modifylag - Modify parameters for a LAG
showlag - List of existing LAGs

43
ibaddr

NM2-36P
<NM2-36p># ibaddr
GID fe80::21:283a:87ee:a0a0 LID start 0x3c end 0x3c
NM2-GW
<NM2-GW>#ibaddr
GID fe80::21:2856:ce62:c0a0 LID start 0x6 end 0x6
HOST MACHINE
#ibaddr
GID fe80::21:2800:13e:6bc3 LID start 0xa end 0xa

44
 Ibstat (switch output)
<NM2-SW># ibstat
Switch 'is4_0'
Switch type: MT48436
Number of ports: 0
Firmware version: 7.2.300
Hardware version: a0
Node GUID: 0x0021283a87eea0a0
System image GUID: 0x0021283a87eea0a3
Port 0:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 60
LMC: 0
SM lid: 60 <Subnet Manager LID>
Capability mask: 0x0250084a
Port GUID: 0x0021283a87eea0a0

45
 ibstat(Host Machine Output)
# ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.7.0
Hardware version: a0
Node GUID: 0x00212800013e6bc2
System image GUID: 0x00212800013e6bc5
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 10 <Unique lid # assigned by Subnet Manager to port 1>
LMC: 0
SM lid: 60 <Subnet Manager lid>
Capability mask: 0x02510868
Port GUID: 0x00212800013e6bc3

46
ibstat(Host Machine Output)(Contd..)

Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 62 <Unique lid # assigned by Subnet Manager to port 2>
LMC: 0
SM lid: 60<Subnet Manager lid>
Capability mask: 0x02510868
Port GUID: 0x00212800013e6bc4

47
ibstatus(switch output)

<NM-SW># ibstatus
Infiniband device 'is4_0' port 0 status:
default gid:
fe80:0000:0000:0000:0021:283a:87ee:a0a0
base lid: 0x3c
sm lid: 0x3c
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)

48
 ibstatus (Host Machine Output)
# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:0021:2800:013e:6bc3
base lid: 0xa
sm lid: 0x3c
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)

Infiniband device 'mlx4_0' port 2 status:

default gid: fe80:0000:0000:0000:0021:2800:013e:6bc4
base lid: 0x3e
sm lid: 0x3c
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)

49
dcsport

dcsport
Maps between switch chip ports and QSFP connectors
dcsport [-port port|-connector connector|-printconnectors]
port is the number of the port (136)
connector is the number of the QSFP connector (0A17B).

50
 dcsport(NM2-36p)
<NM2-36p>]# dcsport -printconnectors
DCS-36P connectors:
Connector 0A maps to Switch port 20
Connector 1A maps to Switch port 22
Connector 2A maps to Switch port 24
Connector 3A maps to Switch port 26
Connector 4A maps to Switch port 28
Connector 5A maps to Switch port 30
Connector 6A maps to Switch port 35
Connector 7A maps to Switch port 33
Connector 8A maps to Switch port 31
Connector 9A maps to Switch port 14
Connector 10A maps to Switch port 16
Connector 11A maps to Switch port 18
Connector 12A maps to Switch port 11
Connector 13A maps to Switch port 9
Connector 14A maps to Switch port 7
Connector 15A maps to Switch port 5
Connector 16A maps to Switch port 3
Connector 17A maps to Switch port 1

51
 dcsport(NM2-36p) (Contd..)
Connector 0B maps to Switch port 19
Connector 1B maps to Switch port 21
Connector 2B maps to Switch port 23
Connector 3B maps to Switch port 25
Connector 4B maps to Switch port 27
Connector 5B maps to Switch port 29
Connector 6B maps to Switch port 36
Connector 7B maps to Switch port 34
Connector 8B maps to Switch port 32
Connector 9B maps to Switch port 13
Connector 10B maps to Switch port 15
Connector 11B maps to Switch port 17
Connector 12B maps to Switch port 12
Connector 13B maps to Switch port 10
Connector 14B maps to Switch port 8
Connector 15B maps to Switch port 6
Connector 16B maps to Switch port 4
Connector 17B maps to Switch port 2

52
 dcsport(NM2-GW)

<NM2-GW># dcsport -printconnectors

DCS-GW connectors:
Connector 0A maps to Switch port 20
Connector 1A maps to Switch port 22
Connector 2A maps to Switch port 24
Connector 3A maps to Switch port 26
Connector 4A maps to Switch port 28
Connector 5A maps to Switch port 30
Connector 6A maps to Switch port 35
Connector 7A maps to Switch port 33
Connector 8A maps to Switch port 31
Connector 9A maps to Switch port 14
Connector 10A maps to Switch port 16
Connector 11A maps to Switch port 12
Connector 12A maps to Switch port 18
Connector 13A maps to Switch port 9
Connector 14A maps to Switch port 7
Connector 15A maps to Switch port 5

53
 dcsport(NM2-GW)(Contd..)
Connector 0A-ETH maps to:
0A-ETH-1 Bridge-0 port Bridge-0-2
0A-ETH-2 Bridge-0 port Bridge-0-2
0A-ETH-3 Bridge-0 port Bridge-0-1
0A-ETH-4 Bridge-0 port Bridge-0-1
Connector 1A-ETH maps to:
1A-ETH-1 Bridge-1 port Bridge-1-2
1A-ETH-2 Bridge-1 port Bridge-1-2
1A-ETH-3 Bridge-1 port Bridge-1-1
1A-ETH-4 Bridge-1 port Bridge-1-1
Connector 0B maps to Switch port 19
Connector 1B maps to Switch port 21
Connector 2B maps to Switch port 23
Connector 3B maps to Switch port 25
Connector 4B maps to Switch port 27
Connector 5B maps to Switch port 29
Connector 6B maps to Switch port 36
Connector 7B maps to Switch port 34
Connector 8B maps to Switch port 32

54
 dcsport(NM2-GW)(Contd..)

Connector 9B maps to Switch port 13

Connector 10B maps to Switch port 15
Connector 11B maps to Switch port 17
Connector 12B maps to Switch port 11
Connector 13B maps to Switch port 10
Connector 14B maps to Switch port 8
Connector 15B maps to Switch port 6
Connector 0B-FC maps to:
0B-FC-1 Bridge-0 port Bridge-0-1
0B-FC-2 Bridge-0 port Bridge-0-1
0B-FC-3 Bridge-0 port Bridge-0-2
0B-FC-4 Bridge-0 port Bridge-0-2
Connector 1B-FC maps to:
1B-FC-1 Bridge-1 port Bridge-1-1
1B-FC-2 Bridge-1 port Bridge-1-1
1B-FC-3 Bridge-1 port Bridge-1-2
1B-FC-4 Bridge-1 port Bridge-1-2

55
 listlinkup(NM2-GW)
<NM2-GW># listlinkup
Connector 0A Present <-> Switch Port 20 up (Enabled)
Connector 1A Present <-> Switch Port 22 up (Enabled)
Connector 2A Not present <CABLE NOT PRESSENT>
Connector 3A Present <-> Switch Port 26 up (Enabled)
...
Connector 8A Present <-> Switch Port 31 down (Enabled)
...
Connector 0A-ETH Present
Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-2 (Bridge-0-2) up (Enabled)
Bridge-0 Port 0A-ETH-3 (Bridge-0-1) down (Enabled)
Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled)
Connector 1A-ETH Present
Bridge-1 Port 1A-ETH-1 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled)
Bridge-1 Port 1A-ETH-3 (Bridge-1-1) down (Enabled)
Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)
....

56
 Monitoring Subnet Manager
# sminfo
sminfo: sm lid 60 sm guid 0x21283a87eea0a0, activity count 590793 priority 5 state 3
SMINFO_MASTER
# getmaster
20110513 11:49:00 OpenSM Master on Switch : 0x0021283a87eea0a0 ports 36 Sun DCS 36
QDR switch gmpasw-ib2.uk.sun.com enhanced port 0 lid 60 lmc 0
# getmaster -l
Last ring buffer history listed:
whereismaster-daemon is running
20110513 08:56:11 No OpenSM Master seen in the system
20110513 09:26:17 No OpenSM Master seen in the system
20110513 09:56:22 No OpenSM Master seen in the system
20110513 10:26:29 No OpenSM Master seen in the system
20110513 10:56:36 No OpenSM Master seen in the system
20110513 11:26:42 No OpenSM Master seen in the system
20110513 11:49:00 OpenSM Master on Switch : 0x0021283a87eea0a0 ports 36 Sun DCS 36
QDR switch gmpasw-ib2.uk.sun.com enhanced port 0 lid 60 lmc 0

57
 Monitoring the Subnet Manager(Contd..)

Ibdiagnet -r
..
-I---------------------------------------------------
-I- Summary Fabric SM-state-priority
-I---------------------------------------------------
SM - master
The Local Device : Port=0 lid=0x003c guid=0x0021283a87eea0a0
dev=48438 priority:5
SM - standby
Port=14 lid=0x003b guid=0x0021283a8823a0a0 dev=48438 priority:5
......
......

58
Monitoring Infiniband Fabric

59
 Identifying all Switches in the fabric

#Ibswitches
Switch : 0x0021283a87eea0a0 ports 36 "Sun DCS 36 QDR
switch gmpasw-ib2.uk.sun.com" enhanced port 0 lid 60 lmc 0
Switch : 0x002128b380e8c0a0 ports 36 "SUN IB QDR GW
switch v-exalogic-a-gmp03-ib02-gw 192.168.1.202 l" enhanced
port 0 lid 6 lmc 0
Switch : 0x002128b3d308c0a0 ports 36 "SUN IB QDR GW
switch v-exalogic-a-gmp03-ib01-gw 192.168.1.201 l" enhanced
port 0 lid 15 lmc 0
Switch : 0x0021283a8823a0a0 ports 36 "Sun DCS 36 QDR LC
switch gmpasw-ib3.uk.sun.com " enhanced port 0 lid 59 lmc 0

60
 Identify all HCAs in the Fabric

# ibhosts
Ca : 0x00212800013e6bc2 ports 2 "gmpadb04 S 192.168.2.124 HCA-1"
Ca : 0x00212800013e6a8e ports 2 "gmpadb02 S 192.168.2.122 HCA-1"
Ca : 0x00212800013e6ae2 ports 2 "gmpadb03 S 192.168.2.123 HCA-1"
Ca : 0x00212800013e6a92 ports 2 "gmpacel07 C 192.168.10.31 HCA-1"
Ca : 0x00212800013e6b06 ports 2 "gmpadb01 S 192.168.2.121 HCA-1"
Ca : 0x00212800013e6a56 ports 2 "gmpacel05 C 192.168.2.115 HCA-1"
Ca : 0x00212800013e6ad2 ports 2 "gmpacel06 C 192.168.10.30 HCA-1"
Ca : 0x00212800013e6aa2 ports 2 "gmpacel03 C 192.168.2.113 HCA-1"
Ca : 0x00212800013e6aea ports 2 "gmpacel04 C 192.168.2.114 HCA-1"
Ca : 0x00212800013e6aaa ports 2 "gmpacel01 C 192.168.2.111 HCA-1"
Ca : 0x00212800013e6ada ports 2 "gmpacel02 C 192.168.2.112 HCA-1"
Ca => Channel Adapter

61
 ibcheckerrors

# ibcheckerrors
#warn: counter SymbolErrors = 65532 (threshold 10) lid 60 port 255
#warn: counter XmtDiscards = 16074 (threshold 100) lid 60 port 255
Error check on lid 60 (Sun DCS 36 QDR switch gmpasw-ib2.uk.sun.com) port
all: FAILED
#warn: counter SymbolErrors = 65532 (threshold 10) lid 59 port 255
Error check on lid 59 (Sun DCS 36 QDR LC switch gmpasw-ib3.uk.sun.com )
port all: FAILED
#warn: counter RcvErrors = 299 (threshold 10) lid 67 port 1
Error check on lid 67 (gmpacel04 C 192.168.2.114 HCA-1) port 1: FAILED

## Summary: 13 nodes checked, 0 bad nodes found

## 58 ports checked, 1 ports have errors beyond threshold

62
ibnetdiscover

ibnetdiscover
Vendorid
Mellanox 0x2c9
Cisco 0x5ad
Silverstorm 0x66a
Voltaire 0x8f1
IBM 0x5076
QLogic 0x1fc1

63
 ibnetdiscover(contd..)

#ibnetdiscover
..
vendid=0x2c9
devid=0x673c
sysimgguid=0x212800013e6a59
caguid=0x212800013e6a56
Ca 2 "H-00212800013e6a56" # "gmpacel05 C 192.168.2.115
HCA-1"
[2](212800013e6a58) "S-0021283a8823a0a0"[6] # lid 54 lmc 0
"Sun DCS 36 QDR LC switch gmpasw-ib3.uk.sun.com " lid 59 4xQDR
[1](212800013e6a57) "S-0021283a87eea0a0"[6] # lid 5 lmc 0 "Sun
DCS 36 QDR switch gmpasw-ib2.uk.sun.com" lid 60 4xQDR
..

64
perfquery

Usage: perfquery [-d(ebug) -G(uid) -a(ll_ports)

-l(oop_ports) -r(eset_after_read) -C ca_name -P
ca_port -R(eset_only) -t(imeout) timeout_ms
-V(ersion) -h(elp)] [<lid|guid> [[port] [reset_mask]]]
Examples
perfquery # read local port's performance counters
perfquery 32 1 # read performance counters from lid 32, port
1
perfquery -e 32 1 # read extended performance counters from
lid 32, port 1
perfquery -a 32 # read performance counters from lid 32, all
ports

65
perfquery(Contd..)

Perfquery examples
perfquery -r 32 1 # read performance counters and reset
perfquery -e -r 32 1 # read extended performance counters
and reset
perfquery -R 0x20 1 # reset performance counters of port 1
only
perfquery -e -R 0x20 1 # reset extended performance counters
of port 1 only
perfquery -R -a 32 # reset performance counters of all ports
perfquery -R 32 2 0x0fff # reset only error counters of port 2
perfquery -R 32 2 0xf000 # reset only non-error counters of port 2

66
 perfquery(contd..)

Perfquery 6 1
# Port counters: Lid 6 port 1
PortSelect:......................1
CounterSelect:...................0x1b01
SymbolErrors:....................65535 < 8/10b encoding error
LinkRecovers:....................0 < link stability
LinkDowned:......................1 < times link went down
RcvErrors:.......................0 < crc hit on data alway s inc symbol
err also
RcvRemotePhysErrors:.............0 < error seen upstream
RcvSwRelayErrors:................0 < not a error
XmtDiscards:.....................65535 < tx buffer full

67
perfquery(Contd..)

perfquery 6 1 (Contd..)
XmtConstraintErrors:.............0 <congestion
RcvConstraintErrors:.............4 < congestion
LinkIntegrityErrors:.............0 < encoding error
ExcBufOverrunErrors:.............0
VL15Dropped:.....................0
XmtData:.........................336773314 < 32 bit words transmitted
RcvData:.........................248591376 < 32 bit words received
XmtPkts:.........................5507244
RcvPkts:.........................3452658

68
smpquery

smpquery portinfo lid# port#

smpquery switchinfo lid#

69
ibdiagpath

Performing Comprehensive Diagnostic for a route

Ibdiagpath -v -l slid dlid

70
 Displaying a Route Through the fabric
ibtracert slid dlid
where slid is the LID of the source node and dlid is the LID of
the destination node in the fabric.
# ibtracert 15 14
From switch {0x0021283a8389a0a0} portnum 0 lid 15-15
"Sun DCS 36 QDR switch localhost"
[23] -> ca port {0x0003ba000100e38a}[2] lid 14-14 "nsn33-43
HCA-1"
To ca {0x0003ba000100e388} portnum 2 lid 14-14 "nsn33-43
HCA-1"
The route starts at switch with GUID 0x0021283a8389a0a0
and is using port 0.The switch is LID 15 and in the
description, the switch host's name is Sun DCS 36 QDR
switch localhost. The route enters at port 23 of the HCA with
GUID 0x0003ba000100e38a and exits at port 2. The HCA is
LID 14

71
 ibdiagnet

Determining which links are experiencing significant

Errors
ibdiagnet -c 100 -P all=1
In this instance of the ibdiagnet command, 100 test
packets are injected into each link and the -P all=1 option
returns all counters that increment during the test
ibdiagnet -v -r
Note According to the InfiniBand specification 10E-
12 BER, the maximumallowable symbol error rate is
120 errors per hour

72
Controlling The Switch

73
Commands

Controlling the infiniband Fabric

Clearing error counters: ibclearrrors
Clearing data counters: ibclearcounters
Resetting a port: ibportstate lid port reset
Setting port speed(ibportstate lid port speed <value>,
disabling a port, enabling a port

74
 disableswitchport
disableswitchport [--reason=reason] connector|Switch port
reason is the r eas on for d is ablin g the p or t, Blacklist or
Partition
connector is the n u m ber of the Q S F P c on n ec tor ( 0A17B).
port is the n u m ber of the p or t ( 136)
T h i s h a r d w a r e c o m m a n d d i s a b l e s a Q S F P c o n n e c to r
a n d p o r t o n th e s w itc h c h i p . T he c om m an d ad d r es s es either
the c on n ec tor or the p or t on the s w itc h c hip
The --reason option enables you to use a passphrase to lock the
state of the port:
Blacklist A connector and port pair are identified as being
inaccessiblebecause of unreliable operation.
Partition A connector and port pair are identified as being
isolated from the InfiniBand fabric.

75
ibdevreset

ibdevreset
Resets the switch chip
opensm (pid 2117) is running...
Environment daemon running (PID 7356)
Stopping Enviroment daemon
Resetting Switch
Starting Enviroment daemon
Starting IB Subnet Manager

76
managementreset

managementreset
This hardware command resets the management controller, the
CPLD, and the I4 switch chip. The command requests a reboot,
and in most situations, this should bedone. The managementreset
command also forces links to retrain upon management controller
services startup. The -r option bypasses the query for reboot
and reboots automatically.

77
Exadata: Debug Utility

/opt/oracle.SupportTools/ibdiagtools/Infinicheck
Checks network status in current cluster between hosts and
storage cells
Executed on host machine

78
 Debug utility in Exadata(Contd..)

<Host># /opt/oracle.SupportTools/ibdiagtools/verify-topology

[ DB Machine Infiniband Cabling Topology Verification Tool ]

[Version 11.2.1.3.b]

Looking at 1 rack(s).....
Spine switch check: Are any Exadata nodes connected ..............[SUCCESS]
Spine switch check: Any inter spine switch connections............[SUCCESS]
Spine switch check: Correct number of spine-leaf links............[SUCCESS]
Leaf switch check: Inter-leaf link check..........................[SUCCESS]
Leaf switch check: Correct number of leaf-spine connections.......[SUCCESS]
Check if all hosts have 2 CAs to different switches...............[SUCCESS]
Leaf switch check: cardinality and even distribution..............[SUCCESS]

79
Exadata and Exalogic Version Command

/usr/local/bin/imageinfo
Kernel version: 2.6.18-194.3.1.0.3.el5 #1 SMP Tue Aug 31
22:41:13 EDT 2010 x86_64
Image version: 11.2.2.2.0.101206.2
Image activated: 2011-02-09 13:09:23 +0000
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

80
command

Command only in NM2-GW

/usr/local/bin/showsmlog

81
 Subnet Manager is not running

<HOST-MACHINE># ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.7.0
Hardware version: a0
Node GUID: 0x00212800013f0baa
System image GUID: 0x00212800013f0bad
Port 1:
State: Initializing <---------------------- !!
Physical state: LinkUp
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x00212800013f0bab

82
Subnet Manager is not running(Contd..)

Port 2:
State: Initializing <--------------------- !!
Physical state: LinkUp
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x00212800013f0bac

83
NM2-GW: 10Gig Ethernet link issues
(Troubleshooting tips)

Verify whether physical links are up on both NM2-GW

(listlinkup) and Cisco/Brocade/Extreme ports
What cables customer is using?
What transceivers is cu using (specially on
Cisco's/Brocade/Extreme switches end)?
As of today 3rd party switches should be connected to NM2-
GW through vendor specific transceivers
Rule out MTU/pause-frames incompatibility between
NM2-GW & External 10Gig Ethernet Switches by
using "getportstatus 0A-ETH".
If not, set appropriate settings using "setgwethport 0A-ETH".
(Try MTU 9216, RX/TX pause frames as Global)

84
NM2-GW: 10Gig Ethernet link up issues
(Troubleshooting tips) (Contd..)

Is the NM2-GW version 1.1.2 or greater?

Provide snapshot of GW port dialog? (click on the GW
connector in ilom web interface)
Also make sure:
listlinkup shows 0A-ETH-1,2,3 & 4 and 1A-ETH-1,2,3 &4as
"up" and "enabled"
There are no TX/RX errors under "getportcounters 0A-ETH-x"

85
IB Switch Information gathering

version
env_test
listlinkup
/usr/bin/ibdiagnet -skip dup_guids -pm
/usr/sbin/ibcheckerrors -v
sminfo
getmaster -l
perfquery lid# port#
Ibstat , ibstatus
ibnetdiscover

86
IB Switch Information gathering(Contd..)

Switch:
/etc/opensm/opensm.conf
/var/log/opensm.log
/var/log/messages
Snapshot of Fabric Monitor (in ilom web interface)

87
Cisco Switch : Information gathering

show hardware
show interface status
show interface flowcontrol
show interface gigabitethernet 1/(port #)

88
Join email aliases

INFINIBAND_TRIAGE_US@oracle.com
exadata-support@sun.com
exalogic-support@sun.com

89
References

Sun Datacenter Infiniband Switch 36 Command

Reference
Sun Datacenter Infiniband Switch 36 User's Guide
Sun Network QDR Infiniband Gateway Switch
Administration guide
Exalogic
http://download.oracle.com/docs/cd/E18476_01/index.htm

90
Q&A

91