Professional Documents
Culture Documents
# df k
(Checks for disk space usage in 1K blocks)
Filesystem
1024-blocks
Free %Used
Iused %Iused Mounted
on
/dev/hd4
8650752
5687508
35%
39729
2% /
/dev/hd2
5242880
2950972
44%
35227
3% /usr
# df g
(Checks for disk space usage in GigaByte blocks)
Filesystem
GB blocks
Free %Used
Iused %Iused Mounted
on
/dev/hd4
8.25
5.42
35%
39729
2% /
/dev/hd2
5.00
2.81
44%
35227
3% /usr
# df gP (POSIX view with
Filesystem
GB blocks
/dev/hd4
8.25
/dev/hd2
5.00
Note that the (df k or -g) lists the disk usage (%Used) as well as the
inodes usage (%Iused).
Be sure to pay close attention and try not to get the two confused when
checking file system space.
Auto
hdisk0
rootvg
10752MB
45
yes
yes
hdisk1
rootvg
2560MB
45
yes
yes
hdisk2
rootvg
8192MB
45
yes
yes
or
# swap -s
allocated = 5505024 blocks used = 2458677 blocks free = 3046347
blocks
2) Load Average
# uptime
11:14AM
up 10 days,
0.05, 0.03
21:02,
2 users,
*Note: The load average numbers give the average number of jobs/processes in the run queue
over the last 1, 5, and 15 minutes. The lowest possible load average is zero. A load average of
one or two is about typical. The load avg. of 3 and above could indicate a critical issue on the
system.
B. SYSTEM PERFORMANCE
1) CPU and Memory Usage
The vmstat command reports statistics about kernel threads, virtual memory, disks, traps, and
CPU activity.
*us = user time, sy = system time, id = CPU idle time, wa = CPU cycles to determine that the
current process is wait.
# vmstat 5 5
System Configuration: lcpu=8 mem=16384MB
kthr
memory
page
----- ----------- -----------------------r b
avm
fre re pi po fr
sr cy
5 1 4818381 24300
0
2
2 636 859
53 5
6 1 4817085 25591
0
0
0
0
0
23 4
7 1 4811637 31031
0
0
0
0
0
49 8
2 1 4813001 29650
0
0
0
0
0
6
4 1 4818874 23769
0
0
0
0
0
7
faults
cpu
------------ ----------in
sy cs us sy id wa
0 2048 321280 9460 24 18
0 1838 593223 4798 53 21
0 1975 265643 4706 30 13
0 1814 95041 7491
8 10 76
7 81
Disks:
hdisk1
hdisk0
cd0
tin
0.0
tout
0.5
% tm_act
0.6
0.7
0.0
avg-cpu:
Kbps
tps
8.2
7.2
0.0
% user
% sys
0.3
0.2
% idle
99.3
% iowait
0.2
Kb_read
Kb_wrtn
1.2
2030462
5660599
1.1
1116762
5660603ma
0.0
0
0
Note: %user shows the percentage of CPU utilization at the user level and %sys shows the
percentage of the CPU utilization at the system level.
# sar 5 5
AIX jrspa22t 2 5 00283EDD4C00
07/26/06
%usr
22
53
52
52
39
%sys
13
4
9
3
8
%wio
2
1
1
1
2
%idle
64
42
38
44
52
44
48
USER
PID
COMMAND
root
258066
syncd
patrol 7778462
PatrolAg
lAg
root
8769704
bgsagent
root
172116
wlmsched
bsomqp022642060
java
root
7958554
topas
root
1417340
ncmsqp042674916
patrol 6553618
PatrolAg
t
ncmsqp023493972
jrspa22t
PPID PRI NI
1
60 20
DATA
RES
88
Interval:
TEXT PAGE
RES SPACE
1
TIME CPU%
PGFAULTS
I/O OTH
160 1966:11
2.9
75 30 14933
1.1
027708
62 20
6313
835
9:29
1.1
400
16 41
17
17 1090:36
0.5
26:38
0.4
0:01
0.4
202
245
838 1349:10
17 40486
47:57
674 4157 242:53
0.4
0.3
0.2
0
0 seosd
0
0 java
0 1721
0.2
6313
2191530
60 20 15194
11 24748
2969668
58 41
19
1
2822388
1
1 41
400
60 20 25443
70 30 2754
3690670
60 20 17462
2790
2790
11 27864
29:37
25
0 java
3) LAN Status
The netstat command shows network status for each protocol or routing table. The i flag
may be used to determine collisions and I/O errors.
# netstat -i
Name
en0
en0
en2
en2
lo0
0
lo0
0
lo0
6076
Mtu
Network
Address
Ipkts
1500
link#2
0.9.6b.3e.57.61
424536
1500
89.10.12
prl28284
424536
1500
link#3
0.9.6b.ce.54.cb
4297312
1500
55.10.32
breac01t-55
4297312
16896
link#1
0
16896
127
loopback
5254
0
16896
::1
0
0
Ierrs
0
0
0
0
5254
Opkts
239376
239376
140332
140332
0
Oerrs
0
0
2
2
6076
0
0
0
6076
0
5254
Coll
0
# netstat -rn
Routing tables
Destination
Gateway
Flags
Refs
Use
If
0
en0
0
en2
0
138677
en2
0
1
lo0
0
4
en2
1
2163
en0
0
=>
# ifconfig -a
en0:flags=5e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPR
T,64BIT,CHECKSUM_OFFLOAD,PSEG,CHAIN> inet 89.10.12.31 netmask 0xffffff00
broadcast 89.10.12.255
en2:flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPR
T,64BIT,CHECKSUM_OFFLOAD,PSEG,CHAIN>inet 55.10.34.184 netmask 0xfffffc00
broadcast 55.10.35.255
lo0:flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BI
T> inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1/0 tcp_sendspace 65536 tcp_recvspace 65536
Receive Statistics:
------------------Packets: 166277808
Bytes: 38982878854
Interrupts: 153893117
Receive Errors: 0
Packets Dropped: 0
Bad Packets: 0
Alignment Errors: 0
No Resource Errors: 0
Receive Collision Errors: 0
Packet Too Short Errors: 0
Packet Too Long Errors: 0
Packets Discarded by Adapter: 0
Receiver Start Count: 0
General Statistics:
------------------No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
Simplex 64BitSupport ChecksumOffload
PrivateSegment LargeSend DataRateSet
Gigabit Ethernet-SX PCI-X Adapter (14106802) Specific Statistics:
--------------------------------------------------------------------
Link Status : Up
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex
A recent system reboot could explain alarms on the system. The reboot may have been scheduled
or may have been caused by a system panic, hardware failure, or power failure. Further
investigation should be done. Check the CAMCS logs to see if a system panic occurred or check
cron to see if a reboot script was executed.
Check for the systems current run level. Please note that AIX operates at Run Level 2. Other
Run Levels are available, but are rarely used.
who r
To check for any configuration errors after a system reboot, run the following command to
see the bootlog:
# alog o f /var/adm/ras/bootlog | more
The console log can be viewed using this command:
# alog o f /var/adm/ras/conslog | more
1) Hardware Devices
lsdev displays information about devices in the device configuration database.
Flags: -C lists information about a device that is in the Customized Devices object class.
-c specifies a device class name.
-H displays headers above the column output.
To list the disks that are in the Available state in the Customized Devices object class..
# lsdev -CH -c disk
name
status
location
description
hdisk0 Available 1S-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive
status
location
description
L2cache0
aio0
cd0
en0
en1
en2
en3
ent0
(1410ff01)
ent1
(1410ff01)
Available
Defined
Available
Available
Defined
Available
Defined
Available
1G-19-00
1L-08
1c-08
1j-08
1n-08
1L-08
Available 1c-08
L2 Cache
Asynchronous I/O (Legacy)
IDE DVD-ROM Drive
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
Standard Ethernet Network Interface
10/100 Mbps Ethernet PCI Adapter II
10/100 Mbps Ethernet PCI Adapter II
lspv provides information about known physical volumes on the system along with its physical
disk name, physical volume identifier (PVIDs) and volume group.
# lspv
hdisk0
hdisk1
000c8edc02dccea9
000c8edc851ee972
rootvg
rootvg
active
active
# lspv hdisk0
PHYSICAL VOLUME:
hdisk0
VOLUME GROUP:
PV IDENTIFIER:
000c8edc02dccea9 VG IDENTIFIER
000c8edc00004c00000000fc851ef361
PV STATE:
active
STALE PARTITIONS:
0
ALLOCATABLE:
PP SIZE:
64 megabyte(s)
LOGICAL VOLUMES:
TOTAL PPs:
542 (34688 megabytes)
VG DESCRIPTORS:
FREE PPs:
86 (5504 megabytes)
HOT SPARE:
USED PPs:
456 (29184 megabytes)
FREE DISTRIBUTION: 25..60..00..00..01
USED DISTRIBUTION: 84..48..108..108..108
rootvg
yes
7
1
no
The p flag will list all physical partitions of physical volume hdisk0.
# lspv -p hdisk0
hdisk0:
PP RANGE STATE
REGION
LV NAME
1-4
used
outer edge
N/A
5-29
free
outer edge
30-109
used
outer edge
110-141
used
outer middle
142-201
free
outer middle
202-217
used
outer middle
218-221
used
center
N/A
222-325
used
center
/
326-381
used
inner middle
382-433
used
inner middle
/usr
434-541
used
inner edge
/usr
TYPE
hd5
MOUNT POINT
boot
hd9var
hd6
jfs
paging
hd3
jfs
hd8
hd4
jfslog
hd4
hd2
hd2
/var
N/A
/tmp
jfs
jfs
jfs
jfs
542-542
free
inner edge
PHYSICAL VOLUME:
hdisk0
VOLUME GROUP:
PV IDENTIFIER:
000c8edc001363a5 VG IDENTIFIER
000c8edc00004c00000000fc851ef361
PV STATE:
active
STALE PARTITIONS:
6
ALLOCATABLE:
yes
Note Stale Partitions Disk is BAD.
rootvg
PP SIZE:
TOTAL PPs:
FREE PPs:
USED PPs:
FREE DISTRIBUTION:
USED DISTRIBUTION:
(84+48+108+108+108)
7
1
no
64 megabyte(s)
542 (34688 megabytes)
86 (5504 megabytes)
456 (29184 megabytes)
25..60..00..00..01
LOGICAL VOLUMES:
VG DESCRIPTORS:
HOT SPARE:
84..48..108..108..108
# lspv -p hdisk0
hdisk0:
PP RANGE
1-4
N/A
5-29
30-109
110-141
N/A
142-201
202-217
218-218
N/A
219-221
222-222
/
223-231
/
232-232
/
233-240
/
241-241
/
242-325
/
326-381
382-382
383-400
401-401
402-433
434-541
542-542
STATE
REGION
used
used
used
free
used
*stale
used
*stale
used
*stale
used
*stale
used
used
*stale
used
*stale
used
used
free
LV NAME
TYPE
outer edge
MOUNT POINT
hd5
boot
free
outer edge
outer edge
hd9var
outer middle
hd6
jfs
paging
/var
outer middle
outer middle
center
center
center
center
hd3
jfs
jfslog
hd8
jfslog
jfs
hd4
hd4
center
jfs
hd4
center
center
hd4
hd2
hd2
hd2
hd2
hd2
N/A
jfs
hd4
center
inner middle
inner middle
inner middle
inner middle
inner middle
inner edge
inner edge
/tmp
hd8
jfs
hd4
jfs
hd4
jfs
jfs
jfs
jfs
jfs
jfs
jfs
/
/usr
/usr
/usr
/usr
/usr
2) Volume Groups
To list volume groups that are currently active on your system, type:
lsvg -o
# lsvg -o
rootvg
List detailed information and status about the volume group.
# lsvg rootvg
VOLUME GROUP:
rootvg
000c8edc00004c00000000fc851ef361
VG STATE:
active
VG PERMISSION: read/write
megabytes)
MAX LVs:
256
LVs:
9
OPEN LVs:
8
TOTAL PVs:
2
STALE PVs:
0
ACTIVE PVs:
2
MAX PPs per PV: 1016
LTG size:
128 kilobyte(s)
HOT SPARE:
no
VG IDENTIFIER:
PP SIZE:
TOTAL PPs:
64 megabyte(s)
1084 (69376
FREE PPs:
USED PPs:
QUORUM:
VG DESCRIPTORS:
STALE PPs:
AUTO ON:
MAX PVs:
AUTO SYNC:
BB POLICY:
LPs
PPs
PVs
LV STATE
1
42
1
33
20
2
84
2
66
40
2
3
2
2
2
closed/syncd
open/syncd
open/syncd
open/syncd
open/syncd
20
40
open/syncd
open/syncd
open/syncd
72
144
open/syncd
18
18
42
18
18
84
1
1
2
open/syncd
open/syncd
open/syncd
N/A
N/A
N/A
/
N/A
N/A
N/A
# lsvg -p rootvg
rootvg:
PV_NAME
hdisk2
hdisk3
hdisk0
hdisk1
PV STATE
active
active
active
active
TOTAL PPs
135
135
135
135
FREE PPs
5
0
6
21
FREE DISTRIBUTION
01..00..00..00..04
00..00..00..00..00
00..00..00..00..06
00..00..10..00..11
PCM/friend/scsiscsd
fail_over
0
50
0
nonactive
0x40000
00283edd26fdf5680000000000000000
3
single_path
36400
False
True
True
True
True
True
True
False
False
True
False
E. Running SNAP
Note: You must have an open PMR with pSeries Support (IBM) before continuing. All
references to the PMR number below will be in the format of xxxxx.YYY where xxxx
is the problem number and YYY is the branch number.
1) Call IBM
Search the report for General Info and view the HW_MODEL field.
GENERAL INFO
Next Section
Previous Section
====================================================================
GENERAL INFO: senthil : 0x590a0c1f : Fri 03-04-11 14:04:31 CST : 80.1
====================================================================
HOSTNAME: senthil
HOSTID: 0x590a0c1f
PRIM_IP_ADDRESS: x.x.x.x
HW_VENDOR: IBM
HW_MODEL: IBM,7029-6C3
OS_LEVEL: AIX 5.2
SYSTEM_MEMORY: 2048 Mb
DDSABLE: TRUE
DOMAIN: none
Follow the steps below to run snap and ftp the output to IBM:
To gather the basic information on a machine like error logs configuration, AIX driver
levels, run
# snap -r
(this removes any prior snap data)
# snap -gc
NOTE: Depending on the amount of SSA drives this could last anywhere from a few minutes to
2 hours, so be careful.
To gather the SSA info, use: # snap -gbc
To gather the SSA and TCPIP info, use: # snap gtbc
To gather all system configuration information: # snap ac
Example of output:
bos62833[root]: snap -r
Nothing to clean up
bos62833[root]: snap -gbc
Checking space requirement for general
information...........................................................
......................................................................
......................................................................
......................................................................
......................................................................
....... done.
..Checking space requirement for ssa information.......... done.
Checking for enough free space in filesystem... done.
********Checking and initializing directory structure
Creating /tmp/ibmsupt directory tree... done.
Creating /tmp/ibmsupt/ssa directory tree... done.
Creating /tmp/ibmsupt/general directory tree... done.
Creating /tmp/ibmsupt/general/diagnostics directory tree... done.
Creating /tmp/ibmsupt/testcase directory tree... done.
Creating /tmp/ibmsupt/other directory tree... done.
********Finished setting up directory /tmp/ibmsupt
Gathering general system
information...........................................................
......................................................................
......................................................................
......................................................................
......................................................................
....... done.
Gathering scanout information..done.
Gathering ssa system information.......... done.
Creating compressed pax file...
Starting pax/compress process... Please wait... done.
-rw-------
1 0
834911 Feb
8 00:08 snap.pax.Z
IBM support may request additional options to be executed with the snap command. From
man snap, these are the different Flags:
-a Gathers all system configuration information. This option requires
approximately 8MB of temporary disk space.
-A Gathers asynchronous (TTY) information.
-b Gathers SSA information.
5.2
Identify the dump space settings. Note that the dump will only
write to the primary or secondary and will not span to the
secondary if the primary fills:
# sysdumpdev l
primary
secondary
copy directory
forced copy flag
always allow dump
dump compression
/dev/hd7
/dev/hd71
/var/adm/ras
TRUE
TRUE
OFF
Display statistical info about the most
recent dump:
# sysdumpdev -L
0453-039
Device name:
/dev/hd7
Major device number: 10
Minor device number: 8
Size:
23327232 bytes
Uncompressed Size:
191149876 bytes
Date/Time:
Fri Feb 11 10:50:40 CST 2005
Dump status:
0
dump completed successfully
Estimates the size of the dump (in bytes) for the current running
system:
# sysdumpdev e
0453-041 Estimated dump size in bytes: 4280287232
snap gfkDNd
cd /<file system>/ibmsupt/dump
ls (Ensure that unix.Z, dump.snap, and dump.Z are present.)
snap cd /<file system>/ibmsupt
This will create a snap.pax.Z file in the /tmp/ibmsupt directory. The file will need to be renamed
to pmr#.branch#.snap.pax.Z.
# mv snap.pax.Z <pmr#.branch#.snap.pax.Z>
F. SHUTDOWN
The shutdown command halts the operating system. Only a user with root user authority can run
this command. Do not attempt to restart the system or turn off the system before the
shutdown completion message is displayed; otherwise, file system damage can result.
Make sure you are on the correct server prior to entering shutdown command:
Enter: hostname
To shutdown and restart the system:
# shutdown Fr
Other flags that could be used with the shutdown command are:
- h Halts the operating system completely.
-m Brings the system down to maintenance (single user) mode.
-d Brings the system down from a distributed mode to a multiuser mode.
-i Interactive mode. Displays interactive messages to guide the user through the shutdown.
The last command can be used to help determine when the system was last shut down.
# last
shutdown
shutdown
shutdown
shutdown
tty0
tty0
pts/1
Feb 11 14:05
Feb 10 20:23
Feb 04 07:08
G. HARDWARE ASSISTANCE
How to run Diagnostics
The diag command is menu driven and is used to run diagnostics for a suspected problem.
# diag
Press <Enter> to advance past the information screen.
Select Diagnostic Routines.
Select Problem Determination.
This instructs the diag command to test the system and analyze the error log.
You may run a diagnosis on a particular device by using the d flag.
H. LOGS
The first place you should go when troubleshooting problems in AIX is the error report
(errpt).
First run errpt without any options to get an overview of current errors:
# errpt|more
IDENTIFIER TIMESTAMP T C RESOURCE_NAME
B6048838
0725140606 P S SYSPROC
TERMINATED
DESCRIPTION
SOFTWARE PROGRAM ABNORMALLY
B6048838
TERMINATED
B6048838
TERMINATED
B6048838
TERMINATED
B6048838
TERMINATED
B6267342
B6267342
B6267342
B6267342
0725133506 P S SYSPROC
0725122506 P S SYSPROC
0724140106 P S SYSPROC
0721033906 P S SYSPROC
0721032506
0721032506
0721032506
0721032506
DISK
DISK
DISK
DISK
P
P
P
P
H
H
H
H
hdisk1356
hdisk1356
hdisk1355
hdisk1355
OPERATION
OPERATION
OPERATION
OPERATION
ERROR
ERROR
ERROR
ERROR
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
Recommended Actions
CORRECT THEN RETRY
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
6
USER'S PROCESS ID:
7540756
FILE SYSTEM SERIAL NUMBER
44
INODE NUMBER
1474687
PROCESSOR ID
16
CORE FILE NAME
/pac/brsmdp07/bea/app/user_projects/domains/collections/core
PROGRAM NAME
java
ADDITIONAL INFORMATION
abort E8
??
Symptom Data
REPORTABLE
You can display errors that were encountered during the last day
by specifying a date in your search.
# date
Wed Feb 23 14:57:39 CST 2005
# errpt -a -s 0222145601 |more
-a display information in a detailed format
-s
display all records posted after the StartDate
Example: errpt -a -s (mmddhhmmyy)
year minus 24 hours