You are on page 1of 63

IBM Software Group

RDz Workbench Integration with Fault Analyzer


Jon Sayles/IBM Rational Eco System Team

2008 IBM Corporation

IBM Trademarks and Copyrights


Copyright IBM Corporation 2007,2008, 2009. All rights reserved.
The information contained in these materials is provided for informational purposes only, and is
provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for
any damages arising out of the use of, or otherwise related to, these materials. Nothing
contained in these materials is intended to, nor shall have the effect of, creating any warranties
or representations from IBM or its suppliers or licensors, or altering the terms and conditions of
the applicable license agreement governing the use of IBM software. References in these
materials to IBM products, programs, or services do not imply that they will be available in all
countries in which IBM operates.
This information is based on current IBM product plans and strategy, which are subject to change
by IBM without notice. Product release dates and/or capabilities referenced in these materials
may change at any time at IBMs sole discretion based on market opportunities or other factors,
and are not intended to be a commitment to future product or feature availability in any way.
IBM, the IBM logo, the on-demand business logo, Rational, the Rational logo, and other IBM
Rational products and services are trademarks or registered trademarks of the International
Business Machines Corporation, in the United States, other countries or both. Other company,
product, or service names may be trademarks or service marks of others.

Course Contributing Authors


Thanks to the following individuals, for assisting
with this course:
Russ Courtney/IBM
Reginaldo Barosa/IBM-Rational
David Bean/IBM-Rational
Ed Steele/IBM-Rational
Dave Willsey/IBM-Rational

Course Overview
Audience
This course is designed for application developers who have learned or
programmed in COBOL, and who need to do z/OS Traditional Development and
Maintenance as well as build leading-edge applications using COBOL and
Rational Developer for System z.

Prerequisites
This course assumes that the student has a basic understanding and knowledge
of software computing technologies, and general data processing terms,
concepts and vocabulary, as well as a working knowledge of COBOL and z/OS.
Knowledge of SQL (Structured Query Language) is assumed for database
access is assumed as well.
Basic PC and mouse-driven development skills, terms and concepts are also
assumed.

Course Topics
Course Name: Rational Developer for System z Foundation Training
Course Description: Learn how to use Rational Developer for System z to do z/OS traditional
development, maintenance, support and for Enterprise Modernization of z/OS applications
Pre-requisites: Some experience developing COBOL applications using z/OS is expected. A working
knowledge of SQL is also recommended.
Course Length: ~5days or if done in self-paced mode, at your own pace
Topics (Agenda)

Getting Started - installing and configuring RDz - and the course materials, and using Eclipse
The RDz Workbench
Code analysis tools
Editing
Compiling programs
Debugging local COBOL programs
The Data Perspective:
Working with relational data sources
Modifying test data
Editing and testing SQL statements
Working with remote system resources:
Connecting to a mainframe
Data management
Accessing and editing files
z/OS Application Development
Creating MVS Subprojects
Creating and customizing project properties
Debugging z/OS Applications
Debugging Batch Applications
Setting Debug Tool for Online Applications
Working with File Manager
Creating test data
Editing complex file-types
Working with mainframe ABENDs using Fault Analyzer
Creating Fault History views
Analyzing and solving mainframe ABENDs
Creating and modifying BMS Maps using the BMS Map Editor 5

UNIT

The RDz Workbench


Topics:

Analyzing Mainframe Abends


ABEND Codes and Reasons
Using IBM's Fault Analyzer
Appendix

Unit objectives
After completing this unit on Production Support/Application Testing/Software Defect
and IBM Mainframe COBOL ABEND Research, you should be able to:
Define the steps in a generalized methodology of ABEND resolution
List the various sources of ABEND inputs, including:
PD Tools documents
Other sysout
Dynamic trace facilities
List the common types of COBOL program ABENDS

ABEND Overview
When an application ABEND (ABnormal END-of-job) occurs, z/OS stops executing
your program, closes files and buffers and generates a single high-level message in
the form of a System Completion Code (Sxxx).
The System Completion Code is usually written to an output listing file through
your //SYSOUT DD * JCL entry.
This completion code indicates why the system has decided to stop executing your
application.
It is related to, but often only loosely related to what is really wrong with your
application
Because of this the System Completion Code represents the starting point for your
analysis of the problem.
She won't be laughing
when she gets back to
her desk and finds out
that last night's
production jobs blew
sky high!

Debugging Assistance
Along with the System Completion Code, using IBMs Problem Determination tools (PD
Tools) Fault Analyzer you can obtain various reports which describe What/Where and
How the ABEND occurred.
Valuable information contained in the Fault Analyzer report-files includes:
The System Completion Code (and often a short text description of what it designates)
A short explanation of the cause of the ABEND
The COBOL instruction (statement) or line number, which contained the invalid operation causing
z/OS to halt execution
Variables of interest and code surrounding the instruction that halted execution
A "core-dump" (a hexadecimal printout) of the internal machine storage and registers relevant to
the areas of your program surrounding the COBOL instruction which caused z/OS to halt
execution.

This information is critical to begin understanding and researching the problem, but it is
sometimes insufficient to solve the underlying application problem, which could be any
combination of:

Incomplete, incorrect or invalid COBOL procedural logic


A typo such as a misplaced period, or incorrectly specified field
Incorrect or invalid input data
Batch jobs run out of sequence
Input files missing or corrupted (hardware errors)
Errors which relate to JCL problems
etc.
9

Getting Started
There are as many different ways to analyze and research COBOL ABENDs as there are
individual approaches to writing procedural logic. However, if you've never done
software production support work, consider starting with the following structured
problem-solving approach:
1.
2.
3.
4.
5.

Preparation
Research
Hypothesis
Solution
Resolution

As a final note before we begin discussing the above, understand that there are often two
distinct phases of application Production Support:
1. Data Center on-call ABEND resolution - wherein a technician receives notification that a job
or transaction has ABENDd and must be "fixed" within an extremely short timeframe (usually
minutes to hours). In this case, the technician's main concern is to "patch" the problem - get
the system back online, or get the batch job-stream back into production ("Patch-It").
2. NextDay problem resolution - wherein technician(s) actually track down and solve the
problem that caused the ABEND ("Fix-It").

The steps that follow represent a structured approach to "FixIt" in that they go well
beyond the scope of the emergency measures used to "patch" the problem during
an OnCall emergency.

10

1. Preparation and Information Gathering 1 of 2


Collect all necessary background information on the ABEND. WHAT
happened, WHERE the ABEND occurred and HOW the ABEND occurred:
\

Start with Fault


Analyzer's reports
as they contain a
broad and deep set of
formatted, structured,
analysis information
Collect additional
supporting ABEND
output (SYSOUT)
from the job,
DISPLAY statements,
etc.)
Obtain copies of the
run-time:

JCL
Program source
All copybooks (or
expanded source
listings)

11

1. Preparation and Information Gathering 2 of 2


If batch, from the JCL learn the dataset names of input and output files
accessed by the program (which you may need to browse as part of your research)

RAA Batch Job Diagram


RAA Batch Job Diagram
showing datasets
showing datasets
being passed from step to step
being passed from step to step

Learn the nature of the batch job from system documentation, or from an application business
expert (at least at the level of module-flow and file-access)
12

Research Analyze the Gathered ABEND Data

From the preparation step, construct a mental map (understanding) of the program's
execution (HOW the ABEND occurred - which will start you down the path of research
to understand "WHY" the ABEND occurred

WHY determination usually requires a combination of "Static" and "Dynamic" analysis complementary research and investigative approaches.

These steps need not be followed in this order. Rather, in time you will develop an
"intuition" as to which kind(s) of analysis will be most likely to provide the information
you need to solve your problem.

To assist, you can use application research and analysis tools such as

IBMs Rational Asset Analyzer,


Analyzer Rational Developer for System z
IBM's Debug Tool or CICS I.A. Dynamic Analysis

Static Analysis

So if the reason "why" the ABEND occurred is not apparent at this point, perform
Static and/or Dynamic Analysis on the specific areas of the application relating to the
ABEND.

Note: Recompiling old COB2 programs with Enterprise COBOL can create many of these
problems, even though "Nothing has changed in the program"

13

Static Analysis Overview of Techniques


1. Structural Visualization: is the generation of an accurate mental map, understanding or mental image of the
program's control structure, or logic-architecture. Using the starting point represented by the ABEND condition
(the statement which caused z/OS to halt execution) and using electronic-assisted tools (such as IBMs Rational
Asset Analyzer or Rational Developer for System z), build an accurate understanding of the code invocation at:
The module/file level (System View) - Paragraph/Section level (Hierarchy chart) - Statement level (Flow chart)

Structural Visualization can done be "top-down", by asking open-ended questions; such as learning how a
particular routine "hangs-together logically", or it can be used "bottom-up", by asking specific close-ended
questions about a program, such as "How does this particular paragraph get executed?" "How did this module
get invoked?"
2. Data Flow Analysis: A combination of control structure analysis and data item analysis, which seeks to
determine the usage of particular fields throughout a program. Data flow analysis is used to determine (from a
given instance of a data item) where the next occurrences of that item exist in your program, and how the data
item is used; (as a receiving field in a MOVE or mathematical operation, as the sending field in a MOVE
statement, as part of a logic-branch (IF, PERFORM UNTIL/VARYING, etc.).
3. Data Impact Analysis: An expansion of Data Flow Analysis which traces the movement of data from field-tofield throughout a program, or throughout an entire application; including I/O (screens and files). Using Data
Impact Analysis, you can identify all fields that might have had an impact on the contents of a field (before the
ABEND occurred). And just as importantly - you can learn the affect changing this field will have on the behavior
of the application.
4. Textual or Data Item Usage: Utilized more for application maintenance and enhancement requests, this type of
Static Analysis involves searching for "categories" of program-items, such as "List all fields that contain *JUL*,
*GREG*, *YR*, *YEAR* (suspect date candidates for Year2000 conversion), or list all such fields with two digits
(numeric) or two-byte (alphanumeric) definitions.
5. Code Partitioning: Again, utilized more for application maintenance, enhancements and application
reengineering, Code Partitioning involves mentally organizing and analyzing code by function or process, such
that you understand and can distinguish the usage of code by business process. For example: Find all code
that relates to the calculation of premium renewal payments or Isolate the code that edits a particular file,
with an eye towards creating a shared subroutine from the code.

14

Dynamic Analysis Overview of Techniques


1. Tracing: Source-level interactive debugging. Watch the program execute statement-bystatement, and line-by-line. This is very useful for detailed-debugging, particularly of dense or
complex instructions. Some software (for example, the Rational Developer for System z)
allows you to trace the program logic, attempting to re-create the sequence of events (COBOL
statements) that transpired up to and including the ABEND condition. Tracing is an invaluable
method for detailed debugging. However, given the size and scope of production applications,
it is generally more practical to trace specific problem areas of a program.
2. Interactive Execution: Execute (run) a program, stopping at selective Breakpoints (Pause
execution each time a certain field-value changes, or when a value exceeds some threshold),
and examining the contents (value) of specific fields. Interactive Execution must be done by
(or with) an application analyst who understands how the system is supposed to operate.
Interactive Execution is useful for observing control flow, and is often combined with line-by-line
tracing by setting selective breakpoints, monitoring values, "running" the application to the
breakpoints, and then tracing the code line-by-line.
3. Selective Data State Collection: Execute code and establish a functional summary of specific
data states that it creates. Use these states in subsequent test runs to compare results of
current values to expected values.
4. Coverage: Analyze the number of times each COBOL statement is executed for a given run.
Note that PD Tools/Debug Tool can run a report that shows code coverage. This technique is
extremely useful for analyzing test data coverage of a given application. And it can be used
effectively for debugging if it makes apparent problems such as infinite loops (S222, S322 and
B37 ABENDs), over-loading tables - (loading tables beyond the maximum OCCURS clause
and overlaying storage, which cause things like: S0C1, S0C4, and S0C7 ABENDs).

15

Hypothesis - Determine WHY the ABEND occurred 1 of 3


Your preparation and research is probably all you need to be able to
describe WHAT, WHERE and HOW the ABEND occurred
At what point in the program the logic failed, and what sequence of
COBOL statements caused the failure).
However, before modifying any logic, you must determine WHY these
statements (or sequence of events) caused this particular failure
"Why did this production input file contain spaces in a numeric field?"
"Why did the program's logic perform the Initialization routine twice?"
"Why did the Read routine execute past end-of-file?"
etc.
Only through a determination of WHY will you be able to make a change to
production business logic safely, and with confidence that;
Your change will resolve the ABEND
Your change will not introduce new (additional) ABENDs

16

Hypothesis - Determine WHY the ABEND occurred 2 of 3


Sometimes it is relatively easy to come to an understanding of WHY certain ABEND conditions
occurred. For example, perhaps a period was left off the appropriate termination point for an IF
statement - which caused execution to perform an operation out of sequence. Or perhaps an
IF .. NUMERIC test (which should have been coded for all numeric fields in a file) was
forgotten. Or a paragraph was performed through the wrong paragraph-exit, or a production
job was released before certain files were available (causing I/O errors). These types of
ABEND situations can be understood (and usually resolved) fairly quickly. But, not always.
What if - in the case of the IF statement with the incorrect termination point - the logic that has been
coded, correctly processed the first 100,000 records in the file?
Making a change to a critical IF condition could very well affect other down-stream processing within the
program, wrecking havoc with subsequent routines.

Or what if - in the case of the file containing blanks in the numeric fields - the input file was supposed to
be "clean" (validated) by this point in the job-stream - having gone through allegedly "exhaustive" edits
in prior modules.
By simply adding an IF test you may solve your program's specific ABEND, but you will not have resolved
the actual problem - which exists somewhere else in the system.
In other words, localized/piecemeal approaches to resolving production ABENDs are not recommended - as
they usually change the problem, instead of solving it. And sometimes they just spawn new problems.

It should be noted that, a clear understanding of the


business functionality automated
by this process is usually
required to resolve WHY something has
gone wrong.
Calling on business experts or "application/business" experts
who understand "the big
picture" - and the context in which
the job executes is the rule rather than the exception to
this process.

17

Hypothesis - Determine WHY the ABEND occurred 3 of 3


Developing a clear and accurate determination of WHY a problem that lead
to an ABEND condition exists may take a considerable amount of time,
depending on the:
Size, complexity and structure of the code
Your familiarity with the program's business purpose - coupled with your ability to
grasp the point of each statement (assuming you didn't write the code)
Type of ABEND and reason for the problem (some are more diabolical than
others)
Size of the input/output files, and capabilities of your file editor

Note that, in addition to an understanding of the reason for the ABEND, the
results of your investigation should produce an understanding of the
solution to the problem (the fix itself).

18

Solution - Fix the Problem and Test Your Solution


Take the appropriate action to resolve any business - or system-wide
issues.

Depending on how extensive the damage caused by the problem,


or for how long any problems have persisted undetected:

Files may have to be restored from backups from a previous point-in-time


Jobs may have to be re-run from a previous point-in-time (synchronized with file generations)
Files may have to be modified with "one-shot" programs, written to resolve issues that require
"surgery" on the data

Take the appropriate action to fix the technical (coding) problem:

Edit program source - modifying the existing production


logic and/or
Modify the JCL (if the error included JCL issues)
You may have to edit files using File Manager

Test your solution:

Compile and Link the new version of the application


Create an "image copy" of the production file system, in
order to test your fix
Re-Run the batch job and analyze results
Run "Regression Tests" against the new code analyze
for unexpected results

19

Resolution

Build in the test environment


Migrate changes to back in to production
Promote your changes to production
Schedule and re-run the cycle

Batch
Job

http://en.wikipedia.org/wiki/IBM_Rational_ClearCase

20

Unit objectives
After having completed this unit on Production Support/Application Testing/Software
Defect and IBM Mainframe COBOL ABEND Research, you should now be able to:
Define the steps in a generalized methodology of ABEND resolution
List the various sources of ABEND inputs, including:
PD Tools documents
Fault Analyzer reports
Other SYSOUT
Static analysis
Dynamic trace facilities
List the common types of COBOL program ABENDS

21

UNIT

The RDz Workbench


Topics:

Analyzing Mainframe Abends


ABEND Codes and Reasons
Using IBM's Fault Analyzer
Appendix

22

ABEND Completion Codes and some typical causes


While there is a wide variety of reasons for ABEND conditions ("WHYs")
WHY in
production systems, it is possible (and useful) to categorize and organize
HOW certain conditions often lead to certain types of ABEND completion
codes - in order to expedite or streamline your analysis and research (an
80/20 approach to analysis).
The following information on a few common z/OS ABEND completion
codes, and the conditions which generated them is included for you to
make effective use of PD Tools/Fault Analyzer listings and the above
debugging, research and analysis process.
Notes:
This information is available to some degree within the RDz product in
the Lookup View
There is another source of ABEND/Debug information you might want to
take a peek at:
http://www.jsayles.com/ibm/cobol/ABEND.htm
http://it.geocities.com/aminardi/computer/CODES.html

23

S001: Record Length/Block Size Discrepancy


Reason(s)
S001-0: Conflict between record length specifications (program vs. JCL vs. dataset label)
S001-2: Damaged storage media or hardware error
S001-3: Fatal QSAM error
S001-4: Conflict between Block specifications (program vs. JCL)
S001-5: Attempt to read past end-of-file
Instructions:
OPEN, CLOSE, READ, WRITE
Frequent Coding Causes:
S001-0: Typos in FD or JCL
S001-2: Corrupt disk or tape dataset
S001-3: Internal z/OS problem
S001-4: Forgot to code BLOCK CONTAINS 0 RECORDS in FD (default Block is 1)
S001-5: Logic error (either forgot to close file, or end-of-file-switch not set, overwritten or ignored)
Tools to debug/RDz equivalent return codes:
S001-0: Cannot occur on RDz with Local ASCII/Windows (Line Sequential) files
S001-2: Norton Utilities if on Workstation/COBOL application
S001-4: Cannot occur on Workstation/COBOL (no blocking for Line Sequential files)
S001-5: Logic error: Use RDz's Perform Hierarchy or RAA's Program Flow Diagram to detect
Dynamic:
S001-0: During Debug set a Watch Monitor on the 01 record
S001-2: Need to have PC/IT technician investigate (may need to reformat disk)
S001-4: Always code BLOCK CONTAINS 0

24

S013: Conflicting DCB Parameters


Reason(s)
S013-10: Dummy data set needs buffer space; specify BLKSIZE in JCL
S013-14: DD statement must specify a PDS
S013-18: PDS member not found
S013-1C: I/O error search PDS directory
S013-20: Block size is not a multiple of the LRECL
S013-34: LRECL is incorrect
S013-50: Tried to open a printer for Input of I/O
S013-60: Block size not equal to LRECL for unblocked file
S013-64: Attempted to Dummy out indexed or relative file
S013-68: Block size > 32K
S013-A4: SYSIN or SYSOUT not QSAM file
S013-A8: Invalid RECFM for SYSIN/SYSOUT
S013-D0: Attempted to define PDS with RECFM FBS or FS
S013-E4: Attempted to concatenate > 16 PDSs

Instructions:
OPEN, CLOSE, READ, WRITE
Frequent Coding Causes:
Most of these ABENDs occur running und z/OS (some may not even occur under z/OS, although older modules running
OSVS or VS COBOL II code that have not been recompiled can produce them).
Most are due JCL/COBOL
FD inconsistencies.
Tools to debug:
Static
S013-18: Open multiple windows on RAA Batch Job Diagram and program Environment Division - SELECT
ASSIGN.
25

SOC1: Invalid Instruction


Reason(s)
- SYSOUT DD statement missing
- The value in an AFTER ADVANCING clause is < 0 or > 99
- And Index or Subscript is out of range
- An I/O verb was issued against an unopened dataset
Instructions:
OPEN, CLOSE, READ, WRITE,
WRITE Table handling routines
Note also that during Debug SYSOUT-DISPLAYs are written to the "console", recall how to view

Frequent Coding Causes:


- Incorrect logic in setting AFTER ADVANCING variable (or failure to understand 0-99 limits)
- Incorrect logic in table handling code, or number of table entries has outgrown PIC of variable (e.g. PIC 99, but
100 entries)
Tools to debug:
Static
SYSOUT problem: Open multiple windows on RAA Batch Job Diagram and program Environment

Division - SELECT ASSIGN.

In RAA: Double-click on GO TO verb, or PERFORM chain, or paragraph name.


In RDz: Select Paragraph name/Perform chain and select: Open Declaration
Dynamic:
Set Watch Breakpoint and Monitor on table index or AFTER ADVANCING variable.
Set conditional advanced break point on subscript (i.e. SUB<100).
26

S0C4: Protection Exception


Reason(s)
The program is attempting to access a memory address that is not within the applications z/OS Address Space

Frequent Coding Causes:


- JCL DD statement is missing or incorrectly coded
- Incorrect logic in table handling code (referencing a table subscript < 1 or > max-table-size),
- Number of table entries has outgrown PIC of variable (i.e. PIC 99, but 100 entries).
- In IMS/TM systems, an MFS LL (length) field value is smaller than the actual input MSG length.
Tools to debug:
Static
- DD statement problem: Open multiple windows on RAA Batch Job Diagram and program Environment
Division - SELECT ASSIGN
- IMS LL problem: Analyze through multiple Edit Windows (same solution as DD).
- Incorrect linkage problem:
- Open multiple windows on CALLing and CALLed programs - verify linkage declarations.
Dynamic
Incorrect linkage problem:
- Set Breakpoint and Monitor on linkage declarations.
- Set conditional advanced break point on subscript (i.e. IDX < 100).
RAA - Utilize Run-Unit View to view all modules in Debug run-stream (both active and inactive).
Incorrect logic
In RDz/Debug - set a conditional break point on subscript (i.e. IDX < 100).

27

S0C7: Data Exception


Reason:
Machine instruction expecting numeric data found invalid data
Instructions:
Arithmetic, IF-THEN-ELSE, MOVE (if receiving field is numeric - )
Note: RDz will also S0C7 if sending field is numeric and contains non-numeric (MOVE pic9field TO picXfield)
Frequent Coding Causes:
- Incorrectly initialized, or uninitialized variable
- Missing or incorrect data edit
- 01 to 01 level MOVE if sending field is shorter than receiving field
- Move of Zeros to Group-level numeric fields
- MOVE CORRESPONDING incorrect
- or - MOVE field1 to field2 incorrect assignment
Tools to debug:
Static
RAA report with options data selector on MOD or ALL
Dynamic
Set Watch points and Monitor on field.
Run through to S0C7.
Locate the field definition, or use CSI report.
Solutions:
Add edit checks for all numeric fields and MOVE statements.

28

S0CB: Divide by zero


Reason:
CPU attempted to divide a number by 0.
Instructions:
DIVIDE, COMPUTE

Frequent Coding Causes:


- Incorrectly initialized, or un-initialized variable
- Missing or incorrect data edits (i.e. failed to check divisor for zero value)
Tools to debug:
Static
RAA report on all DIVIDE and COMPUTE instructions or using RDz double-click on these verbs and select Filter
from the Context Menu

Dynamic
Run through to the S0CB
Locate to field definitions of the offending fields

Solution:
Add edit to check for zero divide:
IF divisor > ZERO
THEN
COMPUTE ...
ELSE
PERFORM error-processing routine

Add ON SIZE ERROR to all arithmetic verbs.


29

S222/S322: Timeout/Endless loop


Reason:
Timeout due to program logic caught in "loop" through instruction set with no exit.
Note: RDz Workstation binaries (Run-Time System) does not honor the concept of timeout.
Frequent Coding Causes:
- Invalid logic or fall-through logic
- Invalid end-of-file logic
- End-of-file switch overlaid
- Subscript not large enough
- Perform Thru wrong Exit
- PERFORM UNTIL "End-Of-File", but not performing "READ" routine to reach EOF condition

Tools to debug:
Static
RAA screens on logic in PERFORM chain
RAA Display Perform Thru
Dynamic
PD Tools (mainframe) Debug to S222
Analyze counts (color)
Query and Monitor on subscript
Set an Advanced Break Point - Conditional on count
Solution:
From within Debug, use RAAi to identify logic which could cause looping.
Select and click on PERFORM THRU, PERFORM UNTIL, GO TO.
Place break points on potential error lines.

30

S806: Module Not Found


Reason:
CALL made to program which could not be located along normal search path
(STEPLIB top-to-bottom, JOBLIB top-to-bottom, LINKPACK)
Instructions:
CALL, or JCL EXEC PGM=XXXX
Frequent Coding Causes:
- Module deleted from library, or never compiled to library
- Module name spelled incorrectly
- STEPLIB does not contain load library with module
- I/O error occurred while z/OS searched the directory of the library

Tools to debug:
Static
Do RSE search on module name.
Dynamic
Set Program Advanced Break Point (Entry) to set program break before entry to system.
Solution:
Spell name correctly

31

B37/D37/E37 dataset space exceeded


ABENDS - B37/D37/E37 (RTS-028)
B37:
Disk volume out of space.
D37:
Primary space exceeded, no secondary extents defined.
E37:
Primary and secondary extents full. In TSO, PDS directory needs compress.
E37-04: Disk volume table of contents (VTOC) is full.
Reason:
MVS could not find space for output WRITE.
Note: RDz Local COBOL Run-Time System does not honor the concept of Primary/Secondary space "Out of space"
occur until disk actually fills up

condition will not

Instructions:
WRITE
Frequent Coding Causes:
- Not enough space initially allocated to output file(s).
- (more likely) Logic error - program in (infinite) loop writing output file(s) - see S222/S322 reasons.

Tools to debug:
Static - if (unlikely as this may be) you're debugging locally (on your Workstation) you can find the out-of-space file and statistics on it as follows:.
Do directory list on the file you are writing to:
Go to DOS Window
Type "DIR fname.ext"
On the host the JCL will show the DDNAME and z/OS filespec of the dataset in question
Dynamic
Set an advanced conditional break point to break on a certain number on iterations
See S222/S322 reasons and solutions
Also, set break point on file WRITE statements
Note: If not logic error and actually have full disk, you may have to clean up your disk and erase files:

32

Database Abends
DB2 application access rarely abends, they "die gracefully" due to return code processing.
DB2:
SQLCODE

A unique integer which describes DB2's reaction to your request.


SQLCA

A large 01 block, which contains several other fields pertinent to debugging, particularly the
SQLWARNs.
Suggestion:
Set Line Breakpoint and/or Variable Monitor on SQLCODE and other key feedback areas
- or Set Line Breakpoint and Watch Monitor for /"On-Change Break"
Double-click on field, Ctrl/F3
Notes on UDB Offloading and Database Abends
UDB handles SQLCODE processing in a manner consistent with DB2. However, it should be noted
that UDB does not handle SQLWARNO-n and the SQLCA (SQLERRD, etc.) fields compatibly.
DB2 ABENDs
If DB2 tables have not been defined, you may get a -204 return code on your SQL statement.
UDB may allow you to fix Database errors dynamically, without completely stopping your animator session:

33

UNIT

The RDz Workbench


Topics:

Analyzing Mainframe Abends


ABEND Codes and Reasons
Using IBM's Fault Analyzer
Appendix

34

Topic Considerations

Note: In this topic you will learn how to use Fault Analyzer to
debug an ABEND. The screen captures all describe connecting to a public z/OS
machine that IBM makes available during classes.
If you are taking this course through standard IBM services delivery you should
be able to use the properties (I/P address, port#s, etc.), logon IDs and passwords
that your instructor provides you with.
But you may also be taking this course standalone and in that case, you will
need to speak to your company's Systems Programming staff to learn how to
connect and logon.
It goes without saying that the actual file names in the screen captures of
mainframe libraries and datasets will vary. So you should focus on the process
and steps and "how to" and don't be perplexed at differences in screen
captures.
You also may be using your company's own Source Control Management system
to do things like builds, compiles, etc. In that case much of the remote
functionality in RDz will be customized and tailored to your company's unique and
idiosyncratic procedures and protocols.
35

Unit objectives
After completing this unit, you should be able to:
Work with ABEND analysis reports created by IBM Fault Analyzer
Browse Report and Mini-Dump pages
Retrieve various Fault Analyzer view information
Browse and search ABEND codes
Use the various productivity features in the Fault Analyzer perspective

Note:
This presentation and Lab is not a comprehensive IBM Fault
Analyzer unit. It is only intended to introduce you to the
RDz/Fault Analyzer interface. It is assumed that you are already
working with IBM Fault Analyzer on the z/OS host.

36

Overview and Incentive!

Face facts:
ABEND research ("shooting a dumps") is not
how you want to spend your week-nights
The good news? You don't have to.
Fault Analyzer:
Identifies the line where execution halted
Shows the points-of-interest surrounding the ABEND:
Variables and variable values
Statements
Data and buffers

Gives you a serious head start on the What/Where and How of ABEND
debugging
37

What is Fault Analyzer?

IBM Fault Analyzer is a tool that helps you determine the cause of an application
ABEND. It is used to determine:
What happened, how it happened, what program, what line/statement,
which variables,
variables what files, were involved, etc.

Fault Analyzer provides the necessary information to perform root cause analysis
on an application ABEND.
You do not have to interpret low-level, system dumps and wade through HEX data.
Information is presented in report format

IBM Fault Analyzer for z/OS gathers information about an application and the
surrounding environment at the time of an abnormal end (ABEND), providing you
with the valuable information you need to work through

After analyzing information about your application and its environment, Fault
Analyzer generates an analysis report (IDIREPORT)
IDIREPORT that describes the problem in
terms of application/program statements and variables

38

Fault Analyzer for z/OS Overview


Fault Analyzer is part of IBMs Problem Determination Tool family of
products: http://www-01.ibm.com/software/awdtools/deployment/
It runs in both test and production with very little overhead, and:
Provides support for analyzing***:
IMS and CICS online application and system failures - with debugging facilities for all of the IBMmainstream online files and databases
IMS-DL/I, DB2, VSAM, IDMS,
IDMS etc.

WebSphere Application Server for z/OS system failures


WebSphere MQ application failures
Batch (QSAM/VSAM/DB2) application failures
Helps you analyze failures when they occur or reanalyze them after the fact
Expands error messages and codes that apply to your failure with interactive reanalysis and includes a
feature for using application-specific messages and codes to supplement those supplied by IBM
Creates a fault history file with an interactive display that helps you track and manage application failures
Starts automatically when an application fails, eliminating the need to recompile programs or change the
job control language (JCL)
Integrates with Rational Developer for System z enables developers to diagnose application problems
without changing user interface

For Fault Analyzer product information, see:

http://www-01.ibm.com/software/awdtools/faultanalyzer/

39

***See Notes

Fault Analyzer Operational Process


The purpose of Fault Analyzer (FA) is to determine the cause of any ABENDs in an
application program.
You do not have to read through application or system dumps, because the product
has the ability to isolate the exact instruction that caused a particular error.
The Analysis engine provides automatic analysis when the application fails.
When an ABEND occurs, Fault Analyzer activates automatically, and then records details in
a fault history file (see screen capture below)
Fault History files contain information about the faults analyzed by Fault Analyzer for z/OS.
Using Fault History files, re-analysis is available when real-time ABEND analysis isnt
enough (you can extract additional information in batch or interactive mode)

ABEND happens
Fault Analyzer exits are invoked
Salient details (points of interest)
written and stored
40

Fault Analyzer Batch Job Outputs


Fault
FaultAnalyzer
Analyzer
IsIsautomatically
automatically
invoked
invokedififyour
your
program
programABENDs
ABENDs
An
AnIDIREPORT
IDIREPORTisis
produced
producedthat
that
provides
providesABEND
ABEND
analysis
analysis
information
information

41

Reviewing ABENDs in the Fault Analyzer Perspective

Besides the IDIREPORT, you may also wish


to use RDz's Fault Analyzer perspective, to
analyze and debug an ABEND situation.
To do that you'll need to:
1. Switch to the Fault Analyzer perspective in RDz
2. Specify the history file to connect with, that populates a
Default ABEND view with failed online and batch job
IDIREPORTs and other outputs
3. Learn how to navigate the Fault Analyzer perspective, to
make use of the information contained therein

The next slides contain the step details


42

Fault Analyzer Perspective 1 of 2


Steps:
1.

Open the Fault Analyzer Perspective

43

Fault Analyzer Perspective 2 of 2


2. Enter: FAULTANL.<version>.HIST
Ex.

FAULTANL.V10R1.HIST

44

Fault Analyzer Perspective Overview


Content
ContentArea
Areashows
shows
Fault
FaultAnalyzer
Analyzer
IDIREPORT
IDIREPORT
detailed
detailedinformation
information
on
onABEND
ABEND

Fault
FaultAnalyzer
Analyzer
Explorer
Explorer

IDIREPORT
IDIREPORT
Available
Availableinin
Outline
Outlineview
view
Can
Cannavigate
navigate
using
usingthe
the
Outline
Outlineview
view

Additional
AdditionalFault
FaultAnalyzer
Analyzer
views
viewsexist
existfor
forother
other
ABEND
ABENDdocumentation
documentation
including
includinghistory
historydata
data
45

Fault Analyzer Default List of History Files


From the Default tab
Scroll up and down to find a particular ABEND
Double-click an ABEND history file, to bring up its IDIREPORT and other stats
Sort the list by any of the column headings

Can also work with options of the Context Menu with each ABEND
entry

46

Fault Analyzer Browse Report S0CB


Program
Programline
linewhere
where
the
S0CB
occurred
the S0CB occurred

The IDIREPORT presents


a formatted, high-level
summary of the points of
interest necessary to
debug ABEND conditions
in your application.

Click
ClickS0CB
S0CBfor
for
an
explanation
an explanation
ofofthis
thisABEND
ABEND

Specifically, to answer the


questions:
What happened?
What z/OS ABEND
condition

Where did it happen?


what line or statement
was executing when it
happened

How did it happen?


What additional
information is available
for debugging purposes

47

Fault Analyzer Browse Report S0C4


Here's an example of an
IDIREPORT which shows that
RPT-REC is
"not addressable"
which is a euphemism for:
"There's something really
wrong with the

file/FD/JCL DD
connection"

48

Fault Analyzer Browse Report S0C7


The IDIREPORT and
supporting text varies from
ABEND to ABEND
depending on:
Type of ABEND
Information available
at the time of the
ABEND
Run-time platform

Note:
CUST-ACCT-BALANCE's
value is shown in hex (as it
represents invalid data)

49

Fault Analyzer Browse Report S0C9


A S0C9 is like a S0CB
(divide by zero) except
that a S0C9 occurs
because of an
excessively large fixedpoint number obtained as
the result of a decimal
division operation

50

Fault Analyzer Browse Report S0C1


Here is information on an IMS (DL/I) S0C1 ABEND

51

Fault Analyzer Browse Report S806


IDIDREPORT information on a module-not-found (S806) ABEND

Most likely SAM2 is


either a typo, or
the program did not
successfully
compile/link into the
Load Module

52

Fault Analyzer Lookup View for MVS and File Return Codes
The Lookup view
shows a great
deal of
background
information on:
ABEND codes
DB2 SQLCODE
IMS PCB Feedback
VSAM File Status
etc.

You can use the view,


or double-click on the
ABEND code shown
in the IDIREPORT

53

Fault Analyzer Integration Mini-Dump Reading 1 of 2


Fault Analyzer also provides for the
reading/browsing of System Dump data
in Hex/Character format.

Select
Selectan
an
ABEND
ABEND

Scroll
Scrollthrough
throughthe
thedump
dump
Issue
Issuenavigation
navigationcommands:
commands:
Show
Shownnn,
nnn,+nn,
+nn,etc.
etc.
54

Fault Analyzer Integration Mini-Dump Reading 2 of 2

You can assign analysis notes to the


dump.
1. Right-click over the storage address
2. Add your note (click OK)

3. Your note becomes


highlighted text inside the
dump
55

Fault Analyzer Organize ABEND History Views


Define
Defineviews
viewswhich
whichcontain
containaaset
setof
of
history
historyfiles.
files.
Display
Displayfault
faultentries
entriesfrom
fromall
allhistory
historyfiles
files
as
asififthey
theywere
wereininone
onedataset.
dataset.

2
56

Fault Analyzer Fault History View

The
TheDefault
Defaultview
viewshows
showseither:
either:

AAHistory
HistoryFile
File

AAview
viewof
ofaaHistory
Historyfile
file

57

Fault Analyzer The TSO/ISPF Interface

Available
Available
Line
Line
Commands
Commands

AAlist
listof
ofHistory
HistoryABENDs
ABENDs
Batch
Batchand/or
and/orOnline
Onlinemay
maybe
beininthe
thesame
samelist
list

58

Checkpoint
1. What RDz Perspective is used to view Fault
Analyzer reports?
2. How does RDz obtain Fault Analyzer Information?
Where does the information originate?
3. RDz Fault Analyzer interface has a Lookup View.
What is it used for?
4. How can you jump to the program statement
where the ABEND occurred with the RDz Fault
Analyzer interface?
59

Summary
Having completed this unit, you should now be able to:
Work with ABEND analysis reports created by IBM Fault Analyzer
Browse Report and Mini-Dump pages
Retrieve various Fault Analyzer view information
Browse and search ABEND codes
Use the various productivity features in the Fault Analyzer perspective

60

Workshop (for Sandbox Users) Big Picture


In this workshop you are going to:
1.

Copy a several datasets from your instructor's zServerOS TSO ID to your ID

Details on the next slide

2.

Modify JCL dataset names (and high-level qualifiers) to match your Sandbox ID

3.

Compile a program named: HOSPCALC which contains different types of COBOL


ABENDs generated from invalid COBOL logic in different parts of the program

4.

Run HOSPCALC (it will ABEND) and from the Fault Analyzer IDIREPORT:

Find the error in the COBOL source, and use the IDIREPORT ABEND analysis data to fix the error

After you've solved the problem, you will save your edits, and re-compile HOSPCALC. Then run the
program until you either get the next ABEND or get a zero return code

Compile
Compile
Link
LinkEdit
Edit

HOSPCALC
Load Module

Fix HOSPCALC COBOL error


Analyze IDIREPORT
61

Workshop Setup Steps 1 of 2


If you are working on the Sandbox, do the following:
1.

In Remote Systems, create a new MVS Filter named: RDZFA


for: DDS0001.POT.*

2.

Expand the RDZA filter, and:

Copy DDS0001.POT.COBOL member: HOSPCALC to <EM4Zxx>.POT.COBOL PDS

Copy DDS0001.POT.JCL member: HOSPCALC to <EM4Zxx>.POT.JCL PDS

Copy DDS0001.POT.JCL member: HOSPCRUN to <EM4Zxx>.POT.JCL PDS

3.

This is the compile JCL for HOSPCALC

This is the run JCL, for HOSPCALC

Right-click over DDS0001.POT.DATA and use Allocate Like to create a new


PDS named: <yourEM4Zxx>.POT.DATA
- with the same dataset characteristics

Example with ID: EM4Z07

62

Workshop Steps 2 of 2
4. Expand the EM4zxx.POT.JCL library:

Edit HOSPCALC.

Replace all occurrences of: DDS0001

Save your edits

with

<yourEM4Zxx>

Edit HOSPCRUN.

Replace all occurrences of: DDS0001.POT with

Save your edits

<yourEM4Zxx>.POT

5. Submit your modified HOSPCALC JCL. When the job finishes, check the results in your JES My Jobs
filter (it should compile successfully but check that the LKED step ran)
6. Submit your modified HOSPCRUN JCL. The job should ABEND. In JES - My Jobs,
Jobs view the IDIREPRT
step, to see the specific system completion code and surrounding Fault Analyzer ABEND analysis
information

Note that HOSPCALC contains four separate ABEND errors that you will hit, one at a time: an 0C7, an 0CB, an 0C4 and a
combined 0C7/0C4 ABEND (Sounds strange ... Essentially Fault Analyzer will find two different problems on one statement)

The comments and descriptions in the IDIREPRT combined with your COBOL knowledge should be sufficient to debug and
fix these problems, but feel free to elicit help from your instructor if you're stuck.

7. Fix the COBOL error that caused the ABEND. Then repeat steps 5 and 6 until you find and fix all of the
ABENDs in HOSPCALC. You're done when HOSPCRUN finishes with a zero return code.

63

You might also like