<Insert Picture Here>

FMW Diagnostic Framework

Diagnostic Framework (DFW) Background

Database group has built a generic diagnostic

framework to be adopted by all Oracle products
Primary goal of this framework is to reduce time to
resolution of bugs/issues
Aids in detecting, diagnosing, and resolving problems
FMW has adopted the concepts and parts of the
generic framework, and added Java/FMW specific
extensions
Targeted problems are critical errors caused by code
bugs, metadata corruption, customer data corruption,
deadlocked threads, and inconsistent state
Copyright 2011 Oracle and/or its affiliates. All rights reserved.

DFW Goals
First-failure diagnosis
Limiting damage and interruptions after a problem is
detected
Reducing problem diagnostic and resolution time
Simplifying customer interaction with Oracle Support

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

DFW Tools and Technologies

Automatic capture of diagnostic data upon first failure

Standardized log formats
Diagnostic rules
Incident detection log filter
Incident packaging service (IPS)
Integration with WebLogic Diagnostics Framework
(WLDF)
EM Support Workbench
ISS viewer

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

DFW Scope and relevance to layered products

DFW is applicable for :

Oracle WebLogic Server with JRF
Layered Fusion Middleware products that run on top of
Oracle WebLogic Server such as :
Oracle BI
Oracle SOA Suite
Oracle BPEL Process Manager
Oracle BAM
Etc.

Components such as OHS and Web Cache are

currently not covered.

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

Definitions
Problem: A problem is a critical error in the product. Each problem
has a unique problem id and a key.
Problem key : A text string that includes an error code

Incident: An incident is a single occurrence of a problem.

Incident package : An incident package (package) is a collection of
data about incidents for one or more problems.
Incident Packing Service : Enables creating and manipulating an
incident package.
Incident flood control : A flood controlled incident is one which does
not generate an incident dump to prevent information overload.

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

Definitions (contd..)

WebLogic Diagnostic Framework (WLDF) :

WLDF
consists of a set of components that work together to collect, archive,
and access diagnostic information about a WebLogic Server instance
and its applications.

Harvester:

Harvester in the WLDF component responsible for

collecting metrics from specified Mbean attributes.

Watches : A watch is a rule which monitors weblogic server runtime

Mbeans or log messages for a specified condition.

Notification :Once the watch condition is satisfied an associated

notification can be raised using SMTP, SNMP, JMX, or JMS .

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

Definitions (contd..)

Automatic Diagnostic Repository (ADR): is a filebased repository for diagnostic data such as traces, dumps, the alert
log, health monitor reports, and more.

ADRCI - The ADR Command Interpreter (ADRCI) is a utility that

enables you to investigate problems, and package and upload firstfailure diagnostic data to Oracle Support, all within a command-line
environment. ADRCI also enables you to view the names of the trace
files in the ADR, and to view the alert log with XML tags stripped, with
and without content filtering.
WLST : DFW has WLST commands for querying Problems/Incidents
across WLS servers, view incident dump files, create manual incidents,
execute dump manually etc

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

DFW Components

Automatic Diagnostic Repository

Java Diagnostic Dumps
Java Diagnostics Data Extractor
Diagnostic Framework Mbeans
Diagnostic Framework (ADR) Command Line
Interface (CLI)

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

Automatic Diagnostic Repository / ADR

Stores diagnostic data (trace, dumps, metadata) for
all Oracle products inside a single file-system
directory hierarchy
ADR data is highly structured
Metadata (e.g. problem/incidents) stored in
relations
Dumps for incidents are performed in separate files
Each WebLogic Server stores diagnostic data in
subdirectories of its own home directory within the
ADR

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

10

Sample ADR Directory Structure for WebLogic

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

11

ADR Home & ADR Base

ADR home - is the root directory for all diagnostic

data for a particular instance of an Oracle
product/component.
ADR Base - ADR homes for different
products/components are grouped together under the
same root directory called the ADR base.

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

12

Java Diagnostic Dumps

Perform targeted diagnostic data collection

Simple Java interface
Implemented by components and applications
Registered at runtime
Can be run manually and automatically
Out Of The Box Diagnostic dumps include:

Threads
Execution Context (all active ones)
Active HTTP requests
Class histogram
DMS Metrics
Logs (by ECID)
Logs (by timestamp up to -/+ 5 minutes)
WLDF Server Image

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

13

Java Diagnostic Dumps layout

An incident on WebLogic Server may include the
following dumps:
Name: jvm.threads
Dump Files: jvm_threads1_i1.dmp
Name: dms.metrics
Dump Files: dms_metrics2_i1.dmp
Name: odl.quicktrace
Dump Files: odl_quicktrace3_i1.dmp
Name: odl.logs
Dump Arguments: ecid=0c4a50bdcb4e41e1:-2f28425f:132890c8fb3:-8000-0000000000000a9d
Dump Files: odl_logs4_i1.dmp
Name: odl.logs
Dump Arguments: timestamp=1316728757090
Dump Files: odl_logs5_i1.dmp
Name: wls.image
Dump Files: diagnostic_image_AdminServer_2011_09_22_21_59_17.zip

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

14

Java Diagnostics Data Extractor (DDE)

Active in each WebLogic Server
Rule driven incident detection and creation
Incident detection
Java.util.logging log filter
WLDF Diagnostic Module JMX notifications

Incident creation
Default set of dumps to execute
Rule based dump execution

Java API for creating incidents

Gathers incident meta data (correlation keys, component info,
etc.)
Flood control
Support for creating manual incidents

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

15

Java Diagnostics Data Extractor (Contd..)

Out Of The Box rules detect INCIDENT_ERROR level log
messages
Out Of The Box WLDF Watch/Notification Configuration:

Out of memory errors

Uncaught exceptions
Thread deadlocks
Stuck threads

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

16

Management Beans (MBeans) - JMX

DFW provides MBeans that can be used to configure

the Diagnostic Framework
Can configure an WLDF Watch and Notification to
create an incident
Enable or disable the detection of incidents through
the log files
Enable or disable flood control and setting parameters
for flood control
Size management to configure max size for all
incidents

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

17

Management Beans

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

18

ADRCI Command-Line Utility

ADR Command Interpreter

Utility that enables users to investigate problems, and
package
A list of common ADRCI commands is included in a
subsequent slide.

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

19

WLST and DFW

listProblems()
Problem Id Problem Key
1
BEA-101020 [HTTP]
listIncidents(id=1)
Incident Id Problem Key
2 BEA-101020 [HTTP]
1 BEA-101020 [HTTP]

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

Incident Time
Fri Feb 26 13:42:01 PDT 2010
Tue Feb 23 06:17:39 PDT 2010

20

WLST & DFW

showIncident(id='1')
Incident Id: 1
Problem Id: 1
Problem Key: BEA-101020 [HTTP]
Incident Time: Tue Feb 23 06:17:39 PDT 2010
Error Message Id: BEA-101020
Execution Context: 0000IExqUvyAhKB5JZ4Eyf1Afdj600009i
Flood Controlled: false
Dump Files : dms_ecidctx1_i1.dmp jvm_threads2_i1.dmp
dms_metrics3_i1.dmp odl_logs4_i1.dmp odl_logs5_i1.dmp
diagnostic_image_AdminServer_2010_02_23_06_17_42.zip
readme.txt

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

21

WLST & DFW

executeDump(name='dms.metrics',
outputFile='/tmp/dumpout.txt', id='1')
Dump file dms_metrics1_i1.dmp added to incident 1

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

22

Incident Creation Log filter

J.U.L/ODL

log message

logging.xml
create

invoke

Execute
dumps

load
application
Rules/register
dumps

Write message

Incident Filter

Create incident?

DDE

create incident
Execute
dumps

load
Default
rules
Register
Default
dumps

Server.log
ADR

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

23

Incident Creation Generated by WLDF Watch

Notification
Fusion Middleware configures a WLDF Diagnostics
Module FMWDFW
It contains a set of Watch and Notification rules for
detecting a specific set of critical errors and creating
an incident for each occurrence of those errors.
By default it contains the following watch conditions:
Deadlock
StuckThread
UncheckedException such as NullPointerException,
StackOverflowError, or OutOfMemoryError

Custom Watch conditions can be added

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

24

Incident Creation Generated by WLDF Watch

Notification
Managed Server

WLDF Diagnostic

Watch
Notification

Register

Diagnostic
Rules

Image Bean

DFW Diagnostics Data Extractor

Write

Notify

1
Register

Invoke

Execute

Execute

Create
Incident

ADR

Component Dump

Application Dump

Write

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

25

Incident Creation Notification by WLDF Watch

Notification

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

26

Incident Packaging Service

Package and intelligently gather Incident data

Related incidents packaged based on:
ECID
Timestamp
Component

Prompt the user to dump additional data (IPS actions)

Intrusive dumps
Configuration info
Test cases

Initiate and package RDA report

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

28

EM Support Workbench
Manage reported problems
System alerted or user-reported
Manage life-cycle and history of problems found
Single target view, group view, stack view, enterprise view

Present patches and workarounds when available

Create Service Requests
Packages data using IPS
Upload incident information and test cases to Oracle
Provide guided resolution for non-bug problems
Automatically track and synchronize Service Request
status from Oracle

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

30

Workbench - Domain Summary

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

31

Workbench - Problems

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

32

Workbench - Problem Details

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

33

Workbench - Incident Details

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

34

Workbench - Package

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

35

ADRCI Commands

Common ADRCI commands include:

HELP
SET BASE
SET HOMEPATH
SHOW ALERT
SHOW PROBLEM
SHOW INCIDENT
SHOW TRACEFILE
IPS CREATE PACKAGE
IPS GENERATE PACKAGE
PURGE

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

36

Manual Incident creation and packaging

createIncident(messageId='MDS-50500',
description='sample incident')
Incident Id: 1 Problem Id: 4 Problem Key: MDS-50500 [MANUAL]
Incident Time: 23rd February 2010 11:55:45 GMT Error Message Id: MDS-50500
Flood Controlled: false

MW_HOME/wlserver_10.3/server/adr/adrci
Set ADR BASE and ADR HOMEPATH variables
IPS CREATE PACKAGE INCIDENT 1
Created package 1 based on incident id 1, correlation level typical

IPS GENERATE PACKAGE 1 in /tmp

Generated package 1 in file /tmp/BEA337Web_20100223132315_COM_1.zip,
mode complete

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

37

RDA & DFW

Remote Diagnostic Assistant is a command line diagnostic tool which

provides a comprehensive picture of the middleware environment.
If the diagnostic dump information captured in the incident package
proves to be insufficient for analyzing the root cause, an RDA report
may be generated as additional input for support.

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

38

Additional References
DFW Links

DFW Project page

DFW Support Internals
DFW Customer Documentation
DFW Internal Dump Documentation
DFW Functional specification
DFW WebLogic Server port functional specification
DFW Developers guide
DFW JavaDoc
ADR Customer Documentation

FMW Diagnostics

Diagnostics Spy Application

FMW Diagnostics OraTweet Group fmwdiag
FMW Diagnostics Connect Group
Mailing list: fmwdiag_ww

Copyright 2010 Oracle and/or its affiliates. All rights reserved.

39

Copyright 2011 Oracle and/or its affiliates. All rights reserved.

40