Analyzing Logs for SAP BusinessObjects BI4!

EV Technologies!
EV Technologies is an SAP BusinessObjects solutions rm ! SAP Software Solutions Partner! SAP Certied Solutions provider! Sybase Certied Solutions provider! SAP BusinessObjects Enterprise Certied! SAP Mentors! SAP-PRESS Authors! ASUG Members/Volunteers! Migration experts classic BusinessObjects products to SAP BusinessObjects XI R2 XI 3.1- BI4!

Coy Yonce!
Manage product backlog and upcoming new features for Sherlock and related solutions! Implement and advise customers on usage of Sherlock! Previously with SAP for 11 years!
6 years in support! 3 years as a TAM! 2 years as a Product Manager for CR!

Diversified Semantic Layer

Unprofessional journalism at its finest Recorded by a bunch of guys in the SAP global community Dont miss podcasts both on product news and application, as well as interviews with other BOBJ people in the community Follow on twitter at @dslayered
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the le again. If the red x still appears, you may have to delete the image and then insert it again.

http://dslayer.net

Agenda!

We Approve this Message!

Troubleshooting & Monitoring!

Pesky User Errors
Some pre-dened lters are no longer available in the Universe (IES 00003) Connec?on or SQL sentence error (DA0005) Internal error: Please contact your System Administrator. (CRS 200001)

Performance and Stability

Schedules are not comple?ng within a dened window or as quickly as they used to

User logins are taking greater than 10 seconds to complete

Naviga?on seems slow

Con?nual Monitoring

IntermiVent user error or service availability issue over an extended period

Introduc?on of 500 new users and need to monitor for impact on the deployment

Agenda!

We Approve this Message!

Enable Server Logging!

For each service in CMC

We Approve this Message!

For each applica?on in CMC

Enable Client Logging!

We Approve this Message!

BO_Trace Informa?on Design Tool WebIRichClient_trace WebI Rich Client designer_trace Universe Designer

Log location!

BOBJ Service Log Loca/on Windows: <BOBJ Install Path>/SAP BusinessObjects Enterprise XI 4.0/logging/ UNIX: <BOBJ Install Path>/sap_bobj/enterprise_xi40/logging/lica?on server Web Applica/on Log Loca/on $userHome/SBOPWebapp_ $applica?on_$IPaddress_$port/. Where $userHome is the users home directory used to run your web applica?on server

End to End Trace!

We Approve this Message!

Agenda!

We Approve this Message!

Architecture!

We Approve this Message!

Client Tier

Architecture!
Web Tier
We Approve this Message!

Management Tier

Storage Tier

Architecture!

We Approve this Message!

Processing Tier

Architecture!

We Approve this Message!

Data Tier

Example Process Flow!

We Approve this Message!

Troubleshooting Toolkit!
Linux or Unix: Built-in binaries (e.g., grep, nd, awk, uniq, lsof) Windows: Install UnxU?ls and Process Explorer Develop a toolkit - Log analysis (UnxU?ls, GLF Viewer) - Database tes?ng and querying tool (RazorSQL, DatabaseBrowser, pklite) - Archiving (WinRAR, 7-Zip) - Network tracing (Wireshark, Fiddler) - Process monitoring (Process Explorer) - Web service tester (Wizdl)

We Approve this Message!

Parsing Commands - GREP!

We Approve this Message!

Parsing Commands - FIND!

We Approve this Message!

Parsing Commands - FIND!

We Approve this Message!

Parsing Commands - AWK!

We Approve this Message!

Parsing Commands - AWK!

We Approve this Message!

Parsing Commands - UNIQ!

See all unique errors for the same process ID with the same caller and callee

Parsing Commands - UNIQ!

See all unique errors from all logs regardless of process ID, caller, or callee

Parsing Commands - LSOF!

We Approve this Message!

Format of logs!
Loca/on|Guid|Time|Tzone|Trace|Log| Importance|Severity|Excep/on|DeviceName| ProcessID|ThreadID|ThreadName|ScopeTag| MajorTick|MinorTick|MajorDepth|MinorDepth| RootName|RootID|CallerName|CallerID| CalleeName|CalleeID|Ac?onID| DSRRootContextID|DSRTransac?on| DSRConnec?on|DSRCounter|User| ArchitectComponent|DeveloperComponent| Administrator|Unit|CSNComponent|Text

Important Fields Loca?on

We Approve Time Message! this

Trace Importance Severity Excep?on ProcessID CallerName CalleeName Text

Importance << <= == >= >> xs s m l xl S W E A F

Severity Success Warning Error Assert Fatal

Format of logs!
|6fed8a0b-f579-3784-78c0-188c8f4cd4b0|2012 12 27 15:55:55:303|-0600|Error|Error|>>|E| | WebIRichClient|11700|7124|| |1|12|1|0|Webi SDK.CorbaServerImpl.doProcess()| evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1| WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1| We Approve this Message! CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:RequestProc:user: Administrator, doc: "SCI019 - Desktop Intelligence Migra/on Planning", error stream: <ERRORS> kc3cdbtbdacs.cpp:322:long __cdecl DoReportCSError(const class Connec/onServer::Error &,const int,const char *const ): TraceLog message 3 |e9d8fe68-9485-db74-6b85-6ae45762ea16|2012 12 27 15:55:51:755|-0600|Error|Error|>>|E| | WebIRichClient|11700|7124|| |1|0|1|0|Webi SDK.CorbaServerImpl.doProcess()| evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1| WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1| CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:C3_dacs:CS::Error : (DBD(1),ID:769) "ORA-00936: missing expression " . [kc3cdbtbdacs.cpp;322]

Format of logs!
kc3CoreEngineImpl.cpp:1750:enum IECore::IECoreStatusCode __stdcall IECore::kc3CoreEngineImpl::processDPCommandsEx(const class IECore::UTF8Buer &,const class std::map<long,class std::vector<class IECore::CommandAeachment,class std::allocator<class IECore::CommandAeachment> >,struct std::less<long>,class std::allocator<struct std::pair<long We Approve this Message! const ,class std::vector<class IECore::CommandAeachment,class std::allocator<class IECore::CommandAeachment> > > > > &,class std::basic_ostream<char,struct std::char_traits<char> > &,class std::map<long,class std::vector<class IECore::OutputAeachment,class std::allocator<class IECore::OutputAeachment> >,struct std::less<long>,class std::allocator<struct std::pair<long const ,class std::vector<class IECore::OutputAeachment,class std::allocator<class IECore::OutputAeachment> > > > > &): TraceLog message 20 |7915e975-te7-f7b4-98b8-641128acecd7|2012 12 27 15:55:55:298|-0600|Error|Error|>>|E| | WebIRichClient|11700|7124|| |1|2|1|0|Webi SDK.CorbaServerImpl.doProcess()| evtbobj12:11700:39.139:1|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:11700:39.139:1| WebIRichClient.processDPCommandsEx|localhost:11700:7124.102:1| CrFvLIwIA0_yjUZ0gB1ZLVI46|||||||||||**ERROR:IECore_kc3CoreEngineImpl:Caught a BOExcep/on: : ExecuteQuery/RefreshBatch failed [kc3CoreEngineImpl.cpp;1750]

Format of logs!
../../shared/infostore_facade.cpp:81:unsigned long __cdecl InfoStoreFacade::GetConnec?onServerNames(const class ATL::CComPtr<struct ISEnterpriseSession> &,class std::vector<class ATL::CAdapt<class ATL::CComBSTR>,class std::allocator<class ATL::CAdapt<class ATL::CComBSTR> > > &): TraceLog message 3 |d3f3595e-a84d-a654-18a6-b4bc34c0160a|2012 11 We Approve this Message! 25 07:21:32:035|-0800|Error| |>>| | | connec?onserver_EVTBOBJ12.Connec?onServer32| 6844|3592||}|2688|0|2|2|Webi SDK.CorbaServerImpl.doProcess()|evtbobj12:6644:1121.16164:1| webiserver_EVTBOBJ12.WebIntelligenceProcessingServer.openDocumentMDP|localhost: 5156:2164.324:1|connec/onserver_EVTBOBJ12.Connec/onServer32.DBAliases|localhost: 6844:3592.56:1|Cg3aECviNUxykU8buYB4IPM3c1f|||||||||CS|| InfoStoreFacade::GetConnec/onServerNames: 0.172 Args=(session="3&1=System Account, 0P&W={},2z&2r=evtbobj12:6400,8P&35=System Account, 0P&3k=@evtbobj12:6400,8P&4E=4963JaihlCSvtpxhjjWxdMCINCs, 0P&4F=10,0P&5U=4964JdIQkyNN2vPrgmTvWK7OZnS4963JaihlCSvtpxhjjWxdMCINCs,0P&63=server- token,0P&?u=1353856731473,9L&S5=1,08&Tm=36500,03&lu=1033,83&pa, 0P&ua=AWmaEx4Z.NVPpAEthuTGAjc, 0P&ub=AeLq4SaDG0dEqgPyY4C1Tp8,0P&ux=AdSP.rt24tREj8Jmr7nF_f8,0P&uy=-1352502874742,0L" )

General Approach!
1. 2. 3. 4. 5. 6. 7. Iden?fy applicable errors (notate errors and ?mes) Search for same ?me period from related logs Store all of this into a single text le Create a similar le of a working session Compare them to nd dierences Lookup error messages on SMP, SCN, & BOB Filter down to poten?al causes based on understanding of architecture, process ow, components involved, error message, and knowledge of user workow

User Errors!

We Approve this Message!

Example: SQL Error!

../odbcapi.cpp:399:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLExecute(void *): TraceLog message 24256577 |69fedf49-25a9-6894-78c2-6400f5fd8b6a|2013 02 05 21:44:04:138|-0600|Informa?on| |>=| | | aps_EVTBOBJ12.Adap?veProcessingServer| 3284|8108||}|0|9|0|3|-|-|-|-|-|-|||||||||||| CS::ODBCAPI::SQLExecute: 0 |87be3f6f-6131-dd44-c8fd-bea0e6955979|2013 02 05 21:44:04:140|-0600|Error|Error|>>|E| | aps_EVTBOBJ12.Adap/veProcessingServer| 3284|8108|| |0|10|0|2|-|-|-|-|-|-||||||||||||^[|ODBC^[|MS SQL Server 2008^[|JobId:2199138928^[|EXIT SQLGetDiagRec with return code 0 (SQL_SUCCESS) SQLSMALLINT 3 SQLHANDLE 0x00000000003FF160 SQLSMALLINT 1 SQLTCHAR * 0x000000006FBBDDD0 [42S02] SQLINTEGER * 0x000000006FBBDCB4 (208) SQLTCHAR * 0x000000006FBBDDE0 [[Microsol][SQL Server Na/ve Client 10.0][SQL Server]Invalid object name 'BOE140_Audit.dbo.Managed_En/ty_Status_Metrics'.] SQLSMALLINT 513 SQLSMALLINT * 0x000000006FBBDCB0 (123

We Approve this Message!

Example: BI LaunchPad!
|DBB5F33CD65E4177A2C808B20B476A351b|2013 02 25 14:07:38.184|-0600|Informa?on|Error|>=|E| | BIlaunchpad| 1092| 39|hVp-8080-5 | |12|0|0|0|BIlaunchpad.WebApp|evtbobj12:1092:39.466:1|-|-| BIlaunchpad.WebApp|evtbobj12:1092:39.466:1|Ci2QIVp7Lk7vmD5ZAzaOWwc1d0||||||||| com.businessobjects.adv_ivcdzview.DHTMLLogger||viewReport[IllegalArgumentExcep?on] <reference> (Error: RWI 00011) Invalid reference: refID=1.4, followBid=; Anchor=; Slots=/ |DBB5F33CD65E4177A2C808B20B476A351c|2013 02 25 14:07:42.369|-0600|Error| |>>| | |BIlaunchpad| 1092| 39|hVp-8080-5 | |424|0|0|0|BIlaunchpad.WebApp|evtbobj12:1092:39.471:1|-|-|BIlaunchpad.WebApp| evtbobj12:1092:39.471:1|Ci2QIVp7Lk7vmD5ZAzaOWwc1d5||||||||| com.sun.faces.renderkit.html_basic.LabelRenderer||component testSaveAs_SaveAsTypeLabel must be enclosed inside a form

We Approve this Message!

Performance and Stability!

We Approve this Message!

Example: Timings!
../odbcapi.cpp:174:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLBindParameter(void *,unsigned short,short,short,short,unsigned __int64,short,void *,__int64,__int64 *): TraceLog message 24253030 |c3b35919-21-2134-ca5e-138bdc261b70|2013 02 05 21:42:56:381|-0600|Informa?on| |>=| | | We Approve this Message! aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|14|0|2|-|-|-|-|-|-|||||||||||| CS::ODBCAPI::SQLBindParameter: 0.001 ||||||||||CS::ODBCAPI::SQLDescribeParam^^ ../odbcapi.cpp:346:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLDescribeParam(void *,unsigned short,short *,unsigned __int64 *,short *,short *): TraceLog message 24253032 |c750414d-9f5c-f424-9a33-d7518d07cabb|2013 02 05 21:42:56:382|-0600|Informa?on| |>=| | | aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|16|0|2|-|-|-|-|-|-|||||||||||| CS::ODBCAPI::SQLDescribeParam: 0 ../odbcapi.cpp:399:short __cdecl WODBC3_MSSQLServer::ODBCAPI::SQLExecute(void *): TraceLog message 24253041 |af6ba143-f480-2a34-4a69-5964f19b87d0|2013 02 05 21:42:56:382|-0600|Informa?on| |>=| | | aps_EVTBOBJ12.Adap?veProcessingServer| 3284|5728||}|0|25|0|3|-|-|-|-|-|-|||||||||||| CS::ODBCAPI::SQLExecute: 0

Example: Server Starting!

.\ProcServerChild.cpp:636:-: TraceLog message 5 |c5f301-de60-c874-7990-1a82cc1a99fc|2012 11 25 07:00:12:532|-0800|Error| |>>| | | cr2011proc_EVTBOBJ12.CrystalReports2011ProcessingServer_child_CRPE10| 7032|6836|| ||||||||||||||||||||||ras21-core: caught IPSSExcep?on: <RASServerExcep?on We Approve system aessage! ( errCode=<ErrorFailure> message=<Internal error. Please contact your this Mdministrator.> rootCause=<RASPROCREPORT0205> debugString=<CR cong object not found OR the number of objects is not 1> )> .\ProcServerChild.cpp:1038:-: TraceLog message 6 |t9e4663-8ea5-7c24-6973-32e93eac64e5|2012 11 25 07:00:12:540|-0800|Error| |>>| | | cr2011proc_EVTBOBJ12.CrystalReports2011ProcessingServer_child_CRPE10| 7032|6836|| ||||||||||||||||||||||ras21-core: did not ini/alize sucessfully, so stopping immediately

Example: CMS Overloaded!

(err.m_IDSError == IDS_ERR_CMSSYSTEMDB_NODATABASECONNECTIONSAVAILABLE : Error collec?ng resources: CMS system database is overloaded, please try again. If the problem persists, please contact the system administrator(FWB 00027) Wri/ng to the database We Approve this Message! Server Intelligence Agent: server SIAKWEB9VM.CentralManagementServer stopped with exit code = 1 (0x1). |B3AF369E866B48DF969863AD19492A3713|2012 03 09 19:27:16.189|-0600|>>|E| | SIA_SIAKWEB9VM| 3516| 31|SIAKWEB9VM.CentralManagementServer (Process Agent)| ||||||||||||||||||||com.businessobjects.cad.syslog||[Node Name: SIAKWEB9VM] [User Name: SYSTEM]

Example: WPS Down!

|01d8c282-b69c-cf94-9ae7-c322b6ce000e|2012 03 09 20:04:34:361|-0600|==| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|5044|| | 159|236|3|2| CMC.WebApp|kweb9vm:6656:39.9950:1|cms_SIAKWEB9VM. CentralManagementServer.commitEx4|localhost:8476:8100.9967:1| We Approve this Message! webiserver_SIAKWEB9VM. WebIntelligenceProcessingServer. setParamEx|localhost:4396:5044.39:1| CqKqRpGoXEQQjHPyW2Diwec26db|||||||||||OSCASimpleFactory:: disabledServer() true

Example: WPS CPU!

|25340224-4f65-5384-2955-a4729409dea4|2012 03 09 20:21:34:908|0600|>=| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|4640|| |0|724|0| 4||||||-||||||||||||GenericContainer_MetricManagerImpl:Total CPU /me (seconds):16.3594 We Approve this M | |99b5b174-24e7-36f4-4b06-3cf8912t257|2012 03 09 20:28:05:455|0600|>=| | essage! webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|4640|| |0|968|0| 4||||||-||||||||||||GenericContainer_MetricManagerImpl:Total CPU ?me (seconds):208.156

Example: WPS RAM!

|332b35cd-a260-d924-9852-f0641f4c8e35|2012 03 09 20:24:42:455|-0600|>=| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|8356|| |15369|82|1|6| BIlaunchpad.WebApp|kweb9vm:6656:39.9987:1|BIlaunchpad. WebApp|kweb9vm:6656:39.9987:1|webiserver_SIAKWEB9VM.WebIntelligence We ProcessingServer.getMap|localhost:4396:8356.1828:1| Approve this Message! CqKqRpGoXEQQjHPyW2Diwec2700||||||||||| GenericContainer_MetricManagerImpl:Memory level (0=normal, 1=low, 2=high, 3=max):0 |cdb8f014-2fce-ab64-39e1-0d74b9574308|2012 03 09 20:27:05:424|0600|>=| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 4396|2076|| |0|225|0| 7||||||-||||||||||||GenericContainer_MetricManagerImpl:Virtual memory size (Mb):1043

Example: IFRS & OFRS!

|da26f544-5c28-03c4-5829-4618bdd19a95|2012 03 13 07:32:46:002|-0500|>>| | | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 9084|5636||}|2724|0|2|1| BIlaunchpad.WebApp|kweb9vm:6656:41.13148:1| webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer.openDocumentMDP|localhost: We Approve 9084:5636.4457:1|CComFile.CopyTo|localhost:9084:5636.4457:314| this Message! CqKqRpGoXEQQjHPyW2Diwec3359|||||||||||Step: CopyTo: 0.141 InfoStoreOpera?onHelpers.cpp:1422:void __cdecl Helpers::InfoStore::ObjectFileHandle::extractContent(const class bo_u8string &) const: TraceLog message 202744 kdgstoreCE.cpp:1054:long __cdecl StoreFacade::ImportState_orig(class bo_u8string &,struct idgStore::DOCSPEC &): TraceLog message 202753 |fdb865ce-6d06-1aa4-4b2a-8756e85db811|2012 03 13 07:32:46:080|-0500|>>|E| | webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer| 9084|5636|| |2733|0|1|8| BIlaunchpad.WebApp|kweb9vm:6656:41.13148:1|BIlaunchpad.WebApp|kweb9vm: 6656:41.13148:1| webiserver_SIAKWEB9VM.WebIntelligenceProcessingServer.openDocumentMDP|localhost: 9084:5636.4457:1|CqKqRpGoXEQQjHPyW2Diwec3359|||||||||||**ERROR:dg_storeCE:Unable to retrieve the document from the FRS [kdgstoreCE.cpp;1054]

Continual Monitoring!

We Approve this Message!

Continual Monitoring!
SAP BOBJ SAP BOBJ SAP BOBJ

We Approve this Message!

Log Processing

SAP BOBJ SAP BOBJ Log Storage

Agenda!

We Approve this Message!

Automation!

We Approve this Message!

How do they work?!

2012 12 27 15:55:51:755 **ERROR:C3_dacs:CS::Error : (DBD(1),ID:769) "ORA-00936: missing expression " . [kc3cdbtbdacs.cpp;322] Caller: Webi SDK.CorbaServerImp l.doProcess()

Callee: WebIRichClient.proc essDPCommandsEx

Data store

How do they work?!

How do they work?!

Agenda!

We Approve this Message!

More Information!
Timothy Coman and Toby Johnston with SAP @ Remote Supportability Blog SAP BusinessObjects BI4 Process Flows Business Intelligence Plaorm Administrator Guide SAP BusinessObjects BI System Administra?on Eric Vallo & Greg Myers @ SAP Press We Approve this Message! Universal Command Guide: For Opera?ng Systems Guy Lotgering Command References Find Awk Lsof Grep Uniq

More Information!
Contact: Coy Yonce! Email: coy@evtechnologies.com! On the Web: evtechnologies.com! Twitter: @coyyonce!