FYA

----- Forwarded by Dheeraj Pal/X/PH/Novartis on 05/07/2011 03:47 AM -----

Keith Greaves/X/PH/Novartis
To Dheeraj Pal/X/PH/Novartis@PH
04/07/2011 10:24 PM
cc Amandeep Singh/X/PH/Novartis@PH, Amit
Bhati/X/PH/Novartis@PH, Axel
Cornut/PH/Novartis@NOVEXCH, Deependra
Srivastava/X/PH/Novartis@NOVEXCH, Prasoon
Pathak/X/PH/Novartis@NOVEXCH, Jay
Vellanki/PH/Novartis@PH
Subject Re: CHBSCMT2 server -- Analyzed report(1)

Hi Dheeraj,

thank you very much for the information. I compared the NSD from CHBSCMT2 that you sent with the previous
8 NSD's from CHBSCMT3 (different servers and different days) and they are all almost exactly the same.
There are serveral of these NSD's. Here these crashes look to have been caused by an attachment that was trying to
be opened.

@[ 1] 0x60176b81 nnotes.Panic@4+417 (60b0595f)

@[ 2] 0x60002949 nnotes.LockHandle@12+249 (654d2020,2c5ff5b4,2c5ff5c0)
@[ 3] 0x600027b6 nnotes.OSLockObject@4+22 (60dff020)
@[ 4] 0x608b01ff nnotes.NSFNoteExtractFileUsingPipes@40+175 (145,10,2c63e484,2c63f770,0,0,0,0,0,0)
@[ 5] 0x607fd62c nnotes.NSFNoteExtractFileExt@24+108 (73,10,e484,2c63f770,0,0)
@[ 6] 0x609d0176 nnotes.MailExtractMessageAttachment@16+38 (73,10,cf4ce484,2c63f770)
@[ 7] 0x0041e027 nBTCal.ParseMailMessage+1691 (2e294143,736c78,78636f64,787300)

############################################################
### PASS 2 : FATAL THREAD with STACK FRAMES 2/4 [ nBTCal: 1354: 16f8]
### FP=2c5ff58c, PC=60176b81, SP=2c5fee18
### stkbase=2c640000, total stksize=831488, used stksize=266728
Exception code: c0000005 (ACCESS_VIOLATION)
 Best regards, Mit
freundlichen
Grüssen, Meilleures
salutations,

Keith Greaves

Novartis
Pharmaceuticals
Corporation

PHSDN00000, GIS
Client Services

USEH, 112-1021A,
USEH, 112-1025A

Novartis Pharmaceuticals
Corporation

One Health Plaza

East Hanover, NJ 07936-

1080

USA

Phone: +1 862 7785116

Email :
keith.greaves@novartis.co
m

Dheeraj Pal/X/PH/Novartis
To Keith Greaves/X/PH/Novartis@PH
04/07/2011 09:06 AM
cc Axel Cornut/PH/Novartis@NOVEXCH, Amandeep
Singh/X/PH/Novartis@PH, Amit Bhati/X/PH/Novartis@PH,
Prasoon Pathak/X/PH/Novartis@NOVEXCH, Deependra
Srivastava/X/PH/Novartis@NOVEXCH
Subject CHBSCMT2 server -- Analyzed report

Hi Keith,

We have reviewed NSD and few common things which we can communicate to BinaryTree team for all the servers
are: -
It happen while processing the mail4.box. Here is more information from Virtual Thread view.

.. SOBJ: addr=0x37d693f8, h=0xf010401e t=0xc176 (BLK_SDKT)

.. SOBJ: addr=0x01246fc8, h=0xf0104015 t=0xc130 (BLK_TLA)
.. SOBJ: addr=0x01276794, h=0xf010401d t=0xc30a (BLK_LOOKUP_THREAD)
.. SOBJ: addr=0x01275cb0, h=0xf010401b t=0xc275 (BLK_NSFT)
.. Database: F:\Lotus\Domino\Data\mail4.box
.... DBH: 608, By: CN=CHBSCMT2/OU=PH/O=Novartis, WasAccessed=Yes
...... doc: HDB= 608, NoteID=5544270, hNote=0x0012, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544282, hNote=0x000f, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544302, hNote=0x0013, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544422, hNote=0x0002, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544426, hNote=0x0011, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544442, hNote=0x0014, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544454, hNote=0x0015, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544482, hNote=0x0016, flags=2100, class=0001
...... doc: HDB= 608, NoteID=5515726, hNote=0x001a, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544486, hNote=0x0017, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544494, hNote=0x0018, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5544506, hNote=0x0019, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5515766, hNote=0x001b, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5515790, hNote=0x001c, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5520638, hNote=0x001d, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5520646, hNote=0x001e, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5519774, hNote=0x001f, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5572242, hNote=0x0020, flags=0100, class=0001
...... doc: HDB= 608, NoteID=5572354, hNote=0x0033, flags=2100, class=0001
.. file: fd: 1188, F:\Lotus\Domino\Data\mail4.box

Workaround Given
Since NSD are highlighting same problem on all servers and after changing the value of BTCALRESTARTTIMER
from 60 to 45 on CHBSCMT3. It has not experienced any more crash.
We have change the value of BTCALRESTARTTIMER to 45 on other CMT servers as well i.e. CHBSCMT1,
CHBSCMT2 and CHBSCMT4.

Refer to implemented GCM before (2011-077-27444 --> (2)) and now (2011-083-58140 --> (3))

I have attached NSD for you to take a look.

With Regards,

Dheeraj Kr. Pal

HCL

Mail & Groupware Operations

Lotus Notes/Domino Administrator - L3

Phone: +1 408-416-2085 Extn: 4329

Cell: +91-9811769905

Email : dheeraj.pal@novartis.com

--------------------------------------------------------------------------------

Dear Colleagues,

Please follow the below steps with immediate effect. Server should only be start after
following up the below steps.

Procedure
1. Look for the generated NSD file in
F:\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT and copy its filename into the
clipboard
Example: nsd_W32I_CHBSFB01_2011_05_27@14_41_25.log
2.
3. Use Windows Explorer to create a folder with the copied filename without
extension directly in the F: root directory
Example: F:\nsd_W32I_CHBSFB01_2011_05_27@14_41_25
4.
5. Use Windows Explorer to copy the mail.boxes and btexchange.box into this
newly created folder
6.
7. ... ??? maintenance tasks ??? ...
8. Restart the Domino server
9. Inform Bernd Vollmer via mail about the crash:
- mention the NSD file name
- provide additional information, that might be of interest
This gives us the chance to review the crash data and to decide whether we escalate to
BinaryTree and what data we provide.

For any understanding concerns please do not hesitate to contact us.

Thanks | Dheeraj

Dheeraj Kr. Pal

Lotus Domino/Notes Administrator - L3
Mail & Groupware Operations
HCL
 : 1 408-416-2085 Extn: 4326
* dheeraj.pal@novartis.com

 Please do not print this email unless it is absolutely necessary. Spread environmental awareness.