Info regarding DMX

rgunda 40 posts since

May 28, 2008 Can any one tell me what does the fields Symb, TID and identifier signify in the output symdisk -sid
xxxx list mean. Also, can any one tell me on what basis hotspares are defined per a storage array? Tags: symmetrix

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 1. Re: Info regarding DMX Nov 5, 2008 10:14 AM
1) I don't have a symdisk output available.. can you please post an example output ?? It will be easier to answer your
question...

2) number of hotspares defined depends on a lot of variables.. it depends on the code level, on
the number of drives and speed/size of drives in your storage.. A better answer requires a lot of
"it depends" thus it's better if you give more details on the environment.. Code level, box type,
number/speed/size of drives in the backend .. This will help in explaining you actual numbers

rgunda 40 posts since

May 28, 2008 Reply 2. Re: Info regarding DMX Nov 5, 2008 10:34 AM

in response to: Stefano Del Corno The output from symdisk list is:

symdisk -sid xxx list |more

Symmetrix ID : 000125772xxx
Disks Selected : 1290

Capacity(MB)
Ident Symb Int TID Vendor Type Hypers Total Free Actual
------ ---- --- --- ---------- ---------- ------ -------- -------- --------
DF-1A 01A C 0 SEAGATE C146X15 52 140014 125 140014
DF-1A 01A C 2 SEAGATE C146X15 52 140014 125 140014
DF-1A 01A C 4 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C 6 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C 8 SEAGATE C146X15 52 140014 306 140014
DF-1A 01A C A SEAGATE C146X15 52 140014 306 140014

Actually, we have DMX 3 in our place and the code on it is 5671 and we have 146 and 300gb drives and a total of
1290 drives...and the drive speed is 15k rpm and we have hot spares of 18...can u explain why?

Message was edited by:

Stefano Del Corno

Generated by Clearspace on 2010-01-13-07:00

1
Info regarding DMX

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 3. Re: Info regarding DMX Nov 5, 2008 10:46 AM

in response to: rgunda Ident - Identifier of processor

Symb - Symbolic name of processor (01A in your example)
Int - Interface (each processor have 2 interfaces, C and D)
TID - target identifier or if you prefer, address of drive in the look managed by processor.

When you want to dig into a single specific drive you still use "symdisk show" but supply also a "drive identifier"
obtained merging symb,int and tid.

symdisk show 01aC:0

Can you please post output of command "symdisk -hotspare list" ??

When you buy the storage (or when you upgrade it), our salesman use a tool (direct express) that handles all the
math behind hotspares (and a lot of other things). Thus when you order your box, it already have "the right" number
of spares. I think that 1% of capacity is a good tradeoff .. recent codes have stricter rules, requiring more spares,
spread at key positions in the backend.

rgunda 40 posts since

May 28, 2008 Reply 4. Re: Info regarding DMX Nov 5, 2008 2:48 PM

in response to: Stefano Del Corno This is the output from symdisk list

symdisk -sid 01 list -hotspare

Capacity(MB)
Ident Symb Int TID Vendor Type Hypers Total Free Actual
------ ---- --- --- ---------- ---------- ------ -------- -------- --------
DF-1A 01A D 1D SEAGATE C146X15 0 0 0 140014
DF-2A 02A C 1D SEAGATE C146X15 0 0 0 140014
DF-5A 05A D B SEAGATE T300155 0 0 0 286102
DF-11A 11A D 1D SEAGATE T146155 0 0 0 140014
DF-16A 16A C 1D SEAGATE C146X15 0 0 0 140014
DF-1B 01B C 1D SEAGATE T300155 0 0 0 286102
DF-2B 02B D 1D SEAGATE C146X15 49 0 0 140014
DF-6B 06B D 1D SEAGATE T146155 0 0 0 140014
DF-15B 15B C 1D SEAGATE T300155 0 0 0 286102
DF-16B 16B D 1D SEAGATE C146X15 0 0 0 140014
DF-1C 01C D 1D SEAGATE C146X15 0 0 0 140014
DF-2C 02C C 1D SEAGATE C146X15 0 0 0 140014
DF-15C 15C C 4 SEAGATE C146X15 0 0 0 140014

Generated by Clearspace on 2010-01-13-07:00

2
Info regarding DMX

DF-16C 16C C 1D SEAGATE T300155 0 0 0 286102

DF-1D 01D C 1D SEAGATE C146X15 0 0 0 140014
DF-2D 02D D 1D SEAGATE T146155 0 0 0 140014
DF-15D 15D C 1D SEAGATE C146X15 0 0 0 140014
DF-16D 16D D 1D SEAGATE T300155 0 0 0 286102

mlee 83 posts since

Nov 2, 2006 Reply 5. Re: Info regarding DMX Nov 5, 2008 7:14 PM

in response to: rgunda Hello All,

As Stefano has already alluded to the correct quantity of "hot" or "dynamic" spare drives can be quite a complicated
calculation.

The required quantities are automatically calculated by the EMC Sales Tools and added to your initial install or
subsequent upgrade Sales Order.

The minimum quantity and size of the spare drives depend on the actual sales order and is based on the Symmetrix
Model, drive family (Flash or HDD), drive size, drive speed & drive quantities.

The SymmWin program will also enforce (depending on the Symmetrix model) the minimum quantity of spares
required.

For example:

DMX-4 and DMX-4 950 with 4 directors:

- 2 spare drives for every 100 physicals (or part thereof) of each drive type.
With a minimum of 8 spare drives for the entire system.
DMX-4 950 with 2 directors:
- 2 spare drives for every 100 physicals (or part thereof) of each drive type.
With a minimum of 4 spare drives for the entire system.

Note that any "hot" spare drive can be used for either traditional "Dynamic Sparing" OR the newer (Enginuity 5x71
and above) "Global" or "Permanent Sparing" feature.

The reason for the increased number and different types of spares is to provide a viable minimum quantity for
Permanent Sparing operations.

In Dynamic Sparing operations ANY available physical spare drive can be used as a 'temporary" mirror for a failing
drive.

With the newer / better Permanent Sparing feature the spare drive becomes the failing drive (i.e. an automatic
configuration change is performed that "swaps" the failing drive and Permanent Spare drive in the Symmetrix System
bin file).

Therefore Permanent Sparing operations require that the physical spare drive matches the physical attributes of the
failing drive AND that this spare drive is in a suitable "backend" location to fully maintain hardware fault tolerance
otherwise the Permanent Sparing operation will NOT be performed (and we use traditional Dynamic Sparing until the
failing drive can be replaced).

Generated by Clearspace on 2010-01-13-07:00

3
Info regarding DMX

The EMC Sales Tools and SymmWin program cater for the Permanent Sparing minimum requirements (minimum
spare quantities) regardless of whether the feature is currently enabled or not.

I hope this helps....

Best Regards,
Michael.

rgunda 40 posts since

May 28, 2008 Reply 6. Re: Info regarding DMX Nov 7, 2008 2:02 PM

in response to: mlee Thanks a lot guys....:)

dynamox 5,727 posts since

Jan 22, 2009 Reply 7. Re: Info regarding DMX Nov 8, 2008 2:36 PM

in response to: mlee Michael,

can you describe benefits of using permanent sparing option ? So the hot spare takes on a roll of the failed drive, so
when the failed drive does get replaced ..data is not copied it to, it simply becomes an available hot spare?

With permanent spare option, let's say i have disk group 2 and i happen to have a drive fail in that disk group, so my
hot spare kicks in. Does it become a member of disk group 2 at this point ?

Thanks

rgunda 40 posts since

May 28, 2008 Reply 8. Re: Info regarding DMX Nov 8, 2008 4:04 PM

in response to: mlee alo can you please let m eknow what do you mean by

AND that this spare drive is in a suitable "backend" location to fully maintain hardware fault tolerance otherwise the
Permanent Sparing operation will NOT be performed ...I am unable to clearly understand this point....

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 9. Re: Info regarding DMX Nov 9, 2008 10:48 AM

in response to: dynamox Dynamox I'm neither Michael nor a "microcode" expert however
I've personally seen permanent swaps and when you replace failed drive, the service processor
builds a brand new binfile (at least that's what happened when I replaced a failed drive)

, thus I suspect that hotspare becomes part of your disk group 2 while replaced drive becomes part of hotspares
group...

Regarding advantages of permanent sparing .. You have less data moving around the backend

.. Replacing a drive with "good old" dynamic spare requires a first rebuild/copy of data from failed/failing drive to
hotspare and later another copy when you replace failed drive. With permanent sparing you rebuild/copy only once.,

Generated by Clearspace on 2010-01-13-07:00

4
Info regarding DMX

actually reducing time required to rebuild integrity of your data. You obviously need more drives (but I'll give better
details answering another questin in this same thread).

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 10. Re: Info regarding DMX Nov 9, 2008 11:52 AM

in response to: rgunda Just as told before, I'm neither Michael nor a "microcode expert" .. However as already told
elsewhere, let's pick a RAID5 volume as an example.
Building a RAID volume requires 4 / 8 / 16 slices on different drives. But the slices needs to
be on different loops, in different power zones, on different processors, to ensure your data is
safe, even in case of failures in the backend. Our code will do all the best to avoid any SPOF
between you and your beloved data. One of the things that matters, is location of slices in the
backend.. But you already know that the three most important things are location, location, location

When the code invokes a permanent hotspare, faces same issues while creating new devices.. Thus the "new" slice
should respect all restrictions coded in the microcode. If the code can't find a suitable drive, the hotspare won't kick
in.

I'm pretty sure Michael will be more then happy to give an overview of all the different rules our
code follows while creating devices.. But in a word: our microcode DO care about your data

Message was edited by:

Stefano Del Corno

dynamox 5,727 posts since

Jan 22, 2009 Reply 11. Re: Info regarding DMX Nov 9, 2008 6:12 PM

in response to: Stefano Del Corno When the code invokes a permanent hotspare, faces
same issues while creating new devices.. Thus the
"new" slice should respect all restrictions coded in
the microcode. If the code can't find a suitable
drive, the hotspare won't kick in.

won't kick in at all or will revert to dynamic sparing ?

mlee 83 posts since

Nov 2, 2006 Reply 12. Re: Info regarding DMX Nov 9, 2008 8:26 PM

in response to: dynamox Hi Stefano,

I am only a SymmWin & Configuration "expert"

. Are you sure you are not a microcode expert (you seem to know this at least as well as I do

Generated by Clearspace on 2010-01-13-07:00

5
Info regarding DMX

)..... I am really only expanding on your earlier comments:

Hello All,

First to summarize:

One or more traditional "hot" or Dynamic Spare will be invoked by the microcode to ALL hypers of a failing drive as a
temporary ("floating") mirror to ensure that all data is protected until the failing drive can be replaced by an EMC CE.

A Permanent Sparing "swap" is different. It will update the backend layout of the Symmetrix System bin file so
that a suitable spare drive will take on ALL the attributes of the failing drive. The Symmetrix volume numbers are
unchanged, all device attributes, channel assignments, meta & SRDF relationships, etc, are completely untouched.
Consider the configuration change as analogous to (but NOT the same as) an "Optimizer" swap....

Yes, the BIG benefit of Permanent Sparing is that we copy the data ONCE:

Once the automated configuration change (i.e. Permanent Sparing "swap") has occurred the spare becomes the
failing drive and the data is copied / rebuilt from the protection device(s).

One minor complication - IF there were any unprotected hypers on the failing drive an additional Dynamic Spare
would have been temporarily invoked before and during the automated swap to protect this data (and act as the copy
source for this unprotected data after the swap).

- SO the benefits that flow on from this are that we do not need / use ANY mirror positions for a permanent spare
(assuming that all hypers on the failing drive were originally protected). Therefore no impact on TimeFinder Mirror
operations, no impact on SYMCLI scripts (due to an invoked "hot" spare).

- AND even if the additional Dynamic Spare IS required (temporarily) it only copies the data from the unprotected
hypers of the failing drive (NOT the entire drive). The Dynamic Spare only remains invoked for the duration of the
automated swap and data copy / rebuild process (it does not require any EMC CE or PSE Lab intervention to be
split). Therefore we automatically minimize the impact on TimeFinder Mirror operations and SYMCLI changes.

With an automated configuration change and only ONE data copy / rebuild we minimize performance impact,
we minimize the inconvenience of the traditionally invoked "hot" spare and we minimize the time that any
data is unprotected.

- Finally the failed drives can now be replaced at ANY convenient time (the "bad" drive is now flagged as a "Not
Ready" spare in the Symmetrix System bin file). An EMC CE does not need to visit the Data Centre as a matter of
urgency. The Not Ready spare can be replaced at ANY time with no impact (since there is NO data copy or rebuild
required).

- One further advantage of Permanent Sparing is the option to now use the "EMC Certified Data Erasure" feature.
This billable feature will erase the data on the failed drive (now a Not Ready spare) to US DoD specifications before
it is removed from site....

The "disadvantage" of Permanent Sparing is the need for additional spares in the Symmetrix bin file:

- A spare is a "spare", that is it can be used by the microcode as a either a Dynamic Spare or as a Permanent Spare
BUT as previously discussed we need suitably located spares for every drive type (SSD or HDD), size, speed &
format (512 or 520 byte) in the configuration to make Permanent Sparing a viable option.

Now this rolls onto the second question (or really a requested clarification):

Generated by Clearspace on 2010-01-13-07:00

6
Info regarding DMX

As Stefano has already stated when building Raid-5/6 or Mirror protected volumes the SymmWin program selects
suitable backend locations for the Raid members or mirrored pairs.

- The rules ARE complex (and vary with Symmetrix model & Enginuity family) but are designed by Engineering to
avoid single points of failure. Therefore the SAME stringent requirements that apply when creating NEW logical
volumes are also applied to the "relocation" of ALL hypers present on the failing drive.

- Basically the SymmWin program will look at the physical backend location of the failing drive and then look at the
location of all of the suitable (same type, size, speed & format) and "Ready" spares to see which one meets our fault
tolerance "rules" for every hyper of the failing drive.

- If SymmWin fails to find a suitable candidate (i.e. no "Ready" spares of the same type, size, speed & format OR in a
suitable backend location) then the microcode will automatically invoke Dynamic Sparing:

One or more traditional "hot" or Dynamic Spare will be invoked by the microcode to ALL hypers of a failing drive as a
temporary ("floating") mirror to ensure that all data is protected until the failing drive can be replaced by an EMC CE.

- With Dynamic Sparing the microcode will invoke (almost) ANY ready spare as a temporary "floating" mirror to
protect the data. The Dynamic Sparing rules allow any spare of any size or speed to be invoked (we still need spare
drives of the same type & format).

- The physical backend location of a Dynamic Spare relative to the failing drive is NOT a concern. However, we in
configuration have always recommend a minimum of two traditional "hot" spare drives per Symmetrix to avoid the
situation where a Dynamic Spare is located on the SAME DA slice as the failing drive - i.e. invoking a dynamic spare
in this circumstance would cause a severe performance impact.

O.K. I hope this helps clarify all the mysteries.

This posting is much longer than I intended but you asked for it

However, if you still want more information I highly recommend the excellent "Drive Sparing in EMC Symmetrix
DMX-3 and DMX-4 Systems" White Paper from EMC Applied Technology (available on Powerlink).

Best Regards,
Michael.

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 13. Re: Info regarding DMX Nov 9, 2008 11:37 PM

in response to: mlee Hi Stefano,

I am only a SymmWin & Configuration "expert"

.
Are you sure you are not a microcode expert
(you seem to know this at least as well as I do

)..... I am really only expanding on your earlier

comments:

Generated by Clearspace on 2010-01-13-07:00

7
Info regarding DMX

Hi Michael .. it looks like you REALLY expanded my 2 cents .. now we have a million dollar answer

I'm sure I'm neither a microcode expert nor a symwin&config expert .. Probably I simply read carefully some good
documents someone gave me

dynamox 5,727 posts since

Jan 22, 2009 Reply 14. Re: Info regarding DMX Nov 16, 2008 6:57 AM

in response to: mlee Michael,

thank you for very detailed explanation ..i do have a follow up question.

One minor complication - IF there were any unprotected hypers on the failing drive an additional Dynamic Spare
>would have been temporarily invoked before and during the automated swap to protect this data (and act as the
>copy source for this unprotected data after the swap)

so this is pro-active sparing ? The code is suspecting the driving is getting ready to fail so the code invokes hot
spare. But if the drive really spun down and won't spin back up, the unprotected hypers are gone. Is that a true
statement ?

Thanks

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 15. Re: Info regarding DMX Nov 16, 2008 4:33 PM

in response to: dynamox Hi Dynamox ....

so this is pro-active sparing ? The code is

suspecting the driving is getting ready to fail so
the code invokes hot spare.

I'm not Michael .. however sometime it happens .. sometime the code detects a failing drive (before it actually fails) ..

But if the drive really

spun down and won't spin back up, the unprotected
hypers are gone. Is that a true statement ?

If you have "real" (unprotected) BCVs, your statement is true.

I had to re-establish huge metavolumes due to a failed drive (and its unprotected tiny slice)...

-s-

Generated by Clearspace on 2010-01-13-07:00

8
Info regarding DMX

mlee 83 posts since

Nov 2, 2006 Reply 16. Re: Info regarding DMX Nov 16, 2008 7:49 PM

in response to: dynamox Hi Dynamox,

A real hardware expert may like to add some comments here but my understanding is that you are correct on both
counts.

We do (in a way) perform "pro-active" sparing since a failing drive will typically log errors (detected by the Symmetrix
microcode) well before it goes totally out-of-service. Some media errors and drive faults are transient & recoverable
but sufficient to invoke Dynamic or Permanent Sparing because a drive has started to go "bad".

However, you are also correct in that drive hardware may fail completely and suddenly (with little to no warning).
The drive may not be recoverable either automatically or manually so YES your unprotected BCV data will
have been lost. That is the risk you take with unprotected hypers (which is why they are called unprotected

) and this is why your friendly EMC Sales (and Support) Representative(s) would recommend mirrored (or Raid)
protected BCV's.

Also note that Permanent Sparing won't operate (won't perform the swap) if the
temporary Dynamic Spare has failed to successfully invoke against all the unprotected
hypers of the failing drive (another reason to ensure that you use protected BCV's)

Best Regards,
Michael.

SKT 717 posts since

Dec 25, 2007 Reply 17. Re: Info regarding DMX Dec 4, 2008 12:59 PM

in response to: mlee Nice explantion in Permanent spare!!

Stefano Del Corno 3,138 posts since

Jun 20, 2007 Reply 18. Re: Info regarding DMX Dec 4, 2008 1:59 PM

in response to: SKT That's why we all love Michael !!!

That's why I'm here to answer easy questions while we page Michael when we have interesting questions

Generated by Clearspace on 2010-01-13-07:00

9