Dow Ghemicat Tmptomonts Highty Avaitabte Solution for SAP. Cwkonwent, S companies adopt enterprise-wide appli ations w help facili tate operations throughout their organizations, ensuring availabil- ity of these systems is absolutely essential, Applications that were once discrete a location or particular business function can ow be critical t0 the overall business and to thousands of employees that rely on immedi- ate use and access to these appli- ions to serve customers. ‘This is the situation that The Dow Chemical Company, Mid- land, Michigan faced when it migrated its mainframes to an SAP environment running on ‘mainframe DB2 databases, Dow, which, together with its affiliates, has more than 40,000 employees and presence in ‘more than 150 countries, depends on the SAP environment to handle all its critical ‘enterprise financial and supply chain infor- mation worldwide. “Our mainframe SAP_ environment houses our most critical data, it’s relied ‘on for the entire supply chain all over the world, including tracking inventory, ship- ping, sales, onder entry and invoicing,” said Bill Worsley, business continuity manager for Dow Chemical. “Not having access to this application would have a devastat- ing impact on us, bringing operations to a standstill, ‘Additionally, because the system is highly integrated, the loss of any data could have a ripple effect on the rest of the environment. “If we try to reconstruct data from the previous day's backup, we wouldn't know ‘where we left off because a day's worth of data would be lost,” explained Worsley. With this in mind, Dow recognized it needed to establish a recovery point objec tive (RPO) of zero, meaning no data loss, and a stringent recovery time of just four hours. The next step was finding a solu- tion that would meet these requirements. For this, Dow turned to Comdisco to design and implement an availability solu- tion for its SAP environment. Dow had long used Comdisco for hot-site recovery ~—RRDF Sotution Addresses Stringent Recovery Objectives - by Marta Chevere Dow Chemical Production Site ste chu aes solutions and had recently outsourced the management of its continuity program as well. ith a requirement for no or very Tittle data toss and a very short recovery ‘window, the team decided to implement solution using E-Net’s Remote Recovery Data Facility (RRDF) to ensure the avail- ability of Dow's SAP applications. ‘The RRDF software enables recovery to pointof-failure for Dow's mission critical SAP environment using realtime remote journaling and database shadowing 10 8 technology service center. Implementing the Solution Once the decision was made to proceed, the next step was to implement the solu ion Dow's enterprise SAP applications are deployed on two mainframes. One frame runs two DB2 databases to support its operations in North America and the jon; the second mainframe runs two additional DB2 databases to support Dow's operations in Europe and Latin America. To help ensure against a regional disas fer, the recovery environment for Dow's SAP applications is located at an off site technology service center. An added ben- efit of the remote journaling solution is that itis insensitive to distance, allowing the continuity site to be thousands of miles away with little or no additional impact on als ama Jaunaing ot 082 fog over network the production applications, The remote journaling software was installed on both Dow’s production pro- ccessors as well as on dedicated processors at the offsite technology service center: AAs a transaction is processed against any of Dow's DB2 databases at the production location, a duplicate copy of the database log and journal data is captured in real- ‘time and transmitted to the center instantly over a relatively inexpensive network con- sisting of three dedicated TI lines. At the enter, the log and journal information is immediately saved to disk. Several times daily, the disks are archived to tape, allow- ing the disks to be reused while ensuring that the archived data is available should the entire database need to be recreated. send” and “receive” regions. running fon the Dow mainframes. are monitored remotely from the service location. The remote journaling software butlers, filters ‘and compresses DB2 logstreams, thus fully utilizing the available handwvidth, Further more, fully automated spilling and gap recovery features enable speedy recovery from day-to-day link outages, “spikes” in the logging rate, or whatever software and hardware failures might occur. Inaddition to live, real-time remote jour- naling, Dow produces daily backups, also known as “image copies”, of all four of the DB2 databases at its location and ships them offsite to their tape storage pro- vider.DB2 Log Stream Backups ofall regions Recovery of Pacific Region used the previous day's backup tape oH * we have a true high- availability solution Dow's experience shows how versatile and forgiving recoveries can be if companies ave Geeaat” | log data, Many compa point nies back up databases on ered basis, and some use share- level change or “fuzzy" backups. Using a log, This tape was not ‘NI databases cis Sanaa oan avalable at the recovered to elec recovery site this point recovered 10 a consis In the event of a disaster, the offsite tapes are shipped to the offsite facility There, technicians would handle the ini- tial estoration from those tapes and then use 1 DB2 recovery software to do a “roll forward” of the databases, in effect capturing all the transactions that took place from the las tape backup to the point of failu The Sotution is Put to the Tost - Under Unexpected and Potentially Difficult Circumstances Over the past year Dow has successfully tested the solution on several occasions. But it was during one test that Dow real- ized additional benefits ofthe remote jour naling solution The operational procedure controlling the use of DB2 log data needed for remote site recovery had a minor problem. ‘The Pacific region database had already bee backed up on the same day ("Day Two") as the simulated disaster point, so Dow needed log data from that point forward. However this backup tape was not avail able at the recovery site because it had not yet heen ejected from the tape silo t0 be ‘ent off-site. It is not unusual for backup {apes to stay atthe production facility for «period of time, sometimes hours, before they are physically sent off-site. In this case, the simuated disaster point happened to fall at an inconveni ‘The test situation illustrated a disaster recovery planner’s worst nightmar ing the wrong backup tapes or receiving unusable backup tapes. “In a traditional recovery scenario, if you don’t have the right backup tapes to restore from, you t fully recover and you can’t synchro. nize your data,” said Worsley. “We needed to have all the databases reflect ther state as ofthe simulated disaster point time. send Using the remote journaling solution, Dow was able to avert a test failure. As part of the implementation, a process had been established to archive Dow's data on tape and hold these archives for several days - just in case. As a result, Dow was, able to use the older (and available) data base backup tape to do the initial resto- ration, then roll forward using nwo day's worth of DB2 log data from the remote Journals to ultimately reach the DR test’s simulated point of failure, AML databases, including the one for the Pacific region, were recovered to a consistent point in time representing the simulated disaster point. Dow realized during that something was wrong, they had back- ups from “Day One” forall databases, but the log data needed 10 bring the Pacific the recovery test region current to the disaster point provided, Tom Rechsteiner, Dow's data- base administrator, managed the recovery process and recognized that there was i hole in the log stream, caused by the fact that the Pacific regions had been backed up early on “Day Two, ‘To avert the potential loss in data caused by the gap in the Pacific region database and the disaster point, the testing team executed a special, unplanned, reformat process to obtain the log data needed for complete recovery of all the databases, For- tunately for Dow, RRDF provides options for extracting specific ranges of log data, enabling Dow to recover ALL their data bases n adisaster situation, unforeseen issues wn have a significant impact on your abil- ity to recover” said Worsley, “Knowing that we can still successfully recover even I we have the wrong tape oF a tape that has gotten corrupted shipped to the facility -is very reassuring to us in making certain that tent point in time the disaster point ‘The use of “fuzzy” backups means that the databases don’t have to be quiesced for taken offline to make the full daily backup tapes. Too often, backups for con tingency or disaster recovery require out- ages, compromisi wvailability, With the remote journaling solution in place, Dow achieves improved availability in its day-to-day operations as well as complete recovery al the disaster recovery site °, fe Marta Chevere is the Diree- torof Advanced Recovery Ser- vices (ARS) for Comdisea’s Storage Services Group. She oversees the company’s devel- ‘opment and integration of advanced recovery product offerings. Chevere canbe reached at mgchevere@comdisco.com. This article was originally published in the Spring 2001 issue of the _ Disaster. Recovery Journal.< It is reprinted with their permission.