Automation of Backups in Teradata Databases

Abstract
Explanation of the approach adopted to mitigate the risk associated with manual synchronization of data across Teradata databases and optimize the process to ensure significant improvement in meeting Data Delivery SLA deadlines.

The client is a major MNC in the HLS vertical with major operations across the globe and Europe had been identified as a critical market for future growth. An essential part of this growth strategy involved critical reporting in Europe. Under the existing DWBI architecture all the data from source systems was being loaded to the Teradata servers in US every day which was then synched to the EU servers. For over two years this activity was undertaken by the prior DBA team wherein the sync activity would be done manually by taking a backup from US servers and restoring the data to the EU servers. However the challenge involved timely delivery of data to Europe and on numerous occasions the data delivery SLAs were breached due to the significant time required to manually build a restore job daily as well as due to occasional DBA errors. The solution to this issue therefore had significant business impact, since timely data delivery on a regular basis ensured stabilization of operational & strategic reporting across Europe. The automation of this process helped reclaim over 20 man hours every week whereas the extensive error notification and reporting built into the solution ensured streamlining & optimization of an essential technical activity. The innovation proved to be one of the major innovations showcased to the client and helped build credibility of the technical team. The potential benefit of sharing this innovation across projects includes significant savings for replication and sync solutions on Teradata platforms in terms of cost as well as in terms of man hours. The amount of data being synchronized was in the order of several GB and the sync activity was from US to EU. Further most of the tables being synced were being loaded using a Truncate & Load methodology. The existing manual sync process used TARAGUI (Teradata Tiered Archive/Restore Architecture) a graphical interface to launch and build backup/restore jobs along with Netbackup, a Backup & Recovery (BAR) Tool. We analyzed and concluded that the existing process had numerous deficiencies as listed below. Completely manual execution from start to end. Restore job scripts need to be built every single time for each job thus increasing the possibilities of error. Every week over 20 man hours were being spent in this activity. Lack of proper notification & detailed BAR reporting.

We chose to bypass using the graphical interface of the TARAGUI and instead built a seamless solution which would integrate the various components of the sync activity as well as automate the overall data delivery process. The solution was named AutoBAR to reflect the automated use of existing BAR Infrastructure and was built using UNIX while invoking the relevant TARA commands from command line. The Sync process mainly contained 6 sections as listed below. The challenge was to integrate each of the below requirements and build a robust and stable solution. Confirmation from ETL regarding completion of loads. Drop aggregate join indexes on target tables. (Since restore operations invalidate aggregate join indexes) Launch backup job for the particular set of tables. After completion of backup, launch restore job using the backup set generated in previous step. After completion of restore, collect statistics for all the restored tables. Re-create the aggregate join indexes on Target tables.

Firstly AutoBAR detects completion of the ETL loads on the basis of an audit table which is populated by the last Informatica workflow each day. The presence of the desired entry in the audit table results in the aggregate join indexes of the target table being dropped & the corresponding preconfigured backup being triggered on the US server. Once the backup completes successfully the AutoBAR script dynamically builds & runs the restore script on the EU servers using the backup set taken in the previous step. After the restore operation ends successfully, statistics are collected for the target tables and aggregate join indexes are re-created. At each step notifications and alerts were configured to enable easy troubleshooting. The below diagram illustrates the process flow.

Process Flow

We encountered the following challenges while developing the solution Integration of Informatica Workflow completion and the subsequent trigger to the AutoBAR solution. Identifying the backupset and building a restore script dynamically to use the identified backupset. Implementing error detection logic at each step and providing comprehensive notification capability.

The AutoBAR solution thus overcame the inherent deficiencies of the earlier manual process and presented the client with an automated, optimized and enhanced mechanism for data delivery across Teradata platforms. The other competitive approach is a product called NPARC (Named Pipes Arcmain) by Teradata Corp which uses named pipes and the Arcmain command to pass data from source to target systems. NPARC doesnt use storage since the data is transferred in real time from source to target. Our solution uses the traditional backup & recovery infrastructure and thereby uses storage thus offering data availability till the specified retention period. Thus our solution leverages the existing technology along with innovative integration to offer similar functionality at no additional cost of licensing or training whereas NPARC is a paid service and includes significant licensing fees.