Data Warehousing

Discover Teradata Meta Data Services

Get a Broadened View of the Data in Your Warehouse

By: Rudy Ezquerro

Discover Teradata Meta Data Services

Table of Contents
Executive Summary Introduction Discover Meta Data Services Putting Metadata to Work What Makes MDS Unique? Conclusion 2 3 3 7 8 10

Executive Summary It is commonly recognized that in technology, the only constant is change. Nowhere is this more evident than in a data warehouse where, by its very nature, you have a dynamic and complex environment being utilized by a variety of users, such as administrators, analysts, designers, business users, programmers, and even executives. Todays fast changing business environment creates large volumes of data. Although data are an indispensable resource for decision making, value is obtained from data when all of the datas characteristics, meaning, and context are known and made available to users of the warehouse so they can extract information and knowledge from the data. Data warehouse users nd themselves wanting answers to questions such as: > What do these data mean? > When were these data loaded? > Who owns these data? > What is the datas format? > What is the business description for these data? > What business rules are applied to these data? > How were the data calculated? > How were the data transformed? > What items will be impacted if I change this column? This paper will describe how you can track and identify the wealth of information about the data in your warehouse to optimize the data warehouse return on investment (ROI).

EB-5360

>

1007

>

PAGE 2 OF 10

Discover Teradata Meta Data Services

Introduction
You cant effectively use the abundance of data in your data warehouse if you dont have a clear understanding of the data, what they represent, how they interrelate, and therefore how to best use them. The data warehouse, by its very nature, is a complex and dynamic environment containing vast amounts of data. Arguably the data in the warehouse and the knowledge derived from it are your organizations most valuable asset, and metadata is essential for understanding the information in the data warehouse. The most effective use of the data warehouse is made through the use of metadata. It helps an organization identify, describe, link, and disseminate the proper information to all users of the warehouse. Metadata is increasingly noted as a critical factor for understanding the warehouse as it helps uncover the data that are useful, and just as importantly, the data that are not. A metadata management solution can provide: > Standardized data denitions. > Information sharing across the enterprise. > Impact analysis and reveal relationships between data elements. MDS allows users to identify, consolidate, understand, manage, and navigate technical, business, and lineage metadata and provides facilities to integrate related metadata from other sources. MDS makes it possible to provide both technical and business context for the data in the warehouse. MDS helps identify impact analysis, data redundancy, data relationships, change management, standardize denitions, data governance, and provides extensibility to meet additional needs. MDS Repository The MDS repository is a centrally located and maintained set of tables, views, and macros in Teradata Database accessed via ODBC from the MDS Engine on a client. Metadata is stored in the repository as objects in tables. > Information about redundancies before they occur and eliminate duplication. > Data lineage information. > Consistency and quality. > Adherence to compliance and governance requirements. > Employee self service for search and discovery of information. MDS is tightly integrated with Teradata Database and can be easily extended to meet additional metadata management needs. MDS is one of the most extensible and customizable metadata solutions. MDS allows users to amend or extend the metamodels predened in the repository, as well as the ability to dene new metamodels to help track industry, application, tool, process, or any other type of metadata.

Discover Meta Data Services

Teradata Meta Data Services (MDS) is a comprehensive solution for managing technical and business metadata in a Teradata system environment. MDS integrates metadata throughout the data warehouse, providing a single view of the business.

Solution Components Teradata MDS is a client-server based solution. The Teradata Database is the server that houses the metadata repository. MDS is a client component providing utilities to create, load, search, modify, and administer the MDS metadata repository, as well as to program and script interfaces to read and write into the repository. MDS provides various administrative and support tools and utilities to create, manage, and modify metadata in the MDS repository.

EB-5360

>

1007

>

PAGE 3 OF 10

Discover Teradata Meta Data Services

User and Administration GUIs

MDS Utilities

Custom Applications MDS APIs

Graphical user interfaces allow users to

BI Tools

view metadata in the repository, make online changes to the contents of the repository, load/unload/synchronize

XML Bridge

Teradata metadata, dene users, dene and assign security restrictions, and perform conguration changes. Web-based Access (MetaSurf) MetaSurf is a zero footprint, thin Web client interface across the most popular

Auto DIM Update

MDS Engine

Teradata Solution
MDS Repository
Figure 1. MDS Client.

browser platforms that provides a view into the metadata in the repository. MetaSurf includes the ability to search, drill-down, compare, navigate, and update the metadata in the repository, and it can be customized. Through MetaSurf, you can also generate Microsoft Excel reports of the metadata in the MDS repository.

MDS Engine The MDS engine is a client-side library that performs services to persist and retrieve metadata from the repository. It converts object calls to relational form, and is responsible for managing transactions, security, and generating the SQL to access the tables, views, and macros that make up the MDS repository. Application Information Metamodels An Application Information Metamodel (AIM) is a metamodel that denes how metadata are stored and associated to other metadata in the MDS repository. MDS includes several predened metamodels:

Database Information Metamodel (DIM): stores the Teradata Data Dictionary information, as well as any associated business data. It is used to store physical data model and logical data model information.

If your business intelligence tools are Web-enabled, then you can also embed access to read additional metadata from the MDS repository. Automatic DIM Update

Client Load Metamodel (CLM): stores metadata obtained from Teradata client FastLoad, MultiLoad, or TPump utility scripts and output les. Common Warehouse Metamodel (CWM): Standards-based metamodel that stores metadata from data warehouse and business intelligence tools. Custom metamodels can also be dened to store metadata from a variety of external sources.

Automatic DIM Update is an optional feature that dynamically keeps the MDS repository synchronized to reect Data Dictionary changes made to one or more Teradata Database systems. Application Programming Interfaces Various application programming interfaces (APIs) are provided to import and export metadata. The inter-faces allow you to read, create, update, and delete metadata objects and AIMs in the MDS

EB-5360

>

1007

>

PAGE 4 OF 10

Discover Teradata Meta Data Services

Additional MDS Utilities Additional utilities are provided to create, delete, load, and migrate an MDS repository. Product Features Teradata MDS enables technical and business users to understand the data in the warehouse. How useful are data if they cant be located or understood? MDS delivers an integrated view of the information in the data warehouse and provides users with common business denitions, enabling users to nd and act on information in the warehouse. This helps users become self sufcient, improves data management, and reduces reliance on IT staff. MDS provides a variety of ways to load metadata into the repository, such as graphical utilities, Excel spreadsheets, Visual Basic applications, XML scripts, and manually. In turn, access
Figure 2. Metasurf Screen.

to the data in the repository is based on the security permissions of each user.

repository, dene users or groups, and dene and assign security restrictions. If another metadata management solution is already in place, then the MDS APIs may be used to interchange Teradata metadata with the other metadata repository. XML An XML interface is offered, in addition to the APIs, as a standard alternative method for metadata interchange. XML interface provides import and edit capabilities to the metadata stored in the MDS reposi-

tory, thus providing a non-programmatic interface for exchanging metadata with third-party tools. Teradata Client Load Scripts The Metaclient utility extracts and loads metadata from Teradata FastLoad, MultiLoad, or TPump load scripts and output les. Metadata from Teradata load jobs is then linked to the associated warehouse table, view, and column metadata within the repository providing the foundation for lineage metadata management.

Metadata Versioning The MDS repository can retain historical information about data stored in the repository. This historical information tracks changes made to metadata objects over time, allowing you to have traceability of the metadata, view other associated historical metadata, and identify changes that were made between versions of the metadata.

EB-5360

>

1007

>

PAGE 5 OF 10

Discover Teradata Meta Data Services

Security You can set security policies and rules for users or groups of users to limit access into the contents of the repository and enforce privacy regulations. A exible security policy provides the ability to grant user access at either the metamodel level, the metadata class level, or all the way down to a specic metadata object instance. Search and Navigation Using the MDS Web-based access (MetaSurf), a user can issue a free-text search by simply specifying a business term to search, or a more technical user can issue a elded-search where elds in a form are specied. This allows users to search for general or specic data in the repository. Any level of user can easily navigate and analyze the data in the repository and discover interrelated data. Standards-based XML import The XML import functionality provides the ability to import XML generated by third-party tools into the MDS repository. Extensible Stylesheet Language Transformations (XSLT) is used to transform the tools XML into the XML schema supported by MDS. The transformed XML is then imported into the MDS repository. Thirdparty tools include business information tools or modeling tools, which can export their metadata to an XML le. Logical Data Models and Business Descriptions Logical information, such as Attributes, Entities, and Subject Areas from ERwin Business denitions can also be loaded in to the repository using MetaSurf, XML scripts, Excel worksheet, manually, or through the any of the MDS programming interfaces. models, can be loaded into the MDS repository. This reveals how it is associated to physical columns and tables from the warehouse. Users get a complete view of the logical and physical metadata and how they are linked. As an example, the following business questions could be easily answered: > What does gross sales mean? > What is the business denition for this item? > What is the meaning of each of these attributes? > What table is associated to this business entity? > What business rules are dened?
Figure 3.

EB-5360

>

1007

>

PAGE 6 OF 10

Discover Teradata Meta Data Services

Extensibility MDS was designed from the start to be fully extensible, with various interfaces to support access through a variety of environments. Bundled utilities provide the ability to easily customize predened metamodels or to create new metamodels to meet specic needs. For instance, you can add or modify: > Types of metadata. > Metadata properties. > Relationships between types of metadata. > Metamodel denitions. > Users or groups or users. > Security settings. And extensions that you, as a user, make to the MDS repository are maintained in future MDS product upgrades. In other words, you do not lose your customizations as you advance to new releases of MDS. Change Management Facilities are provided to create logs that keep information about who, when, and what changes were made to the Data Dictionary. You can also identify who is accessing the repository, what metadata objects are being accessed, what was changed, and when was it changed. Sample Architecture The architecture for implementing MDS is straightforward. The MDS repository is a database that resides on the Teradata Clients use standard web browsers to access the metadata in the MDS repository. MDS software is not required on the clients, except on clients that directly use the MDS APIs. Database. The major components of an MDS conguration are as follows: The Value of Metadata Microsoft Windows-based workstation is used to perform MDS administrative tasks, such as load or resynchronize Teradata Database metadata, manage users, manage security settings, view information logs, and modify conguration settings. Windows Web Server, commonly Microsoft IIS, has MDS installed, including the MetaSurf Web-enabled application. Business users are demanding more semantic content to the data stored in the warehouse along with the ability to search the data using business terms they understand. Teradata MDS is positioned to help you deliver the content and functionality that your business users and IT users need. By implementing MDS, a company can establish consistent business terminology, a corporate glossary, and an information catalog for the warehouse helping to reach consistent taxonomy. Companies need to know how their data are being used and how those data can be used as a competitive advantage. MDS helps you maintain an efcient and

MDS Administrator

Web Clients

Web Server

Teradata Server
Figure 4.

Putting Metadata to Work

EB-5360

>

1007

>

PAGE 7 OF 10

Discover Teradata Meta Data Services

Governance Data stewardship and governance may be based on metadata to help identify

Metadata Repository Technical Metadata Business Metadata

ownership, traceability, and accountability in order to meet regulatory initiatives. Managed metadata plays a fundamental role in governance as it provides a persistent store to manage enterprise data denitions, processes and procedures, and other information assets that can be administered by assigned stewards.

What Makes MDS Unique?

MDS is the only metadata management solution optimized for and integrated with Teradata. The metadata repository is a set of tables, views, and macros stored in a Teradata Database. MDS provides a blueprint of the warehouse data architecture, effective way to locate, access, understand, and manage data. With MDS, the task of analyzing and implementing changes to the Teradata system can be signicantly less difcult and time consuming. Technical Metadata Technical metadata provides a detailed blueprint of the warehouse, facilitates warehouse maintenance, and helps to accurately predict impact due to changes to the warehouse. Technical metadata can provide the following information: > Physical table and column names, data types, and sizes. > Structure of views, indexes, and macros. > Source and ETL process information. > Physical warehouse model. Business Metadata Business metadata is the description of the data needed by the business users to understand the context and meaning of the data allowing business users to use the warehouse data more effectively. Business metadata provides end users with easyto-understand views of data warehouse information. Business metadata can provide the following information: > Consistent business terms and denitions for tables and columns. > Business description for data elements. > Business rules and valid values. > Subject area descriptions. > Origin of the data. > Logical data model. MDS makes it possible to manage both technical and business context for the data in the warehouse. MDS is included in the Teradata Utility Pack, which is an everyunit-item with Teradata Database. Teradata Support MDS is tightly integrated and optimized for Teradata Database. MDS is a repository database created within the Teradata Database, reads the Teradata Data Dictionary to extract warehouse metadata, supports Teradata-specic extensions, and, MDS can dynamically update the enables users to view the contents of the warehouse, and allows staff to understand the impact from schema changes prior to these changes actually being executed.

Technical Users

Administrators

Business Users
Figure 5.

EB-5360

>

1007

>

PAGE 8 OF 10

Discover Teradata Meta Data Services

repository as changes to the Teradata Data Dictionary are made. MDS supports the latest database features. Automatic DIM Update The Automatic DIM Update is a patented feature that ensures the MDS repository is actively and automatically kept up to date as Teradata Data Dictionary changes occur. When this feature is enabled, it provides dynamic synchronization of the Teradata metadata. Identify Redundancy MDS allows you to manage one or more databases from a Teradata system in a single repository and identify possible duplication of items in the warehouse, such as columns, tables, views, and databases. MDS is also designed to manage metadata from multiple Teradata systems. For instance, you can load Teradata metadata from production and staging systems into a single MDS repository and identify differences.
Figure 6.

Impact Analysis MDS allows you to determine the impact of changes to the warehouse prior to implementation. For example, you can easily determine the impact of the following scenarios: > What data are maintained in this database? > What views are affected if I drop this column? > What tables are referenced by this stored procedure, macro, or view? > Where did this view come from, and what table columns does it reference? > Are there any possible broken views, macros, or stored procedures? > What was the last Teradata DDL change, and when did it take place? View Analysis The SQL used to dene Teradata views, macros, stored procedures, and triggers is analyzed by MDS to identify and store the metadata for base tables and columns referenced. MDS also provides intermediate view information. Extensibility and Customization The same interfaces that are used by MDS own utilities are available to the user. The interfaces support loading, reading, updating, and deleting metadata in the MDS repository. Complete API access provides the exibility needed to create a metadata management solution that meets an organizations specic needs. Extensions to the MDS repository are maintained in future MDS product upgrades.

EB-5360

>

1007

>

PAGE 9 OF 10

Discover Teradata Meta Data Services

Teradata.com
By design, the repository browsing utilities that are included with MDS automatically support custom extensions. In addition, the web application (MetaSurf) is fully customizable allowing you to modify the look and feel and the data content delivered. For example, you can change the look to display corporate colors and logos, or completely change it to match the interface of an existing solution familiar to users. Furthermore, the functionality can be modied to display fewer technical details, more technical details, or to access other web-based applications or enterprise documents. Teradata Data Lineage Information extracted from Teradata client load scripts is linked to the actual column and table metadata. This enables you to identify when and how the warehouse data were loaded, as well as what Teradata load job was used to load the data.

Conclusion
You cant effectively use the data in the warehouse, as well as other sources, if you dont have a clear understanding of the data, what they represent, how they interrelate, and therefore, how to best use them. To make better decisions that positively affect a business, users need information that helps them interpret the data generated in todays business environment. They need effective, accessible metadata that describes the data in the warehouse and other corporate assets. Metadata surrounds raw data with context, transforming them into knowledge. A metadata management solution describes the data in the warehouse, where they are stored, how they relate to other information, where they come from, and how they are related to the business. Without context, data are often meaningless.

Teradata Meta Data Services is a comprehensive solution for managing metadata for a Teradata system. This makes it possible to provide both technical and business context for the data in the warehouse, identify impact analysis, manage change, foster reuse, and provide extensibility to help meet additional metadata management needs. Ultimately good metadata management eliminates the possibility of becoming data rich, but knowledge poor. Whats in your data warehouse? About the Author Rudy Ezquerro is a senior member of the Teradata Meta Data Services development team. Rudy has been with NCR and Teradata for 20 years and has been involved in the architecture and development of MDS since 1999.

This document, which includes the information contained herein, is the exclusive property of Teradata Corporation. Any person is hereby authorized to view, copy, print, and distribute this document subject to the following conditions. This document may be used for non-commercial, informational purposes only and is provided on an AS-IS basis. Any copy of this document or portion thereof must include this copyright notice and all other restrictive legends appearing in this document. Note that any product, process or technology described in the document may be the subject of other intellectual property rights reserved by Teradata and are not licensed hereunder. No license rights will be implied. Use, duplication or disclosure by the United States government is subject to the restrictions set forth in DFARS 252.227-7013 (c) (1) (ii) and FAR 52.227-19. Teradata continually improves products as new technologies and components become available. Teradata, therefore, reserves the right to change specications without prior notice. All features, functions, and operations described herein may not be marketed in all parts of the world. Consult your Teradata representative or Teradata.com for more information. Copyright 2007 by Teradata Corporation All Rights Reserved. Produced in U.S.A.

EB-5360

>

1007

>

PAGE 10 OF 10