PERL AND THE ORACLE DBA

Jared Still, RadiSys Corporation

Perl came quietly onto the IT scene in 1987. Its goal as described by its creator Larry Wall was to combine the best of C, sed,
awk and the Bourne shell into an easy to use language for text manipulation. Since the time of its initial release Perl has gone
through 4 major revisions and succeeded far beyond its authors expectations. The current stable version of Perl is 5.8.6 as of
the time this writing. When you think about it, Perl is now old enough to have attained ‘legacy‘ status.
Perl was designed to be easily extended through the use of modules. Perl modules allow Perl code to be easily packaged for
reuse. There are literally thousand of Perl modules available through CPAN (Comprehensive Perl Archive Network) designed
for a variety of purposes. There are modules available for converting data between different formats, file handling, time and
date conversions, email, interfaces to commercial software, security, graphics and many other topics. The modules that make
Perl most interesting to DBA’s are the DBI (DataBase Interface) and DBD (DataBase Driver) modules.

Early Use of Perl With Oracle

In the early part of the 1990’s, Kevin Stock created a version of Perl that was linked with Oracle’s OCI libraries to allow
direct access to Oracle databases from Perl. This was prior to the release of Perl 5. Perl 4 modules did not have the extensive
capabilities that Perl 5 now has, requiring that the Oracle libraries be linked to Perl at compile time. The result was known as
Oraperl.
Oraperl made possible to use the extensive capabilities of Perl while connected to an Oracle database. This doesn’t really
sound too revolutionary until you consider that prior to Oraperl it was necessary to write external programs in C or some
other 3GL language. Oraperl provided most of the power of C, along with the powerful features of sh, sed and awk in an
easy to use scripting language. The powerful constructs of Perl made it possible to write programs using less code than in C,
and did not require the edit, compile, link routine when developing, instead using a runtime pcode compiler
Creating Oracle database utilities became much easier for Oracle DBA’s familiar with C. For those not familiar with C, Perl
was an easier language to learn than C.
In the early 1990’s (at a previous employer) we were searching for a tool that would allow us to easily write reports based on
pharmaceutical claims data. Some commercial products were reviewed and rejected as being either too costly or too
unwieldy. SQL*Plus has excellent report formatting abilities, but was lacking in its ability to make many complex
computations at runtime. Using PL/SQL was not an option, as the combination of PL/SQL and SQL*Plus lacked the
performance ability required at that time. PL/SQL functions used inline in SQL statements performed somewhat poorly in
Oracle 7. There was also the issue of controlling printers from the report, as the printer chosen at runtime had to be sent the
correct codes to set the printer fonts, something that could not easily be done from SQL*Plus.
It was suggested that Perl be considered, as it appeared to have much of the reporting functionality of commercial tools that
had been examined, and contained many features that commercial packages were lacking. The ability to directly control
printers was vital. The fact the multiple database connections could be made was a nice feature. Of economic importance
was the fact that Perl was freely available (this was a small company). Several contractors were hired to work on our project,
and as they were all proficient in C, they had no trouble learning to use Perl.
The manager of the project had been hesitant at first, but was swayed by the economic arguments.
What has been described so far may not appear to be DBA duties, but I have found that what constitutes DBA duties varies
with the company where one is employed, with company size being a strong influence on that. Small shops often call on the
DBA to provide functionality and expertise outside the realm of classically defined DBA duties.
Our use of Perl was so successful that we began using it for many tasks. The next project was to create backup routines to
regularly backup our Oracle databases, something more in line with traditional DBA tasks. We had been managing backups
using shell scripts, but something more robust was needed. A fairly robust Perl module (Perl 4 at that time) was created to

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

manage the backup of Oracle database, both cold and hot backups. Known as hotback1, it was in use for several years at this
site.

DBI and DBD

The release of Perl 5 in October 1994 brought about many improvements. Among them was the ability to dynamically load
shared libraries at runtime. Also released in 1994 were DBI and DBD::Oracle, authored by Tim Bunce. The new release of
Perl also permitted object oriented programming, making it possible to write easily reusable Perl modules.
DBI was written as a generic database interface. This allowed for Perl scripts that used DBI to be written to the same
interface regardless of database vendor. A separate DBD driver module would be written for each database vendor’s product,
enabling the architecture seen in Figure 1.

DBD::mysql MySQL

DBI DBD::Oracle Oracle

DBD::pg PostgreSQL

Figure 1. Perl DBI and DBD::Architecture

Fortunately for Oracle users, the DBD::Oracle interface was also released in 1994, as the author of the DBI also wrote the
DBD::Oracle module2. The use of Perl with databases, and in particular Oracle databases, began to increase dramatically
with this new release of Perl and the release of the DBI and DBD::Oracle modules.
What does all this mean for the DBA?

Why You Should Use Perl

A DBA is required to use many different technical skills. Besides being conversant in SQL, PL/SQL, database design and
implementation, there are also many tasks that require scripting skills. The tools of choice have traditionally been the Bourne
or Korn shell3 with their built in programmatic elements, along with such tools as sed, awk and other command line utilities.
The DBA that works on the Win32 platform has a somewhat more limited choice of tools, at least as supplied with the
operating system.
1
Still available at http://www.jaredstill.com/ at the time of this writing.
2
Tim Bunce is still maintaining DBI and DBD::Oracle, as of December 2004.
3
And increasingly the Bash shell with the proliferation of Linux.

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

Perl has made possible the creation of DBA utilities that previously required more traditional languages such as C. Complex
data structures unavailable in shell scripting as well as more complex control structures make it an ideal scripting language
for utility use. The fact that many Perl scripts can work on UNIX, Linux and Win32 without modification is also a valuable
asset. Many a DBA must support Oracle on multiple platforms. No wonder Perl is referred to as the “Swiss Army Knife” of
programming languages.
Though there are many features that separate Perl from the more traditional scripting tools, there are two features of
DBD::Oracle that may be particularly attractive to a DBA.
• Multiple connections to multiple databases.
• The ability to work with LONG data types.
When faced with querying data from another database in SQL*Plus while maintaining the current session database
connection, it is necessary to use a database link. While this usually works well its intended use in applications, it is
somewhat unwieldy for the DBA maintaining many databases. The database connections may not be known until runtime. It
is very convenient to simply be able to open another connection to another database.
Another limitation of SQL*Plus and PL/SQL is when dealing with LONG data types. PL/SQL can manipulate a LONG
column only up to 32k in length. Using Perl with DBI it is possible to manipulate LONG data types without resorting to C.
Though modern applications should be using the more modern LOB data types, there are still many applications that rely on
the long data type.
Here are a few of the many features of Perl that make it a powerful addition to the DBA tool box:
• Powerful command line argument parsing
• Direct control of many OS features and protocols such as ockets, ftp, ssh, file handling, semaphores and IPC
• Interaction with the user
• Speed

The Nitty Gritty

Let’s take a look at a simple but complete Perl script makes a connection to a database and queries the DBA_USERS
view. Though this is a relatively short script at ~120 lines, it is fairly robust. The following elements are included in this
script:

• Parses command line arguments

• Creates a database connection
• Creates calls to Oracle to create a cursor, execute it and fetch data.
• Displays a usage screen if there is an error on the command line
• Will produce either a CSV output or a formatted report
• Includes documentation available via the perldoc script

The first 16 lines of the script appear in Example 1. If you are unfamiliar with Unix systems, line 1 may appear somewhat
strange. This known as the magic cookie in shell scripting, and is used to inform the Unix kernel which shell is used to
execute a script, in this case the shell is Perl. The –w tells Perl to include warnings.
Lines 5-10 make up some simple internal documentation. This can be viewed by using the perldoc command. Many Perl
modules contain extensive internal documentation. To view the documentation for the sample script you would use the
command perldoc simple.pl.
At lines 12-15 the Perl modules required for this script are loaded. Notice that DBD::Oracle is not mentioned. The DBI will
automatically load the correct driver at the time the database connection is made.

1: #!/usr/bin/perl -w

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

2:
3:=head1 simple.pl
4:
5: This this simple script queries the table DBA_USERS and outputs
6: the results in either a formatted report or a CSV file.
7:
8: Use simple.pl -help for an example
9:
10:=cut
11:
12:use warnings;
13:use strict;
14:use DBI;
15:use Getopt::Long;
16:
Example 1. simple.pl lines 1-16

The next section in Example 2 consists of lines 17-50 where the command line arguments are processed. At lines 18-24 the
command line processor is called. Line 26 calls the usage routine if the –help argument was given on the command line.
Lines 28-32 are used to assign the command lines options to variables. This is not a required step, it just sets up easier to use
variable names. Lines 34 through 37 then check to ensure that required variable are set, and will exit with an error code if
not.
This script will produce two types of output, CSV formatted output and a standard formatted report. The code at lines 41-43
assigns the –output_type command line option to a variable $outType if it was included, and sets $outType to a default of
‘report’ if it was not included. Line 45 uses Perl’s match operator to ensure that the value specified for the output type was
either ‘report’ or ‘csv’, and will cause the script to exit with an error if there is not a match.
17:my %optctl = ();
18:Getopt::Long::GetOptions( \%optctl,
19: "database=s", # = indicates required
20: "username=s",
21: "password=s",
22: "output_type:s", # : indicates optional
23: "help"
24:);
25:
26:usage(0) if $optctl{help};
27:
28:my($db, $username, $password) = (
29: $optctl{database},
30: $optctl{username},
31: $optctl{password}
32:);
33:
34:usage(1) unless (
35: $db
36: and $username
37: and $password
38:);
39:
40:# set output type - use default if not set on cmd line
41:my $outType = defined $optctl{output_type}
42: ? $optctl{output_type}
43: : 'report';
44:
45:unless ( $outType =~ /^report|csv$/ ){
46: print "type: $outType\n";
47: warn "incorrect -output_type specified\n";
48: usage(2);
49:}
50:
Example 2. simple.pl lines 17-50

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

Now in Example 3 we get to the interesting bits of the script. Lines 52-59 create the connection to the database. The database
is specified via a data source that consists of ‘dbi’, the name of the database driver, plus the connect string for the database in
a colon separated string. Notice lines 56 and 57 where the RaiseError and AutoCommit parameters are set. When the
RaiseError parameter is set, all errors raised by Oracle will be raised in the Perl script as well, causing it to die. If set to 0
the error will be displayed, but it will be up to the script to handle the error. Line 61 will cause the script to die if RaiseError
is set to 0 and the database connection fails, though with RaiseError set to 1 line 61 will never execute. The AutoCommit
parameters controls whether or not a commit will be automatically issued after execution of every DML statement. It is
recommended that this be explicitly set in your Perl scripts, preferably to a value of 0.
The PrintError parameter may also be used here, but is not shown. Its default value is 1 or true; you may prevent Oracle
errors from being printed by setting it to 0.
If you are familiar with the set arraysize command in SQL*Plus, then line 64 will be familiar, as it serves the same function,
that is, instructing Oracle to return data 100 rows at a time, rather than a single row at a time. Your network administrator
will thank you for your thoughtfulness in setting this.
The remaining lines 69-75 are fairly self explanatory. Set the default date format for the session, create a SQL statement,
send it to the database to be parsed, and finally execute it. These will be familiar steps if you have ever used the DBMS_SQL
package.

51:# create a database handle

52:my $dbh = DBI->connect(
53: 'dbi:Oracle:' . $db,
54: $username, $password,
55: {
56: RaiseError => 1,
57: AutoCommit => 0
58: }
59: );
60:
61:die "Connect to $db failed \n" unless $dbh;
62:
63:# set the array fetch size
64:$dbh->{RowCacheSize} = 100;
65:
66:# set default Oracle date format
67:$dbh->do(q{alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss'});
68:
69:my $sql=qq{select username,created,lock_date,expiry_date from dba_users};
70:
71:# create a statement handle
72:my $sth = $dbh->prepare($sql);
73:
74:# execute the SQL
75:$sth->execute;
76:
Example 3. simple.pl – lines 51-76

Example 4 contains the section of code that prints the column names, the report data, and disconnects from the database. Line
78 prints the names of the columns. These are retrieved from the NAME statement handle attribute, which return an array
reference, which is similar to a pointer to an array. The join function is used to format a single string made up of each
element of the array separated by a comma.
The data is retrieved from the database in line 81 by using the fetchrow_array method. There are several methods available
for fetching data, this one has the virtue of being the simplest to work with. Data is returned into the simple array @result.
Line 83 then checks the array for any NULL values (Oracle equates a NULL with an empty string), and assigns a value of
‘N/A’ to those elements of the array. This is to prevent Perl from complaining about working with undefined variables later
on. Any value could be assigned to the NULL elements, even an empty string. This line requires a little more explanation.
Some folks raise an issue that Perl is difficult to work with, and may point to the map function as an example. The map
function is very powerful, and powerful features sometimes require a little more effort to understand. In this case it is being

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

used to examine every element in the array. Perl uses the $_ variable as to capture output of many functions when no specific
variable is specified, or as in this case, may be specified. As each element is assigned to $_ it is then checked to see if there is
a value defined for it, if not, then ‘N/A’ is assigned to it. The output of map is used to populate the @data array. This could
also have been used to repopulate the @result array, but a new array was used for clarity. Functionally this use of map is
equivalent to the following loop:
foreach my $el ( 0..$#result){
$data[$el] = $result[$el] ? $result[$el] : 'N/A';
}

Lines 85–89 check which form of output was specified on the command line and print the data accordingly.
When the loop is complete, line 92 disconnects from the Oracle database.

77:#print column names

78:print join(',',@{$sth->{NAME}}),"\n" if ‘csv’ eq $outType;
79:
80:my @data=();
81:while( my @result = $sth->fetchrow_array ) {
82: # convert NULL values to 'N/A'
83: @data = map ( defined($_) ? $_ : 'N/A', @result);
84:
85: # write the data as formatted output
86: if ('report' eq $outType) { write }
87: # or print it CSV style
88: elsif ('csv' eq $outType) { print join(',',@data), "\n" }
89: else {die "Error! - Invalid output type\n"}
90:}
91:
92:$dbh->disconnect;
93:
Example 4. simple.pl lines 77-93

The remaining lines of the example script appear in Example 5. The usage subroutine is called with an exit value of 0 when
invoked via the –help command line argument, and with a non-zero value when called due to an error in the command line
arguments.
Lines 115-118 specify the format that is used when the write function is invoked on line 86. This is a very simple example of
Perl’s built in report formatting. In this case the output is text that is lined up on the left. Use perldoc perlform to see a fairly
complete description of the formatting capabilities.

94:sub usage {
95: my $exitVal = shift;
96: $exitVal = 0 unless defined $exitVal;
97: use File::Basename;
98: my $basename = basename($0);
99: print qq/
100:
101:usage: $basename
102:
103: -database target instance
104: -username target instance account name
105: -password target instance account password
106: -output_type specify whether to output a 'report' or 'csv'
107: defaults to 'report'
108: example:
109:
110: $basename -database dv07 -username scott -password tiger -sysdba -output_type csv
111:/;
112: exit $exitVal;
113:};
114:

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

115:format STDOUT =
116:@<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<
117:@data
118:.
Example 5. simple.pl lines 94-118

Quite a bit of work is being accomplished in a rather short script, highlighting how powerful Perl is as a scripting language.
This script will run on Unix, Linux or Win32 without modification, and is quite useful as a template for building other scripts.

Putting Perl to Work

Interesting technology is of little value unless you make good use of it; Perl and DBI are no exception. These are interesting
tools, but do require a little investment in time spent learning to use them. Once you have some familiarity with them, don’t
be surprised to find that are now able to automate tasks that may have previously required too much development time. Yes,
DBA’s are also developers, though not in the traditional sense. DBA tools do require development time however, whether
written in SQL*Plus, PL/SQL, shell or Perl.
At this point let’s consider some of the DBA tools that I have found useful over time. When developing a new tool for DBA
use there are a number of criteria that I consider:
• Will this tool make my job easier?
• Will this tool save me time?
• Will it increase my productivity?
• Is the task to be automated one that is repeated regularly?
• Is the task to be automated one that is completely internal to the database?
• Does the job require email notification to be sent?
If automating a task or creating a tool is not going to save me much time in the long run, or make my job as a DBA any
easier, I probably won’t automate it, but just continue to do it manually.
When faced with a new task that should be automated, these criteria are used to determine how the utility should be written.
Simple tasks may be done in SQL*Plus, PL/SQL, shell, Win32 batch language or some combination of these. Tasks that are
completely internal to the database may be done in PL/SQL and called from SQL*Plus via a shell script or CMD file.
Involved tasks that require interaction with the OS or a user, or other elements external to the database are usually written in
Perl. Scripts that need to run unchanged on Unix and Win32 systems will be written in Perl. Tasks that require a great deal
of text manipulation will also likely be done in Perl, as that is one its strengths.
Following are some of the tools developed in Perl because the fit the criteria listed previously. Some of these tools have
required a fair amount of development effort, but the effort has proven to be well worth it.

Is the Database Up?

One of the essential tasks of a DBA is to ensure that the databases are up. To do so, you also need to know when the database
is down. It is preferable to be informed of this fact by a database monitor, rather than by users calling to ask if it is up. The
dbup.pl utility is used to determine if a database is up or down, and notify the oncall DBA if a database is down. This utility
is included in the PDBA Toolkit (see references), with versions for Unix and Win32.
The basis of dbup.pl is that it attempts to connect4 to all configured databases at a preconfigured interval. If the connection
fails it is recorded in a log file. It is then determined what day and time the outage has occurred, and that time is compared to
the required uptime requirements for the database. If the outage has occurred during a time or day that is not configured as
required uptime, the DBA is not paged.
If the outage is during required uptime, dbup.pl then determines if the time is outside the hours that require immediate DBA
notification. This purpose of this is to avoid paging the DBA during the middle of the night when scheduled reboots or other
maintenance may cause the database to be down for a few minutes. The DBA will be paged if the database continues to

4
Passwords are retrieved from the password server discussed in a later section of this document.

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

remain down after three5 consecutive attempts to connect to it. If the outage occurs during the hours specified by the
hoursToPageImmediate parameter then the DBA will be immediately notified.
When regular maintenance is scheduled and the database will be down for a period of time the dbignore.pl script can be used
to set a date and time in the dbignore.conf configuration file. The dbup.pl script will not attempt to connect to the database
until the time and date specified, and this will be recorded in the log file during each periodic check of configured databases.
DBAs can be put on a rotating oncall schedule via the configuration file, as well as assigning a default DBA in the event the
schedule lapses. There is also allowance to page another individual for all outages. This may be useful for notifying a Help
Desk Manager of database outages.

Password Server
A proliferation of databases and applications that run on them along with today’s tighter security requirements has made the
task of password administration more difficult than ever. A DBA may have accounts on several databases, with the password
for each database being different. The passwords for the built in Oracle accounts also must be tracked. This is far too many
passwords to be remembered by an individual. In addition, many automation tools employed by DBAs may also require
passwords to connect to the database. Scripts containing hard coded passwords are too great a security risk as well as an
administration headache. Some method is needed to provide passwords both to DBAs authorized to access them, and to
automation scripts that need to logon to databases on many servers.
The result of these requirements is the password server script pwd.pl, the client script pwc.pl and the requisite configuration
files pwd.conf and pwc.conf. These are all enabled by the PDBA::PWD and PDBA::PWC6 Perl modules.
The pwd.pl script runs as a daemon on Unix systems and as a service on Win32 systems. On both systems the password
server listens on a port for incoming password requests. The incoming request must include a recognized username and the
appropriate password, or the connection will be immediately closed.
If the request is accepted, the password server will return the requested password to the client. The password server may be
configured so that an individual may be limited to accessing only a particular set of passwords in any one database. If the
password requested is one the requester is not authorized to receive the connection will immediately be closed.
If the requester is authorized to receive the password, it will be sent to the client and the network connection then closed.
All communication between the client and server is first encrypted with RC5 prior to being sent over the network.
The client and server configuration files are both in plain text7 and must be protected by operating system permissions to
prevent unauthorized access.
The use of the password server has several benefits. It is no longer necessary to hard code passwords in scripts, or to
maintain multiple lists of passwords on several servers. Password changes simply require a modification to the pwd.conf
configuration file, and a stop and restart of the password server.
A script that needs to retrieve a password can simply do so as seen in Example 6. Methods for retrieving the password in
both Unix and Win32 platforms are provided. The shell example has the added benefit of not making the password available
to OS utilities that examine the command line, such as ps.
Unix method

DATABASE=db01
USERNAME=perfstat
PASSWORD=$(pwc.pl –instance $DATABASE –username $USERNAME)
sqlplus /nolog << EOF
connect $USERNAME/$PASSWORD@$DATABASE
select sysdate from dual;
exit
EOF

Win32 method

5
The number of retries is configurable.
6
All are included in the PDBA Toolkit from the references section.
7
There is a version of the PDBA toolkit that allows encrypting the server configuration file. At this time it must be requested
from the author.

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

set DATABASE=db01
set USERNAME=perfstat
@for /F %%I in ('pwc.pl –instance $DATABASE –username $USERNAME') do (
@set PASSWORD=%%I
)
sqlplus %USERNAME%/%PASSWORD%@%DATABASE%

Command line version

pwc.pl –instance db01 –username system

Example 6. Retrieving a password with pwc.pl

Perl scripts that require database access can also be written so as not to require a password on the command line by using the
PDBA::PWC module. Several of the scripts in the PDBA Toolkit are written to automatically retrieve the password from the
password server if the password has not been specified on the command line. This not only frees authorized users from being
required to remember or lookup database passwords, it also prevents passwords from appearing on the command line and
being detected by ps or other utilities.
The password server is used to supply passwords to the database uptime monitor previously discussed, greatly simplifying
the design and administration of that tool.
A possible enhancement of the password server would be the use of SSH keys for authorization and encryption. This has not
yet been required, but may be in the future.

Should You Use Perl?

Perl definitely has a place in the DBA toolbox. It is a general purpose scripting language that has the power to simplify tasks
that may prove to be too difficult to implement in shell or batch language, PL/SQL and SQL*Plus. There is also the benefit
that Perl is available on many platforms, and scripts may be written to run unchanged between platforms. Though there is an
initial learning curve, the investment is well worth the effort.

www.rmoug.org RMOUG Training Days 2005

Perl and the Oracle DBA Still

References

Perl
http://www.perl.com/

ActiveState Perl for Win32

http://www.activestate.com/Perl.plex?_x=1

DBI 1.46
http://search.cpan.org/~timb/DBI-1.46/

DBD::Oracle 1.16
http://search.cpan.org/~timb/DBD-Oracle-1.16/

DBI and DBD::Oracle for Perl 5.8.2-4 on Win32

ftp://ftp.esoftmatic.com/outgoing/DBI/

Comprehensive Perl Archive Network (CPAN)

http://cpan.org/
http://search.cpan.org/

Perl for Oracle DBAs

Andy Duncan & Jared Still
O’Reilly 2002
http://www.oreilly.com/catalog/oracleperl/

PDBA Toolkit (from Perl for Oracle DBAs)

http://www.oreilly.com/catalog/oracleperl/pdbatoolkit/

This paper, powerpoint and supporting scripts:

http://www.jaredstill.com

www.rmoug.org RMOUG Training Days 2005