How to Troubleshoot Failed Login Attempts to DB Control

Note:404820.1 HOWTO
Last Revision Date: 31-AUG-2008 PUBLISHED

In this Document
Goal
Solution
References

Applies to:

Enterprise Manager Grid Control - Version: 10.2.0.1 to 10.2.0.1

Information in this document applies to any platform.

Goal

The purpose of this Note is to fast-track the debugging process with regard to the inability to
connect or login to DB Console via a web-browser.

Solution

The following checks can be undertaken:

1. If failing to connect to DB Console via remote connection to the DB Console server, are you
able to login to DB Console locally using a web-browser that is running on the actual DB
Console Server? If so then the issue is at the remote workstation (network connectivity to the
DB Console Server, firewall etc).

Preferably you should reference the DB Console server using a Fully Qualified Hostname in
the web-browser URL.

1.0 If a firewall is being used, ensure that the EM Port (default non-secure port 5500 or default
secure port 1158) is open for communication in the firewall e.g.

[oracle@abc ~]$ telnet xyz.oracle.com 5500

Trying <IP Address for xyz.oracle.com>...
Connected to xyz.oracle.com (<IP Address for xyz.oracle.com>).
Escape character is '^]'.

Note: The above example shows the output from a successful telnet.

Should telnet be disabled & ssh communication enabled then the following command can be
used instead:

ssh -X xyz.oracle.com -p 5500

Note: A successful connect to the port using SSH will result in no output otherwise if not
successfull the following error will show:

ssh: connect to host xyz port 1158: Connection refused

1.1 Are you able to ping the DB Console server from your machine? If not then get in contact
with your network admin to resolve this networking issue.

1.2 If you are able to successfully ping the DB Console server from your machine then you
need to determine the order of priority between the "hosts" file & DNS. To do this in a UNIX
environment then issue the following command on your machine:
grep -i hosts /etc/nsswitch.conf

If the result has "files" before "dns" as follows:

"hosts: files dns"

then your "/etc/hosts" file is taking precedence over DNS. Sometimes machines are configured
in this way to minimized the overhead on the network. You need to ensure that your
"/etc/hosts" file is setup correctly with a reference to the DB Console server. Refer to Note
203280.1 Ext/Pub How to Configure the hosts File for 9iAS 9.0.2 Installations.
The contents of this note do not only apply to 9iAS 9.0.2 and are as follows:

Verify that /etc/hosts.* has the following format:

IP_ADDRESS FULLY_QUALIFIED_HOSTNAME SHORT_HOSTNAME ALIASES

The following example shows a properly configured /etc/hosts.* file:

<ip address> <hostname.domain> <hostname> <alias>

1.3 If on the other hand, you can ping the DB Console server from your machine but the
returned value from issuing the "grep -i hosts /etc/nsswitch.conf" command has the "dns"
ahead of "hosts" then DNS is taking precedence. You then need to ensure that (a) an nslookup
on the DB Console Hostname returns the DB Console host's correct IP address & (b) an
nslookup on the DB Console's IP address (Reverse NSLOOKUP) returns the fully qualified
name of the DB Console host. If this is not the case then please consult with you network
admin.

A Good area to check for issues with connectivity & login to DB Console are the following files
found on the DB Console server:

-> $ORACLE_HOME/hostname_sid/sysman/log/emoms.*

Login failures in DB Console will be logged in the emoms.* files.

2. Are you using the correct port in your URL reference to the DB Console?
There are 2 ways to check that you are using the correct port. They are as follows:

2.1 Go to the $ORACLE_HOME/install directory & take a look at the portlist.ini file.
The portlist.ini will contain the port assignments. Look for the "Enterprise Manager Console
HTTP Port (ORACLE_SID) = <Port Number>" entry. Default DB Console port being 5500
(unsecured) whilst 1158 is the default for DB Console if it has been secured. So the URL to
access DB Console should look like the following: http://<DB_Console_Host>:5500/em or
https://<DB_Console_Host>:1158/em i.e. http(s)://<DB_Console_Host>:<Port Number>/em

2.2 Go to the $ORACLE_HOME/hostname_sid/sysman/config directory & look for the following

entries in the emoms.properties file:

oracle.sysman.emSDK.svlt.ConsoleServerPort=<DBConsole_Port_Number>

oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort=<DBConsole_Port_Number>

3. Is DB Console up & running? From the $ORACLE_HOME/bin directory run

emctl status dbconsole

to ensure that the DB Console is up & running. You should receive the following message if it
is:

"Oracle Enterprise Manager 10g is running."

If the DB Console is not running then issue the

emctl start dbconsole

command from the same directory.

If any errors are reported in the starting up of the DB Console then please consult the log &
trace file found in the $ORACLE_HOME/hostname_sid/sysman/log/ directory. In particular the
emoms.* & emagent.* files.

4. Ensure that the Database is up & running as follows:

From the Database host issue:

sqlplus "/ as sysdba"

SQL> startup immediate;

OR issue from the command prompt:

ps -ef |grep $ORACLE_SID

4.1 Ensure that the Database is not in the MOUNT state which means connections for normal
users will fail with the error :
ERROR: ORA-01033: ORACLE initialization or shutdown in progress
To Correct this:
Connect as 'sys as sysdba' user and open the database and verify that the database is in
'READ WRITE' mode for all users.

SQL> alter database open;

SQL> select name, OPEN_MODE from v$database;

NAME OPEN_MODE
--------- ---------
<DB_NAME> READ WRITE

5. Ensure that the Listener is running & listening for the Database by issuing the following
command:
lsnrctl status

You will see a result similar to the following output if it is up & running:

Service "ORACLE_SID" has 2 instance(s).

Instance "ORACLE_SID", status
UNKNOWN, has 1 handler(s) for this service...
Instance "ORACLE_SID", status
READY, has 1 handler(s) for this service...

5.1 Ensure that the listener does not reference any symbolic links (UNIX environment). If it
does then copy across the actual files i.e. tnsnames.ora & listener.ora to their expected
locations, removing the symbolic links. As the DB Console agent is unable to resolve the
references from symbolic links.

6. Ensure that the SYSMAN', 'DBSNMP','MGMT_VIEW' accounts are not locked as follows:

6.1 From sqlplus, login as SYS & run the following command:

select username, account_status from dba_users where username in ('SYSMAN',

'DBSNMP','MGMT_VIEW');

Take note of the accounts that are locked.

6.2 Stop the DB Console by running the following command:

$ORACLE_HOME/bin/emctl stop dbconsole

6.3 Unlock the Accounts that are locked by running the following command:

alter user <username> account unlock;

6.4 Start the DB Console once more by running.

$ORACLE_HOME/bin/emctl start dbconsole

7. Check to see whether the SYSMAN or DBSNMP user passwords have recently changed. If
so, find out what the new password is & use it to login to DB Console as the SYSMAN user.
Then remap the new DBSNMP password into the "Monitoring Credentials" for the Agent that is
monitoring the Database if the DBSNMP password has also changed.

7.1 Ensure that the OEM_MONITOR privilege exists & has been granted to the SYSMAN user
account or any other Administrator level account that has been created for the purpose of
monitoring & administering the Database. Refer to Note 152593.1 How to Grant
OEM_MONITOR Role to Database Preferred Credentials:

8. Other connection Related Problems with DB Console

---------------------------------------------------
[Problem:] ORA-1017 ERROR when logging into DB Console.
Problems with the credentials, when trying to connect with sys as sysdba locally there is no
problem, but when trying to connect remotely an ORA-1017 ERROR is experienced.

[Answer:] This normally has to do with the remote_login_passwordfile parameter, needing to

be set to EXCLUSIVE.
Refer to Note 204166.1 Connecting To Database From EM to db target as SYSDBA Fails With
ORA-1031

[Problem:] Agent sends message "Failed to connect to database instance: ORA-12537:

TNS:connection closed (DBD ERROR: OCIServerAttach)." though everything seems to be
fine.

[Answer:] The ONS may be interfering with effective listener operation as described in :
Note 284602.1 10g Listener: High CPU Utilization - Listener May Hang

Explicitly disable the 10g TNS Listener from subscribing to the ONS.
Edit the active 10g listener.ora file and add the parameter below:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

where <listener_name> would be replaced with the actual listener name

configured in the listener.ora file.
For example, if the listener name is LISTENER (default), the parameter would be:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

then restart the listener

This error message should disappear.

[Problem:] when connecting to DB Console as SYSDBA in a Windows Environment. It returns
insuffient privleges ORA-01031.

[Answer:]
1. Include the following to the sqlnet.ora

SQLNET.AUTHENTICATION_SERVICES= (NTS)

2. Ensure the the OS user has the "Logon as a Batch Job" privilege

[Problem:] New basic installation Of Enterprise Edition Database Control in Windows. The
installation is completed. Can log on the database using SQL plus, but can not access the
database control. It display a blank page.

[Answer:] Ensure the appropriate services have been started & set to auto-startup as follows:

To autostart make the following changes to the registry.

HKEY_LOCAL_MACHINE->SOFTWARE->ORACLE->KEY_<ORACLE_SID>_home1->

ORA_<ORACLE_SID>_AUTOSTART=TRUE

ORA_<ORACLE_SID>_SHUTDOWN=TRUE

ORA_<ORACLE_SID>_SHUTDOWN_TIMEOUT=90

ORA_<ORACLE_SID>_SHUTDOWNTYPE=immediate

Also in "Services" ensure that the "OracleService<ORACLE_SID>" is set to autostartup. As

well as the TNSLISTENER & DBConsole services.

[Problem:] Attempt to start the dbconsole with emctl start dbconsole generates the following
message:

OC4J Configuration issue.

$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_<hostname_sid> not found.

[Answer:] When upgrading the database from a previous version, the emca is not ran thus the
dbconsole is not configured. Depending on the existing configuration different approaches are
done:

If you upgraded the database from a previous version(9i), then you must drop the existing
SYSMAN repository and create a new one using emca.

If you manually create a database and no SYSMAN reporistory was created, run the emca
If you used the DBCA and created the scripts with the Database Control option, the repository
was created, you will need to run the

emca -r

Refer to the following notes:

Note 456111.1 Troubleshooting DbConsole Error - OC4J Configuration issue

Note 297363.1 Oc4j Configuration Issue Starting Em Dbconsole Oc4j 'CONFIG ORA-00922
missing or invalid option':

[Problem:] Migrated a single instance DB from host abc123 to RAC hostname def456. The
services DB is up fine, but when logging into Enterprise Control Manager. It references the
wrong connection string. When logging in, it attempts to connect to abc123 instead of def456.
It seems to be acquiring this string from the DB it, how to modify it so that it picks up host
def456?

[Answer:] Drop and recreate the DB CONTROL with the correct hostname as follows:

A. Login to SQLPLUS as user SYS or SYSTEM, and drop the sysman account &
mangement objects:

-> Clean up the Db Console repository

a. drop user sysman cascade;
b. drop role MGMT_USER;

c. drop user MGMT_VIEW cascade;

d. drop public synonym MGMT_TARGET_BLACKOUTS;
e. drop public synonym SETEMVIEWUSERCONTEXT;

f. issue a "commit;"

B. Export the correct values for the ORACLE_HOME and ORACLE_SID environment
variables

C. Change directories to the $ORACLE_HOME/bin directory

D. Cleanup the External DB Console Configuration files by

issuing:
emca -deconfig dbcontrol db -repos drop

E. Recreate the DB Console Repository & external Configuration files by

issuing:
emca -config dbcontrol db -repos create

9. As a "Catch All", the DB Console can be recreated. Please keep in mind that *RE-
CREATING THE DB CONSOLE WILL WIPE OUT ALL SCHEDULED JOBS*. But it will keep all
the existing job definitions in-tact. This is a known issue for Database Control up to version
10.2.0.2. If you are fine with the possibility of having all your job schedules wiped out then you
may proceed to undertake the steps to re-create the DB Console configuration files. If a re-
build of the *DB Console configuration files ONLY* was insufficient in resolving the issue, then
if you can afford to lost the existing Repository information then also rebuild the Repository in
addition to the DB Console Configuration files.

SCENARIO 1: Delete the DB Control using scripts:

A) To delete only the configuration files:
In 10g R1 run:
<ORACLE_HOME>/bin/emca -x <sid>
In 10g R2 run:
<ORACLE_HOME>bin/emca -deconfig dbcontrol db

B) To delete only the DB Control repository:

In both 10g R1 and R2 run:
<ORACLE_HOME>/sysman/emdrep/bin/RepManager <hostname> <listener_port> <sid>
-action drop

C) To delete both the configuration files and the repository:

In 10g R1 do:

<ORACLE_HOME>/bin/emca -x <sid>

<ORACLE_HOME>/sysman/emdrep/bin/RepManager <hostname> <listener_port> <sid>

-action drop
In 10g R2 do:

<ORACLE_HOME>/bin/emca -deconfig dbcontrol db -repos drop

SCENARIO 2: Manually delete the DB Control:

The procedure is the same for 10g R1 and R2

A) To delete the configurartion files:

- remove the following directories from your filesystem:

<ORACLE_HOME>/<hostname_sid>

<ORACLE_HOME>/oc4j/j2ee/OC4J_DBConsole_<hostname>_<sid>

SCENARIO 3: Create the DB Control

A) To create only the DB Control configuration files, skipping the
repository
creation (this would be done for instance if you dropped only the files and
left
the repository in place):
In 10g R1 do: <ORACLE_HOME>/bin/emca -r
In 10g R2 do: <ORACLE_HOME>/bin/emca -config dbcontrol db

B) To create the DB Control configuration files and repository:

In 10g R1 do:
<ORACLE_HOME>/bin/emca
In 10g R2 do <ORACLE_HOME>bin/emca -config dbcontrol db -repos create

You can use the steps above to recreate the DB Control (configuration files and repository).
There are 2 typical scenarios:
- you only recreate the DB Control configuration files (i.e. if you suspect that the repository is
intact)
- you recreate the configuration files as well as the repository (i.e. if you suspect that the
repository may be corrupt);

NOTE: In 10g R2 you can use the following comand to recreate (drop and create) the DB
Control configuration files and repository:
<ORACLE_HOME>/bin/emca -config dbcontrol db -repos recreate

For additional information on emca commandline options, please see the Oracle Enterprise
Manager 10g Advanced Configuration Guide.
10g R1: http://download.oracle.com/docs/pdf/B12013_03.pdf
10g R2: http://download.oracle.com/docs/cd/B16240_01/doc/em.102/e10954.pdf

10. A piece of advice: Please include the backup of the

$ORACLE_HOME/hostname_sid/sysman/config/*.* files in your backup strategy for DB
Console esp. the emoms.properties file.

AIX Based Systems

Note 332805.1 - Database Control failing on AIX5L system with DISPLAY set incorrectly -
unable to access Enterprise Manager website.
Note 399403.1 - Db Control Hangs When Shutting Down

References

Note 273758.1 - How To: Configure a 10g Database When DB Control Shows The Status Of
Unavailable
Note 274661.1 - Problem: Startup: Cannot Start DBCONSOLE on DHCP Machine after
Reboot
Note 276950.1 - How to setup Enterprise Manager Database Control for SSL (HTTPS) secure
connection
Note 278100.1 - How To Drop, Create And Recreate DB Control In A 10g Database
Note 309595.1 - Attempting to use EM 10g Java Console gets error: Procedure Entry Point
Kpuhhalo Could Not Be Located In Dynamic Link Lib Oci.Dll
Note 330130.1 - Overview Of The EMCA Commands Available for DB Control 10.2
Installations
Note 335571.1 - Enterprise Manager Database Control only starts on one node of my RAC
cluster. How do I start it on another node?
Note 383085.1 - Problem: Running EMCA Fails With "Could not complete the configuration"
When DBSNMP Password is Incorrectly Specified at During DB Control Creation
Note 393727.1 - Failed To Connect To The Database: Io Exception
Note 395162.1 - How to manage DB Control 10.2 for RAC Database with emca
Note 398499.1 - Problem: Startup: EM Database Control Has Stopped Working and Unable to
Start Again
Note 399751.1 - Cannot Logon To Oem Console After Changing IP Address
Note 456111.1 - Troubleshooting DbConsole Error - OC4J Configuration issue

Keywords

HANGING ; TNSNAMES.ORA ; LISTENER.ORA ; LOGON ; DATABASE~CONTROL ; CHAN

GE~PASSWORD ; TNSNAMES.ORA ; UNABLE~TO~CONNECT ;

Help us improve our service. Please email us your comments for this document. .