You are on page 1of 12

Problem

GENERAL ERROR: Vault policy fails in the Activity Monitor with status code 23 (socket read failed) reported.

Error
EXACT ERROR: ld.so.1: ./bpbrmvlt: fatal: relocation error: file ./bpbrmvlt: symbol V_snprintf: referenced symbol not found

Solution
Overview: The above error has been seen in NetBackup 5.1 when the NetBackup 5.1 Vault binaries have been patched to a higher level than the Master's server and client binaries. For example, the error was seen after installing Netbackup 5.1 GA server and client binaries onto a master server, then subsequently patching the NetBackup Vault binaries to 5.1 MP6 without first installing the NetBackup 5.1 MP6 server and client maintenance packs. Troubleshooting: The failure in the Vault job occurs shortly after the job goes active, and the above error can be seen in the bpsched log on the master, if it has been enabled. The above error can also be reproduced by running bpbrmvlt from the command line:
/usr/openv/netbackup/bin/bpbrmvlt

The pack.summary file may assist in determining which maintenance packs have been installed. It can be found in /usr/openv/pack. Resolution: If affected by this issue and it is determined that Vault is at a higher maintenance pack level than NetBackup, there are two possible resolutions:

Uninstall the NetBackup Vault patch so that the Vault binaries are at the same level as NetBackup master server and client binaries. If the NetBackup environment must be at the higher patch level, follow these steps: 1. Uninstall the NetBackup Vault patch 2. Install the NetBackup server and client binary patch. 3. Re-apply the NetBackup Vault patch

Problem
GENERAL ERROR: Vault policy fails in the Activity Monitor with status code 23 (socket read failed) reported.

Error
EXACT ERROR: ld.so.1: ./bpbrmvlt: fatal: relocation error: file ./bpbrmvlt: symbol V_snprintf: referenced symbol not found

Solution
Overview: The above error has been seen in NetBackup 5.1 when the NetBackup 5.1 Vault binaries have been patched to a higher level than the Master's server and client binaries. For example, the error was seen after installing Netbackup 5.1 GA server and client binaries onto a master server, then subsequently patching the NetBackup Vault binaries to 5.1 MP6 without first installing the NetBackup 5.1 MP6 server and client maintenance packs. Troubleshooting: The failure in the Vault job occurs shortly after the job goes active, and the above error can be seen in the bpsched log on the master, if it has been enabled. The above error can also be reproduced by running bpbrmvlt from the command line:
/usr/openv/netbackup/bin/bpbrmvlt

The pack.summary file may assist in determining which maintenance packs have been installed. It can be found in /usr/openv/pack. Resolution: If affected by this issue and it is determined that Vault is at a higher maintenance pack level than NetBackup, there are two possible resolutions:

Uninstall the NetBackup Vault patch so that the Vault binaries are at the same level as NetBackup master server and client binaries. If the NetBackup environment must be at the higher patch level, follow these steps: 1. Uninstall the NetBackup Vault patch 2. Install the NetBackup server and client binary patch. 3. Re-apply the NetBackup Vault patch

Problem

SQL backups are failing within a minute reporting a status code 23 from the perspective of bpbrm

Error

From the job details inside the detailed status tab: 09/15/2010 06:00:15 - Error bpbrm (pid=1234) bpcd on SQLSERVER exited with status 23: socket read failed 09/15/2010 06:00:20 - Error bpbrm (pid=1234) cannot send mail because BPCD on SQLSERVER exited with status 23: socket read failed 09/15/2010 06:00:21 - end writing socket read failed (23)

From SQLSERVER's bpcd log: 06:00:10.923 [12345.10888] <2> logconnections: BPCD ACCEPT FROM 1.2.3.4.58996 TO 21.32.43.54.13724 06:00:10.923 [12345.10888] <2> bpcd main: setup_sockopts complete 06:00:10.930 [12345.10888] <2> vauth_acceptor: ..\libvlibs\vauth_comm.c.337: no methods for address: no authentication required 06:00:15.439 [12345.10888] <8> bpcd peer_hostname: gethostbyaddr failed : The requested name is valid, but no data of the requested type was found. 06:00:15.439 [12345.10888] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 0

06:00:15.439 [12345.10888] <16> bpcd main: Couldn't get peer hostname

Environment

The master server, media server and SQL server were all running on different hosts. The media server name is MEDIASERVER. The SQL server name is SQLSERVER.

Cause

The SQL server is unable to resolve the media server's IP address. BPCD represents the client layer of NetBackup, and before it will fork or execute NetBackup operations, it verifies that the request comes from a trusted server. Because the client layer is unable to resolve the IP address of the media server, it cannot verify that the request comes from a trusted server. The system call gethostbyaddr is used to resolve an IP address to a hostname.

Solution

A workaround is to add an entry for the media server's IP address and name into the C:\windows\system32\drivers\etc\hosts file, e.g. 1.2.3.4 MEDIASERVER MEDIASERVER.FQDN A more formal fix is to have the DNS administrators fix the PTR records for the IP address 1.2.3.4 to point back to MEDIASERVER

Problem
Status Code 23: Socket Read Failed Occurs on backup

Solution
A Status 23 error could be generated when there are differences between the forward and reverse lookup zones entries of the DNS server. Run nslookup for the client servers having problems and verify that the reverse and forward entries in DNS are the same. Example: ==========================
C:\>nslookup ClientServerName Server: DNS_Server.mydomain.net Address: 100.100.100.1 Name: ClientServerName.mydomain.net

<<--- This name has to be the same as the DNS

reverse lookup zone


Address: 10.66.18.61 C:\>nslookup 100.100.100.2 Server: DNS_Server.mydomain.net Address: 100.100.100.1 Name: ClientServerName.mydomain.net

<<--- This name has to be the same as the DNS

forward lookup zone


Address: 100.100.100.2

If entries are inconsistent, the issue should be addressed within DNS. If using host files, ensure that all server entries are updated and valid on client servers, as well as the media servers performing the backups.

Problem
STATUS CODE 23: NetBackup 6.0 client-initiated bplist command fails with EXIT Status 23: socket read failed

Error
socket read failed

Solution
Overview: When bplist is initiated from a NetBackup client, an attempt is made to connect to the NetBackup master server on the bprd port (13720). The master must then connect back to the client. If the master server's bprd process is unable to resolve the hostname of the client based on the IP address of the incoming connection, the operation fails.

#bplist -l /tmp EXIT STATUS 23: socket read failed

Troubleshooting: To verify that the NetBackup master is able to resolve the client IP address, use the following command on the NetBackup master server:
/usr/openv/netbackup/bin/bpclntcmd -ip <client_ip_address>

The following output indicates the NetBackup master server has a reverse lookup issue that needs to be resolved:
#/usr/openv/netbackup/bin/bpclntcmd -ip <client_ip_address> host <client_ip_address>: not found.

Resolution: To resolve this issue, either configure the DNS server to resolve the NetBackup client correctly for both forward and reverse lookup, or add the NetBackup client's host information to the hosts file of the NetBackup master server.

Problem
Status Code 23: Socket Read Failed.

Solution
Status Code 23: Socket Read Failed. ============================================= When performing backups for a NetBackup client server that are failing with a status code 23, follow these steps to troubleshoot the problem: 1-) Enable bpcd logs in the client server UNIX =========== mkdir /usr/openv/netbackup/logs/bpcd

Windows =========== mkdir <Installation Path>\NetBackup\logs\bpcd

2-) From the media server performing the backups, execute the bptestbpcd command UNIX =========== /usr/openv/netbackup/bin/admincmd/bptestbpcd -host <client server name> -verbose

Windows =========== <Installation Path>\NetBackup\Bin\Admincmd\bptestbpcd -host <client server name> -verbose Note: The bptestbpcd command should fail with a status code 23 when being executed from the media server performing the backups. 3-) Review the bpcd log created in the client server, observe the network interface that is making the incoming request from the media server as shown below: 12:39:53.549 [13483] <2> logconnections: BPCD ACCEPT FROM <Media Server IP>.<Port Used> TO <Client Server IP>.<Port Used> 12:39:53.549 [13483] <2> bpcd main: setup_sockopts complete 12:39:53.551 [13483] <2> vauth_acceptor: vauth_comm.c.337: no methods for address: no authentication required 12:39:53.551 [13483] <2> bpcd main: no authentication required 12:39:53.551 [13483] <8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1) 12:39:53.551 [13483] <16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno =1 12:39:53.551 [13483] <16> bpcd main: Couldn't get peer hostname

The Status Code 23 is being generated since the client server is unable to resolve the media server network interface by forward or reverse lookups in DNS. If the media server performing the backups has more than one network interface, check the binding order in the media server to ensure that the network interface used for the backups is the desired network interface for backups.

Correcting the missing entries in DNS will resolve this issue. If access to DNS is not available, adding entries to the hosts file in the client server to resolve the network interface of the media server will resolve the problem.

Problem
BUG REPORT: Restores from a large NDMP backup fails with (EXIT STATUS 23: socket read failed)

Error
Error bpbrm(pid=8632) client restore EXIT STATUS 23: socket read failed

Solution
ENVIRONMENT/CONDITIONS: In a 6.0 MP4 environment large NDMP restores can sometimes fail with Status Code 23. EVIDENCE: Detail Status: 1/4/2008 2:47:25 PM - Error bpbrm(pid=8632) client restore EXIT STATUS 23: socket read failed 1/4/2008 2:48:40 PM - Error ndmpagent(pid=8796) write socket err 10054 (An existing connection was forcibly closed by the remote host. .) 1/4/2008 2:48:41 PM - Error ndmpagent(pid=8796) connection 007F7000 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR) SOLUTION: Upgrade to 6.0 MP5 or later to resolve this issue. WORKAROUND: Disabling DAR will allow the restore to complete, but the restore operation will take significantly longer.

Problem

STATUS CODE 23: Schedule initiated Client backups fail with a NetBackup Status Code 23 (socket read failed).

Error

socket read failed

Solution
Overview: Schedule initiated Client backups fail with a NetBackup Status Code 23 (socket read failed). Troubleshooting: Use the bpclntcmd command to test the connectivity between the master server and the client. Specifically, be sure to use the bpclntcmd -pn command. Log files: N/A Resolution: This problem is caused by incomplete name resolution information in either the hosts file or the DNS. Be sure the client's name and IP address are in the master server's hosts file, and the master server's name and IP address are in the client's hosts file. The hosts files are located: - in the %SystemRoot%\system32\drivers\etc directory (Windows) - in the SYS:etc directory (NetWare) - in the /etc directory (UNIX/Linux)

Problem
BUG REPORT: Large NDMP backups to filers fail with status code 23 (socket read failed) reported.

Error

<16> write_data: ndmp_xm_get_kbytes failed

Solution
Bug: 1195987 Detail/Symptoms: Very large NDMP backups to filer fail with status 23 (socket read failed) reported. Log Files: In the bptm log of the media server, messages similar to the following are found:
19:01:10.180 19:06:07.258 0 19:06:37.258 19:11:37.790 19:11:37.790 [4792.5372] <2> NdmpSession: [514] Sending 15 (GET_KBYTES) "" [4792.5372] <2> NdmpSession: [514] Reply 15 "433281744", error = [4792.5372] <2> NdmpSession: [515] Sending 15 (GET_KBYTES) "" [4792.5372] <2> NdmpSession: [515] Reply 0 "", error = 18 [4792.5372] <16> write_data: ndmp_xm_get_kbytes failed

In reality, the ndmpagent process is still working (and its logs will confirm this as messages will continue to be logged past the reported error), but bptm doesn't "get the message" - and after the default timeout of five minutes (300 seconds), a failure is reported. Resolution: The NDMP_CONNECT_TIMEOUT touch file can be used to increase the value of this timeout. It will need to be located in: Windows: <install_path>\NetBackup\ndmp_connect_timeout UNIX: /usr/openv/netbackup/ndmp_connect_timeout This file should contain the value in seconds of the desired timeout. For instance, a value of 480 would correspond to an 8 minute timeout. This functionality was introduced in the following release: NetBackup 6.0 Maintenance Pack 7 (MP7) To see if this change has taken effect, check the ndmp logs (OID 151) for the following message:
ndmp_net_connect: Timeout Value: 480000

The timeout will be specified in microseconds (that is, the value of NDMP_CONNECT_TIMEOUT multiplied by 1000). Note: Starting in NetBackup 6.5, the value of CLIENT_READ_TIMEOUT is used to set the NDMP connect timeout. The above mentioned touch file is not used. If this condition is seen in a NetBackup 6.5.x environment, please adjust the client connect timeout value to resolve this issue.

Problem
General Error: Status code 23, socket read failed, during client backups or restores, or when loading client properties.

Solution
The above status code can be seen for a number of reasons, and generally indicates a network or hostname resolution issue of some kind. However there may be different reasons Troubleshooting: Typical troubleshooting procedures include verifying that bpcd is listening on the client, and attempting to connect to the bpcd port (13782) on the client in question. In one case bpcd was listening on the client, and the telnet was successful from both the master/media server to the client, as well as from the client to itself. Example: telnet <client name> 13782 When the telnet to the bpcd port is successful, it usually results in the attempt being logged in the bpcd debug log. However, in one case no log was created despite the successful connections being made to the port. Further inspection of the Task Manager showed a large number of bpcd processes running. After terminating all existing processes along with bpinetd, and restarting the NetBackup client service, each subsequent telnet to the bpcd port resulted in an additional bpcd process accumulating in the Task Manager. As a final test to see if there was an issue with the bpcd binary, bpcd was run via a command prompt directly from the NetBackup bin directory:
<Install_path>\NetBackup\bin\bpcd

which resulted in, "Please enter keyfile passphrase" It was then found that there was a copy of the keyfile.dat file in the bin directory. Cause: keyfile.dat file in the <Install_Path>\NetBackup\bin\ directory that resulted in bpcd issuing the prompt and waiting for a passphrase, and remaining as a running process. Resolution: Move the keyfile.dat file out of the <Install_Path>\NetBackup\bin\ directory.

You might also like