https://emc--c.na5.visual.force.com/apex/KB_HowTo?

id=kA0700000004Xbm

Goal

Environment

Resolution

This KB is designed to Understand and Troubleshoot VBA Backup Failures and provide a Resolution Path to fix all
issues with NetWorker VMware Backup Protection Policies.

VBA 1.0.x
VBA 1.1.x
AVE 7.0.60-5
AVE 7.1.60-12
NetWorker 8.1.x
NetWorker 8.2.x

Document each step performed during the flow of troubleshooting for a quicker understanding of the issue by SME or Engineering,
if the issue needs an escalation.
The VBA Backup Failures should be tracked as per the sequence of events as discussed below:
1. Check NetWorker Management Console (NMC) for any specific Client failures. Points to be noted at this stage:

Names of the VM clients failed

Proxy names associated with each failed VM client

Device associated with each failed VM client

Any other VM client successful to same proxy / or same device? (Yes/No - details)

Figure 1

2> A backup job initiates NetWorker Policy that initiated the below processes from NetWorker:
Nsrd
nsrpolicy
nsrjobd
nsrexecd
nsrvba_save (libnsrvba library is used)
All the above processes are logged in daemon.raw logs on NetWorker server

For Windows server, it is present in <Installation Directory>\EMC NetWorker\nsr\logs\daemon.raw

For Linux/AIX server, is present in /nsr/logs/daemon.raw

To render the daemon.raw logs, follow KB 163411

For a complete list of relevant logs associated with VBA, see KB 199256.
nsrvba_save pulls information from Jobs Database of NetWorker server. This is logged in Policy Logs on NetWorker Server.

For Windows server, it is present in <Installation Directory>\EMC NetWorker\nsr\logs\<policyname>\<nsrvba_save_job-id>

For Linux/AIX server, is present in /nsr/logs/<policy-name>\<nsrvba_save_job-id>

For a complete list of relevant logs associated with VBA, see KB 199256.

Any failures at this stage need further review. Collect the information from above logs and proceed to next step. DO NOT escalate at this
point.
The nsrpolicy invokes the NSR Web Service(nsrvmwsd)to start a Backup Job

Figure 2
nsrvmwsd sends a call to VBA Web Service (managed by tomcat instance on VBA) on port 8543 (can be seen using
keywork 'nsrvmwsd' indaemon.raw logs)
VBA confirms the call in return to NSR Web Service on port 8080

This is logged on VBA appliance at /usr/local/avamar-tomcat/logs/catalina.<date>.log

For a complete list of relevant logs associated with VBA, see KB 199256.
VBA Web Service calls MC SDK (Avamar MCS takes over here) internally on VBA appliance

This is logged on VBA appliance at /usr/local/avamar/var/mc/server_log/mcserver.log.* [0-49]

If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs can be skipped).

For a complete list of relevant logs associated with VBA, see KB 199256.
MC SDK invokes 'mccli group start' that has now know which VM client to backup (stored in MCDB). At this time, the actual backup starts
and Avamar has full control of the Backup Job

This is logged on VBA appliance at /usr/local/avamar/var/ebr/server_log/ebr-server.log

If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs can be skipped).

For a complete list of relevant logs associated with VBA, see KB 199256.

4. Create, Assign and Perform Backup Task

NwBackupJobBusiness: Started backup now for group creates a work order which is sent to the avagent on VBA proxy
(responsible to receive work orders). It is logged in the log: /usr/local/avamarclient/var/avagent.log

VBA Proxy is selected by the VBA node depending on the available sessions. By default, the VBA node has one proxy server within itself. One proxy
server can handle a maximum of eight parallel sessions. If there are more than eight sessions, a different free proxy is assigned.
If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support using KB 199024 (Client/Proxy
logs REQUIRED).
For a complete list of relevant logs associated with VBA, see KB 199256.
avagent invokes avnwcomm process, responsible to give process updates to NetWorker Server. This is logged on proxy server at:

[Also called as Plugin Logs] /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>.log, where

VPP-name = VMware Protection Policy Name

WO-ID = WorkOrder ID generated for each backup/restore job by mccli

CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)

DO NOT escalate on errors at this stage and continue with next steps to review the details.
Further, avnwcomm invokes the VBA plugin avvcbimage that performs all Snapshot related tasks on the desired VM Client. This is logged on proxy
server at:

[Also called as REAL Plugin Logs] /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_real.log,

where

VPP-name = VMware Protection Policy Name

WO-ID = WorkOrder ID generated for each backup/restore job by mccli

CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)

Note that, this log have an extra _real keyword as post-fix.

Backup session and save job is created now on NetWorker

The Plugin logs are still updated simultaneously while avnwcomm keeps updating NetWorker server.

To enable debug on avmwcomm logs, follow KB 195367.

DO NOT escalate on errors at this stage and continue with next steps to review the details.

Figure 3
avvcbimage has the full control of the Backup at this stage

avvcbimage contacts VMware and issues commands to perform a Snapshot. This is logged in REAL Plugin logs.

Once the snapshot is created, avvcbimage passes the control to avtar to perform a backup to device.

avtar is the Avamar process responsible to perform all read/write operations.

It logs the acvities in Avtar Logs:

/usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_avtar.log, where

VPP-name = VMWare Protection Policy Name

WO-ID = WorkOrder ID generated for each backup/restore job by mccli

CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)

Note that, this log have an extra _avtar keyword as post-fix.

If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs REQUIRED).
For a complete list of relevant logs associated with VBA, see KB 199256

5> Data Domain Integration

Most VM Backups are backed up on Data Domain devices using VBA NetWorker VMware Protection.

Client Backup metadata is stored on VBA

Client Backup Data is directly written on Data Domain via DDBoost

Snapshot of VM is copied to:/avamar-mtree/STAGING/<CID>/BACKUP-<timestamp>/

Once copy is finished, backup files are moved to permanent location: /avamar-mtree/cur/<CID>/<timestamp_of_backup>/

These can be seen in the Avtar logs on proxy server: /usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<PluginID>_avtar.log

VBA stores the files in Avamar MTREE in "RAW" format, the real VMDK files with a different name given by DDOS internally. See Figure 3.

The Backup Process further copies these "RAW" format files from Avamar MTREE to a CDSF package. This CDSF package is exported to NetWorker
MTree. See Figure 3.

The CDSF export job is logged on the proxy server at:

/usr/local/avamarclient/var/<VPP-name>-<WO-ID>-<CID>-<Plugin-ID>_cdsf_export.log, where

VPP-name = VMware Protection Policy Name

WO-ID = WorkOrder ID generated for each backup/restore job by mccli

CID = Unique Client ID registered with VBA Appliance (See KB 199312 to find CID for troubleshooting)

Note that, this log have an extra _cdsf_export keyword as post-fix.

If any errors found at this stage, collect VBA Logs using KB 199026 and immediately escalate to Avamar Support
using KB 199024 (Client/Proxy logs REQUIRED).

For a complete list of relevant logs associated with VBA, see KB 199256.

Note that, the errors in cdsf_export logs may need Data Domain Team's assistance, This will be initiated by Avamar Team.