Troubleshooting XenDesktop 5 Deployments

Baptiste Duflos, Escalation Manager & Ken Baldwin, Escalation Engineer Tuesday, May 24th 2011

Introduction and objectives

Case study for MCS fails to create pooled machines

Machine Creation Services introduces:

Fully integrated provisioning into the XenDesktop 5 console Desktop lifecycle support and image roll-back capability

Leverages and supports all 3 major Hypervisors

Citrix Confidential - Do Not Distribute

VMs can be created in pooled or private mode

Each VM consists of a Difference disk and an Identity disk VM VM

VM

Storage
Id Disk Diff Disk Id Disk Diff Disk Id Disk Diff Disk

Persistent Identity disk provides AD computer account info

Master Disk One copy of the base image shared by all VMs

Pooled image will reset back to initial state after reboot

AD Identity Service
Broker Data Access

Active Directory

Data Access

SQL

Data Access

Machine Identity Service Infrastructure Service

Machine Creation Service

HCL

HCL

Host Service
Configuration Service
Citrix Confidential - Do Not Distribute

Hypervisors and Storage

Hypervisors

Reproducing the error: failed to create Catalog

Storage

Machine Creation Service

SQL Data Access HCL

Hypervisors

The Catalog could not be loaded due to the following errors: There are no master images associated with this Catalog See CTX127068 for resolutions to this problem

Network
Citrix Confidential - Do Not Distribute

Troubleshooting Methodology initial first look

Validate the Hypervisor is configured correctly Check the image

Check permissions if Check the master image Try using Certs and Validate the Hypervisor Verify the another virtual storage pathand test is not using Configure CTX125578 snapshot image for creation permissions - CTX127546 Proxy.xmlwasnt deleted local attached storage multiple host connections
Citrix Confidential - Do Not Distribute

Troubleshooting Methodology Logs and Traces

MCS, Controller, and Broker

SQL

Service Logging - CTX127492 SQL Trace - CTX127257 CDF Control - CTX111961

Citrix Confidential - Do Not Distribute

Machine Creation Service Log Analysis

CitrixMachineCreationService:-> Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - Entry CitrixMachineCreationService:Returning cached service instances CitrixMachineCreationService: Citrix.XDServiceBase.LogicBase.GetRemoteServiceInstances - Exit CitrixMachineCreationService:Sorting the ServiceInstances. CitrixMachineCreationService:Using the next service instance http://xd5lab.local/Citrix/HostingUnitService/IServiceAPI CitrixMachineCreationService:Conversion error in Property Resolver. Exception is System.NullReferenceException: Object reference not set to an instance of an object. at HostingUnitServiceClient.HusClient.TranslateHostingUnit(HostingUnitInternal hostingUnit) at HostingUnitServiceClient.HusClient.GetHostingUnitDetails(Guid uid) at Citrix.DesktopUpdateManager.SDK.SDKLogic.GetHostingUnit(Guid uid) at Citrix.XDServiceBase.PropertyResolver`2.Resolve(TInput toResolve) CitrixMachineCreationService:Exception caught in PostProvTask, HostingUnit not found, not adding prefix

Citrix Confidential - Do Not Distribute

MCS Log Analysis

MachineCreationServiceLog:2:1:Queued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=1, high priority=0, no-op=0" MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTask580c3ed9-d6ac-44a1-94e2-442e015c531c" MachineCreationServiceLog:2:1:Dequeued task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c, current queue length=0, high priority=0, no-op=0" MachineCreationServiceLog:2:1:Queued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=1, high priority=0, no-op=0" MachineCreationServiceLog:2:1:VMware: Begin copy disk lenir-012603_S4B4-1-baseDisk, task RunTaskb5e8b09e-5568-41eb-86e1-2acae9b98358" MachineCreationServiceLog:2:1:Dequeued task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358, current queue length=0, high priority=0, no-op=0" MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-b5e8b09e-5568-41eb-86e1-2acae9b98358" MachineCreationServiceLog:2:1:EndCopyDisk: task RunTask-580c3ed9-d6ac-44a1-94e2-442e015c531c"

Citrix Confidential - Do Not Distribute

SQL Profile trace Analysis

Look through the trace On the SQL Profile and check sure to trace makefor any permission errors or select the following: any failures for running Security Audit a stored procedure Stored Procedures For our case everything looked normal so we need to focus on the CDF analysis
Citrix Confidential - Do Not Distribute

Using CDF Control

High level failure is: Failed enabling master With CDF Control Hosts. Parsing the CDF trace andto copy all the expert shader feature allows images to all of the you No machines haveexceptions which are typically highlighted in orange us to quickly find been added to the Catalog.can download the public TMF files which will allow you to parse the CDF trace and troubleshoot your issue

Citrix Confidential - Do Not Distribute

CDF Trace Log Analysis

MachineCreationServiceLog:1:1:Converting to a return code, an exception of type: Citrix.Cds.DAL.DALDataStoreException and message: General database error: XML parsing: line 1, character 331, illegal name character. MachineCreationServiceLog:2:1:The DALDataStoreException, has an inner Sql exception with the Number set as 9421. MachineCreationServiceLog:1:1:Creating a new provisioning scheme failed with error ServiceStatusInvalidDB. MachineCreationServiceLog:1:1:System.InvalidOperationException: ServiceStatusInvalidDB At Citrix.DesktopUpdateManager.SDK.NewProvisioningSchemeSupport.NewProvisioningSchemeLogi c c.DoCommitScheme(NewProvisioningSchemeWorkflow context) MachineCreationServiceDAL:8:5:DAL >>> WorkflowAddMetadata(2bcc068d-a5b0-42c0-933b38958a7a74bb, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.)
Citrix Confidential - Do Not Distribute

Root Cause Analysis

Citrix Confidential - Do Not Distribute

Resolution
This issue resulted in Citrix adding a check in the code for each call to path with improved error handling when illegal characters are discovered in the storage naming scheme. The change has been checked into XenDesktop 5 SP1.

Citrix Confidential - Do Not Distribute

Troubleshooting XenDesktop 5 Session Launch using Pass-through Authentication

Problem Definition
XenDesktop 5 sessions fail to launch when using passthrough authentication
Steps to Reproduce:
1. Launch XenDesktop session from a domain-joined Windows PC 2. Desktop Viewer opens, and the progress wheel spins.. 3. VDA Windows logon screen is seen briefly Expected Results: The session logon process completes, and the Windows desktop is presented. Actual Results: The session closes immediately after flashing the Windows Logon screen

Citrix Confidential - Do Not Distribute

Background on the issue

XenDesktop 5 in a POC environment, XenDesktop 4 is already deployed and is in production

XenDesktop 4 sessions prompt for credentials at the Windows logon screen from the same endpoint
Explicit authentication works for both XD4&5

Citrix Confidential - Do Not Distribute

Narrowing Down the Issue

Three main components involved in session launch

Broker
SQL

VDA

Endpoint

Web Interface XML Service Controller

Workstation Agent PortICA

Online Plugin ICA Settings Desktop Viewer

Citrix Confidential - Do Not Distribute

XenDesktop Authentication Methods

Explicit Authentication
User name and password are presented directly to Web Interface site Allows Broker to validate and authenticate VDA session launch request

Pass-through Authentication
User identity is verified by IIS using NTLM or Kerberos Allows Broker to validate the user for desktop enumeration Requires endpoint device to provide credentials directly to the ICA Server

Useful for non-domain joined endpoint authentication

Citrix Confidential - Do Not Distribute

Explicit Authentication
XenDesktop 5 Broker
XML Services Controller Web Interface WCF HTTP(S) ICA SQL

Endpoint

VDA

Citrix Confidential - Do Not Distribute

Pass-through Authentication
XenDesktop 5 Broker
ICA File XML Services Controller Web Interface IIS HTTP(S) ICA WCF SQL

Endpoint

VDA

Citrix Confidential - Do Not Distribute

Reproduce the Issue

Test Cases
1. XenDesktop 4 environment using Pass-through authentication 2. XenDesktop 5 environment using Pass-through authentication 3. XenDesktop 4/5 environments using explicit authentication

Test Results
1. Reached the Windows logon screen, where I was able to login 2. Session launch fails at the Web Interface Site 3. Worked with both XD4 & XD5

Citrix Confidential - Do Not Distribute

Session Launch Fails at Web Interface

XenDesktop 5 Broker
XML Services Controller Web Interface IIS SQL

Endpoint

An error occurred while making the requested connection

VDA

Citrix Confidential - Do Not Distribute

Troubleshooting the Broker

Service Logging - CTX127492 CDF Control - CTX111961 XDPing - CTX123278 Powershell SDK - CTX127254 WCF Diagnostics- MS732009

Citrix Confidential - Do Not Distribute

Broker CDF Analysis

CdsXmlServices:2:1:ProcessCredentials: exception Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trusted CdsXmlServices:2:1:Credential Exception, reason AccessDenied: Citrix.Xms.XmlSupport.CredentialsException: ID only credentials received but TrustRequestsSentToTheXmlServicePort=false at Citrix.Xms.XmlSupport.CredentialsProcessor.ProcessCredentials(CommonCredentials RequestCredentials, CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.BaseTransaction.ProcessCredentials(CredentialType SupportedCredentials, CredentialOptions ProcessingOptions) at Citrix.Cds.Xms.Wpnbr.AddressTransaction.HandleRequest(IXmlMultiplexer multiplexer) at Citrix.Xms.XmlSupport.XmlPerf.WrapTransaction(Type t, Action transaction) at Citrix.Cds.Xms.Wpnbr.WpnbrServer.HandleRequest(HttpListenerRequest request, WindowsIdentity identity) CdsXmlServices:2:1:GetErrorIdFromCredentialsException: AccessDenied -> not-trusted
Citrix Confidential - Do Not Distribute

Troubleshooting: Broker Components

Searched Citrix KB for XML Service issues in XD5 Found that XD5 broker requires XML service to trust ID-Only credentials (CTX128328) Also required for SSO to work through Access Gateway Configure using XenDesktop 5 Powershell SDK (CTX127254)

Citrix Confidential - Do Not Distribute

Session Launch Fails During Session Initialization

XenDesktop 5 Broker
ICA File XML Services Controller Web Interface IIS HTTP(S) ICA WCF SQL

Endpoint

VDA

Citrix Confidential - Do Not Distribute

Troubleshooting VDA: Session Launch

PortICA Service Logs (CTX118837)
Portica.ICA.IcaClientStack.GetCredentials CdsWorkerAgent:2:1:Validate no credentials returned Portica.BizLogic.TakeOwnershipOfCredentials Portica.GinaServer.SendAutoLogonMessage

Workstation Agent Service Logs (CTX127492)

CDF Trace Modules:

CdsWorkerAgent Portica_DLL_PICACredProvider ICA Service Portica_DLL_PICADisplayManager Utils.Kernel32.UnmanagedBuffer.SafeDisposeObj MF_Session_Wfshell Portica_DLL_PICASessionHelper ThreadID=7, disposing=True, pointer=32C60E8, size=1568, MF_DLL_Ctxgina Portica_Library_picaCPHelper source=Citrix.Portica.GinaServer.SendAutoLogonMessage MF_Library_System

Portica.GinaServer.ProcessGinaMsg Received message of type: CancelIcaConnection

Citrix Confidential - Do Not Distribute

Troubleshooting: VDA Components

Enforce Auto Logon (CTX127392) Requires credentials to be passed, or the session is canceled Enabled by default in XD5 for security purposes Can be manually set on VDA
Create DWORD value on the VDA called 'EnforceAutoLogon' in HKLM\Software\Policies\Citrix, and set it to 0

Citrix Confidential - Do Not Distribute

Troubleshooting Online Plugin

Directory must exist, and be writable

Endpoint

ICA Logging - CTX115304 CDFControl - CTX124934 DebugView - BB896647 Client Policies - EDocs

Enable LogEvidence for CST

Citrix Confidential - Do Not Distribute

ICA Log Analysis

[KB-Win7-x32RTM] Address=10.54.67.97:1494 AutologonAllowed=ON BrowserProtocol=HTTPonTCP ConnectionBar=1 InitialProgram=#WinXP 32-bit $P8 Launcher=WI LaunchReference=EE2998E87E058B78E1CAF7050FB40E SessionsharingKey=-R7YM1LL1qw5bcb7LTq21sC UseLocalUserAndPassword=On

Desktop Group
ICA Address

Auto-Logon Allowed
Desktop Viewer Single Sign-On

Citrix Confidential - Do Not Distribute

Pass-through Authentication Requirements

Searched Citrix KB for UseLocalUserAndPassword

Citrix Confidential - Do Not Distribute

Pass-through Authentication Client Policy Settings

Citrix Confidential - Do Not Distribute

Pass-through Authentication CST Override

Allows all regions except Restricted

Citrix Confidential - Do Not Distribute

Client Selective Trust (CST)

Collects and analyzes evidence from session launch details Classifies ICA sessions into one of four regions:
oidTrustedRegion oidIntranetRegion oidInternetRegion oidRestrictedRegion

Checks WI Site against Internet Explorer security zones Blocks certain ICA Client actions (such as Pass-through) based on region settings (CTX124871) Requires CST registry keys to be present (CTX128775)
Citrix Confidential - Do Not Distribute

ICA Log Analysis - CST Evidence

ICA Client connection initialized AddEvidence InitialProgram=#KB-Win7-x32RTM Region All Regions AddEvidence ICAFileAddress=XenDesktop.get.services.citrite.net:1494 Region Trusted Region AddEvidence ServerAddress=XenDesktop.GET.SERVICES.CITRITE.NET CTX124921 Region Trusted Region AddEvidence CGPEnabled=True Region All Regions AddEvidence ServerIPAddress=10.54.67.220 Region All Regions EvidenceRequest Connection Authorisation (event: Open connection to Citrix Server) Granted

Collect Inspect Select Authorize

Citrix Confidential - Do Not Distribute

Desktop Viewer CST Requirements

CST evaluates Initial Program value as evidence Requires the desktop group name to be added to the CST whitelist if Allow pass-through for all connections is not enabled Used DebugView output to determine what evidence was being evaluated

Citrix Confidential - Do Not Distribute

CST Whitelist
KB-Win7-x32RTM] Address=10.54.67.97:1494 AutologonAllowed=ON BrowserProtocol=HTTPonTCP ConnectionBar=1 InitialProgram=#WinXP 32-bit $P8 Launcher=WI LaunchReference=EE2998E87E058B78E1CAF7050FB40E SessionsharingKey=-R7YM1LL1qw5bcb7LTq21sC UseLocalUserAndPassword=On

Wildcards dont work here

Citrix Confidential - Do Not Distribute

Pass-through Authentication
XenDesktop 5 Broker
ICA File XML Services Controller Web Interface IIS HTTP(S) ICA WCF SQL

Endpoint

VDA

Citrix Confidential - Do Not Distribute

Root Cause Analysis

Broker
Required Set-BrokerSite -TrustRequestsSentToTheXmlServicePort $true

Virtual Desktop Agent

Enforce Auto Logon requires the ICA Client to automatically send credentials during ICA session launch

Endpoint
Client Selective Trust requires additional client policies to be used Pass-through authentication is treated more securely than explicit authentication

Citrix Confidential - Do Not Distribute

Resolution
Provided a private binary that instead evaluates the ICA address, which supports wildcards

Client Selective Trust is being replaced by ICA File Signing

Recommending ICA File Signing as a replacement (eDoc)

Citrix Confidential - Do Not Distribute

Resources discussed

For More Information

CTX127492 - How to enable Controller Service Logging in XenDesktop 5 CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics CTX128909 - XenDesktop 5 Logon Process and Communication Flow CTX127969 - Desktop Studio Logging Options CTX127587 - XenDesktop 5 Reference Architecture CTX128190 - How to Change Virtual Channel Priority in XenDesktop 5 CTX127254 - XenDesktop 5 SDK PowerShell Cmdlet Help

Questions and wrap up