With the release of XenApp and XenDesktop 7.12 Citrix brought back one of the most requested features from the XenApp 6.x days – The Local Host Cache (LHC). For those of you new to this term, it essentially provided a way for users to connect to their XA/XD published resources while the SQL based database is down but keeping a local cache on the XenApp servers themselves. LHC now replaces Connection Leasing in 7.x as the primary mechanism to allow connection brokering operations when database connectivity to the site database is disrupted. In this post, my goal is to dig into the architecture of Local Host Cache in 7.12 and how it works.
The above diagram from Citrix Documentation shows the architectural components that make up the Local Host Cache. The feature is disabled out of the box when XA/XD 7.12 is installed. If you are upgrading from a previous version LHC will be disabled under certain conditions. See the table below for further details.
With LHC, users can connect to Apps and Desktops that they have previously not connected to. This was not possible with Connection Leasing where users could only connect to resources that they had previously connected to.
Every broker now has three new services. The primary broker service, the secondary broker service and the configuration sync service.
LHC sychronization during normal operation and central database connectivity is not affected
- During normal operations, the primary broker service communicates with the site database while the secondary broker service remains idle. The CSS makes sure the local db on each of the controllers is synchronized periodially.
- Primary broker service accepts connection requests from Storefront, then communicates with Site DB and provides users access to VDAs registered with the controller and that they request access to.
- Every 2 minutes, a check is made to see if there have been any changes to the primary broker config.
- If a change is detected, then the primary broker uses the Citrix Config Synchronizer Service (CSS) to copy configuration to a secondary broker. This is not an incremental copy but a full copy from the primary broker to the secondary broker.
- Secondary broker then imports the configuration to a local SQL Server Express database on the controller.
- Once the config is copied the CSS service confirms that the config on the secondary broker matches the config on the primary broker.
- Local DB on the secondary broker is recreated each time a config change is detected on the primary broker (checked in 2 minute intervals)
- Secondary broker runs as a Windows service called Citrix High Availability Service
What happens when there is an outage and database connectivity is lost
- During an outage, the primary broker can no longer connect to the site database and stops accepting connections.
- Primary broker instructs secondary broker to start listening for and processing connection requests. An election process ensues to determine which controller takes over the secondary broker role. There can only be one secondary broker accepting connections during a site db outage.
- When the VDAs start communicating with the secondary broker, a re-registration process is triggered and the secondary broker gets current session information about the VDA.
- During the outage period, the primary broker continues to monitor the connection to the site database and when connectivity is restored, it instructs the secondary broker to stop listening for connections and the primary broker resumes brokering connections thereby restoring normal operations.
- When a VDA communicates with the primary broker after it has taken over brokering, a re-registration is triggered.
- The secondary broker removes all VDA registration info during the outage and continues checking for config updates on the principal broker every 2 minutes and updating its LHC when changes are detected.
- If an outage occurs during an LHC sychronization, the current import is discarded and the last successful imported config is used.
- It is important to note that during an outage, only one active secondary broker is available. So from a scalability perspective this could be a limitation. The secondary broker as mentioned earlier is chosen based on an election mechanism.
Local Host Cache and Citrix Cloud
- If you are currently leveraging Citrix Cloud for your XA/XD control plane, the LHC functionality ensures that connectivity loss to the control plane does not impact users from accessing their resources.
- LHC synchronization occurs the same way as it would in an on premises XA/XD deployment and the config changes are synchronized from the Citrix cloud via the Cloud connector.
- To provide fault tolerance when connectivity to the Citrix cloud is lost altogether due to a WAN link failure, Citrix Storefront and potentially Netscaler would need to be on premises.
Local Host Cache Restrictions
- You cannot run Studio and Powershell Cmdlets when LHC is active and site database connectivity is down.
- Site configuration changes cannot be made when the connectivity to the central database is unavailable. This is very similar to the IMA based LHC implementation in XenApp 6.x
- New machines cannot be provisioned as hypervisor interaction is not possible when LHC is operational.
- Users cannot be assigned new resources during the site database connectivity outage.
- Machines with a “Shut down after use” configuration will be placed in maintenance mode when LHC is operational
The two main tools to troubleshoot LHC are the Windows Event Logs and CDF traces.
- The Config Sync Service logs events in the Windows Event logs in relation to LHC synchronization. If no config changes occur during the 2 minute intervals, no events are logged. If CSS receives a config change, the event is logged with event id 503. If the update to the secondary broker is successful, the event is logged with event id 504. If the update fails, the event is logged with event id 505
- When the secondary broker takes over during an outage, event log entries are made indicating that the Citrix High Availability Service has started handling brokering. Once services are restored, you would see logs indicating that the Citrix High Availability service has stopped brokering. There will also be events related to secondary broker election. Event IDs include 3502, 3503,3504 and 3505. When Citrix Cloud is in play, XA/XD proxy log events are present. CDF traces can also be used for advanced troubleshooting.
Enabling Local Host Cache After Upgrading
Local Host Cache is not enabled by default when upgrading from an earlier version of XenApp and XenDesktop 7.x. I have written a blog on how to enable LHC after an upgrade.