Choosing the correct Identity Provider (IdP) for Citrix Cloud

Updated 22-09-2021!

Choosing the correct Identity Provider (IdP) for your new Citrix Cloud environment is one of the most discussed items and one of the first points when starting a new deployment. Most organizations already have an Identity Provider (IdP) and would like to give users the easiest way to migrate to their new deployments. Sadly it’s not always possible to keep it that simple.

I explain the pros and cons as much as possible and what I learned during my latest projects. I don’t have any experience with Okta or SAML, so I can’t give all the details about those issues. If anyone has additional information about the integration with Okta or SAML, please let me know, and I update the article. I look at Citrix Virtual Apps and Desktops services and Gateway Services integration for SSO to SaaS apps or Enterprise web apps.

My projects are starting with a migration of their traditional on-premises Citrix Virtual Apps and Desktops (CVAD) deployment and would like to move to a cloud setup. The reason for this varies per customer. At the start of a new project, we talk about customer requirements. The topics covered most are authentication, security, and application compatibility. When using Citrix Cloud, it is essential to make it clear which IdP to choose at the start of the project because we get more and more authentication options than on-premises when implementing a CVAD environment.

Scenario’s and requirements

Below I describe some scenario’s

  1. Double hop
    There are many customers that use published apps that start from another VM within their published desktop (called a double hop), and there are numerous reasons:
    1. The app is for archive only;
    2. The app doesn’t support the same OS as the desktop;
    3. The app conflicts with other apps;
    4. The app can only be used by some users, and licenses don’t allow us to install it on the same image as the desktop.

      In the case of a double hop scenario, you need an on-premises StoreFront, which means you still need to manage the StoreFront server and keeping it up to date. To configure a double hop with an on-premises storefront take a look here.
  2. Federated Authentication Service (FAS)
    When choosing AAD, Okta, or SAML for your CVAD requires a FAS server (or two for redundancy). FAS uses certificates to sign in to the VDA’s. To use FAS requires a Microsoft Certificate Authority. To make this secure, you have to install it with an offline root CA and an online issuing CA.
  3. SSO to Saas or enterprise web apps limitations
    When choosing AAD, Okta, or SAML with Citrix Gateway services, you could not use all SSO solutions to your SaaS apps. Basic and form-based SSO will not work. This is because when you authenticate, your credentials (username and password) aren’t cached in a hash on the gateway services. Only the authentication token is sent to the Gateway, and thus basic, and form-based SSO can’t get the credentials.   https://docs.citrix.com/en-us/citrix-gateway-service/support-web-apps.html
  4. Second public domain needed for Azure federation
    When choosing AAD, the primary domain is used for authentication to Citrix Cloud. You can’t use the same domain for federation to Azure for SaaS apps. This mostly isn’t a problem because most companies have multiple public domains, but you need to think of it. See: https://docs.citrix.com/en-us/citrix-gateway-service/saas-apps-templates/citrix-gateway-o365-saas.html#authentication-methods-supported-for-office365
  5. Password change not supported within Citrix Cloud
    When choosing AAD, Okta, or SAML, you can’t use the default option to let users change their password, and you need to configure this on the chosen IdP. This won’t stop your project, but you need to keep this in mind.
  6. Existing MFA provider
    When you need to support an existing MFA provider (RSA, Gemalto, etc.), you can get this to work with an on-premises setup only or with Okta.
  7. No support for authentication to Office 365
    When you need to authenticate your Office 365 applications from the Citrix Gateway services, this isn’t supported for Okta and SAML.
    https://docs.citrix.com/en-us/citrix-gateway-service/saas-apps-templates/citrix-gateway-o365-saas.html#authentication-methods-supported-for-office365

Identity Providers

Looking at the available IdP’s, Citrix Cloud currently support the following:

  1. Active Directory
  2. Active Directory + token
  3. Azure Active Directory (AAD)
  4. Citrix Gateway (on-premises)
  5. Okta
  6. SAML 2.0

I explain all the IdP’s one by one and give some pros and cons I experienced during my last projects.

On-premises AD with or without token

I combine options 1 & 2 because the main functionality is the same. The only difference is a token code that gives some extra security.

Using your on-premises AD (with a token) is the easiest setup. You only need to add two cloud connectors, and you’re good to go. You can use all the services without any additional requirements. In the case of Citrix Gateway SSO, this enables you to also use form-based authentication with on-premises SaaS apps.

Pros:

  1. Easy setup
  2. No additional requirements
  3. Free MFA
  4. SSO with form-based authentication
  5. Allows password Changes

Cons:

  1. When using this token, users need to replace their current MFA or need another MFA solution.

Azure Active Directory (AAD)

As more and more companies are using Azure AD, as needed when using Microsoft 365, Microsoft keeps adding features to the suite. One of those features is Azure MFA, where you can use conditional access (with the appropriate license). With conditional access, it’s possible to give users the option to only sign in with username and password from trusted locations (think about HQ and branch offices).

Pros:

  1. You can use your existing Azure MFA for Citrix
  2. You don’t need additional MFA products
  3. Conditional access
  4. Users can sign in with there same username and password as AD

Cons:

  1. FAS server required
  2. Double hop only with on-premises StoreFront
  3. SSO to SaaS or enterprise web apps limitations
  4. Second public domain needed for Azure federation
  5. Password change not supported with Citrix Cloud
  6. Can’t use existing MFA solution

Citrix Gateway (on-premises)

When you have a Citrix Gateway (f.k.a. NetScaler Gateway), you could use that to manage the authentication to your on-premises domain. Using the Gateway gives you more flexibility to the way users are going to sign in.  When looking at MFA, the on-premises Gateway gives you the possibility to use your current MFA solution, like Gemalto, RSA, or Azure MFA. The downside is that you will need to maintain an on-premises gateway. And because that is your point of entry to your environment, you need to keep it very secure.

Pros:

  1. More flexibility in authentication providers with or without MFA
    When using MFA, make sure the LDAP password is the default password (https://docs.citrix.com/en-us/citrix-cloud/citrix-cloud-management/identity-access-management/connect-ad-gateway.html#troubleshooting)
  2. Keep using load balancing
    When using an on-premises gateway, you could leverage all the functionality of the appliance. This gives you the option to use it for load balancing features

Cons:

  1. More costs and management you need to manage, update, and buy support for the appliance. Which will cost you more than using the Gateway Services  managed and updated by Citrix
  2. Password change not supported with Citrix Cloud
  3. Double hop only with on-premises StoreFront

Okta

Pros:

  1. Use existing MFA solutions (if supported)
    https://help.okta.com/en/prod/Content/Topics/Security/mfa/about-mfa.htm
  2. Use Okta as your IdP for multiple scenarios

Cons:

  1. No support for authentication to Office 365
  2. FAS server required
  3. Double hop only with on-premises StoreFront
  4. SSO to SaaS or enterprise web apps limitations
  5. Password change not supported with Citrix Cloud

Security Assertion Markup Language (SAML)

SAML is currently in technical preview. SAML 2.0 is now generally available. All pros and cons are based on information that Citrix provides in its documentation. Take a look at the Citrix blog for more info.

 Pros:

  1. Use the same SAML setup for more partner/app integrations

Cons:

  1. No support for authentication to Office 365
  2. FAS server required
  3. Double hop only with on-premises StoreFront
  4. SSO to SaaS or enterprise web apps limitations
  5. Password change not supported with Citrix Cloud

Looking at all these pros and cons could make it hard to choose the right Identity provider. I hope this article is a good starting point and makes a choice easier so you won’t have the same issues I had during my latest projects.
Would you please let me know if you have some pros and cons I have forgotten to mention or need some additional information?

Setting up a double-hop with Azure AD as IdP

As more and more companies are moving from on-premises environments to a cloud and would like to make the best of their investments, we see a great demand in moving to Azure Active Directory (AAD). AAD gives you some benefits as you could utilize Azure Multi-Factor Authentication (MFA) with Conditional Access. With Conditional Access, it’s possible to give users the option to only sign on with username and password from trusted locations (think about HQ and branch offices).

This article explains how to set up a double-hop scenario with Citrix Virtual Apps and Desktop (CVAD) services (Citrix Cloud), where your identity provider (IdP) is Azure AD.

What’s a Double-Hop, and why use it?

A double-hop scenario is when you start a published app from other resources within your published desktop. An example:

You have an app that doesn’t support the OS you would like all users to have. In most situations, this will be Windows Server 2019 for multi-session purposes. The one app that you would like users to open is a legacy app and only supports Windows 2012 R2 as the latest supported OS. As your organization states that you only work with supported configurations, you can’t install this app on Windows 2019. To ensure the user can access the app, you create a dedicated delivery group based on Windows 2012R2 VM’s and use this delivery group to give users access to the app. Those users start this app within their published desktop as a seamless app and don’t notice if they are local or remote.

The above scenario is called a double-hop. As a company, you could have many reasons to use this scenario. Think about:

  • The app is for archive only;
  • The app doesn’t support the same OS as the desktop (mentioned in the example);
  • The app has conflicts with other apps;
  • The app can only be used by some users, and licenses don’t allow us to install it on the same image as the desktop.

Challenges when using double-hop with Azure AD as IdP

When you sign on to Citrix Cloud with your Azure and open your published desktop, your on-premises environment uses Federated Authentication Services (FAS) to sign on to the VDA. Look at the below diagram to see how FAS is working (thanks to Daniel Feller (https://virtualfeller.com/)).

When you would like to have SSO from your published desktop to Citrix cloud, it’s only possible to use Azure Active Directory Seamless Single Sign-On or using an on-premises StoreFront environment that supports FAS authentication. Azure seamless SSO requires the use of Pass-through Authentication (PTA) or password hash synchronization. The customer where I’m currently implementing CVAD services is transitioning to Azure AD, and they don’t have seamless SSO enabled. That’s the reason we need to use another option to create the double-hop scenario. The only other option, as mentioned, is to install an on-premises StoreFront that supports FAS authentication.

Configuring StoreFront and FAS

In this example, I only have one StoreFront server and one FAS server to keep this guide simple. Below you could see how the authentication is working using a double-hop scenario.

StoreFront & FAS

After you installed and configured StoreFront, we configure it to allow FAS as an authentication method.

  1. First, enable FAS support on your Store, change “/Citrix/Store” to your store name:

    Get-Module “Citrix.StoreFront.*” -ListAvailable | Import-Module
    $StoreVirtualPath = “/Citrix/Store”
    $store = Get-STFStoreService -VirtualPath $StoreVirtualPath
    $auth = Get-STFAuthenticationService -StoreService $store
    Set-STFClaimsFactoryNames -AuthenticationService $auth -ClaimsFactoryName “FASClaimsFactory”
    Set-STFStoreLaunchOptions -StoreService $store -VdaLogonDataProvider “FASLogonDataProvider”

  2. Secondly configuring the Delivery Controllers

    Click Manage Delivery Controllers
    Usually, you would add your On-Premises Delivery Controller here, but now you need to add the Cloud Connectors here. You need to use port 80 (HTTP) and can’t use 443 (HTTPS) as a transport type.

  3. Set Authentication Methods

    Click Manage Authentication Methods
    Enable Domain Pass-Through

  4. Set the GPO Federated Authentication Service

    As FAS is already working, the ADMX files are probably in the PolicyDefinitions central Store, so I skip the step to copy these ADMX files. For the StoreFront server, configure a GPO, so it knows which servers are FAS servers.Open the GPO and navigate to Computer Configuration/Policies/Administrative Templates/Citrix Components/Authentication
    Edit the Federated Authentication Service
    configure the DNS addresses of your FAS servers
    Perform a gpupdate /force to make sure the GPO is applied.

  5. As FAS is configured to access your VDA, we only need to change the rule you use for FAS.

  6. Open the FAS Administration Console and go to the tab rules

  7. I only have 1 rule named “Default” click on the Pencil to edit the rule.

  8. Go to Access Control
    Click Manage StoreFront access permissions

  9. Here you add the Computer account of your StoreFront server.
    Make sure you remove the default “Domain Computers” as that one denies access.
    Click OK

  10. Now click Apply, and your FAS configuration is ready for testing.

Testing

After you configured all the settings as described above, It’s time to test the configuration.

When you sign in to your published desktop, your Citrix workspace app (when configured correctly) is signing in to the StoreFront site, and you could start an app that’s published from your start menu. Please try, and let me know if this is working.

Citrix WEM optimizing multi-session OS machines

Update 16-12-2021:
As of Citrix WEM version 2112 that’s just been released this function is also available for on-premises deployments. Read more about this version here.

You could use Citrix Workspace Environment Management (WEM) for a lot of things within your Citrix Virtual Apps and Desktops (CVAD) environment(s). It’s also suitable to use for your workstations, but I don’t see that very often. I only use it for CVAD environments, which includes our own and our customer’s environments. WEM can be used On-Premises with the Current Release or used as a service using Citrix Cloud.

As more and more of our customers are looking to simplify management, including all back-end (Citrix Delivery Controllers, Storefront, WEM Infrastructure) servers, we started to migrate to Citrix Cloud. This gives the customers the possibility to use still On-Premises workloads (Virtual Apps and Desktop VM’s) but makes management of the previously mentioned back-end servers easier as Citrix manages this. Below you could see a diagram of all “Managed by Citrix” and “Managed by Customer” components.

Citrix managed or Customer managed

Multi-Session Optimization

Back in October 2020, Citrix released a new feature called “Multi-Session Optimization,” which I recently discovered when going through the WEM Administration console. Sadly this option is only available within WEM Services and not yet available with the On-Premises release. I think this is a matter of time because Citrix Cloud is always a step ahead of the On-Premises version. Citrix Docs WEM

How it works

Multi-Session Optimization can be used with a supported Multi-Session. When a user disconnects his session, the WEM agents lowers the CPU and the I/O priorities of processes or applications associated with the session. This means that the applications a user leaves active are then managed by WEM and will use fewer resources. When you have large environments with multiple disconnected sessions, users can use the resources typically used by the disconnected sessions, and then you could have more users share a VM. We have multiple environments where users don’t understand how to sign out from their session and click disconnect. This is because they were used to click the start button and then click sign out. Starting with the release of Windows 10 (same layout for Windows 2016/2019), they need to click start and then click on their user’s icon and then select “Sign Out.” We created additional Sign Out buttons for our customers that we pinned to the start menu but, the clients keep disconnecting.

To enable Multi-Session Optimization you open the Citrix WEM Administration Console from your Cloud management page and go to the System Optimization in the user interface. There you see the option Multi-Session Optimization. Now check “Enable Multi-Session Optimization” and click Apply (see image below). To make sure the Agents receive the new setting you could perform a Cache Refresh. After you enabled Multi-Session Optimization you need to monitor this in your environment and if needed you could exclude some Groups or Processes.

Enable Multi-Session Optimization

Multi-Session Optimization in action

For this article, I only used one session to test this newly discovered option, but it shows how it could save many resources if there are multiple users disconnected from the Multi-Session OS. In the first screenshot, you see an active user with all applications open and you can see that this user is using 1.4GB of memory. This is not that many, but it’s just an example.

Active Session

In the screenshot below, you can see that the user is disconnected, and as you would normally see, the memory usage keeps the same or is even higher in this example.

Disconnected Session

After one minute of waiting, the WEM agents kicks in and changes the priority of the resources associated with the disconnected session. You see that the CPU usage goes from 6.4% to 0.1%, and the memory usage goes from 1.5 GB to 262 MB. This could have a large impact if you have multiple disconnected sessions on one VM.

WEM Kicks in

I’m now going to test this in this environment and keep you updated if I have some additional recommendations that would help you keep your users have the best user experience. The need for resources management is getting even more important as users are using more and more resources, and looking at VMs in the Cloud where you pay for CPU and Memory usage this could save you money. Eventually, users need to sign out because that’s the most effective way of saving resources.

Using Zoom with SBC\VDI

Working from home (WFH) requires having a lot of video calls. I noticed a great demand from users that need to attend or organize a video call. The downside is that there are multiple vendors and that a company can’t force everybody only to use one solution. Users receive invitations from multiple organizations, and they all need their own client applications. When using an SBC\VDI based remote workplace, you need to manage all the clients and make sure the video call works great. Luckily, the vendors are working together and making sure that you could have a great experience even when working in a remote session. One of those vendors is Zoom, they have created a client that is created for a VDI deployment. Zoom currently supports Citrix, VMware, and WVD.

How it works

When making a video call the video rendering requires CPU and Memory, when using a multi-session OS this could cause other users to have a degraded performance. The users that are making the video call could notice latency during the call because every moving image has to be rendered and send to the client. To get a better experience Zoom created software that could offload the rendering to the local clients. How is that offloading working?

In most cases, the media plugin will offload the video encoding and decoding and communicates directly to the Zoom cloud, bypassing the VDI infrastructure. All other information is sent over the VDI channel. There are three options:

The three options
The three options
  1. Everything is within the VDI and all the resources are used within the VDI, communication is from the VDi to the Zoom Cloud
    This won’t work with Citrix as described here: https://support.citrix.com/article/CTX270931
  2. Only the Video encoding and decoding are rendered on the client and all communication is going from the Zoom media plugin through the VDI channel to the VDI and from the VDI to the Zoom Cloud.
  3. Most of the communication is going from the Zoom Media Plugin to the Zoom cloud and only the Authentication and window location is going from the Zoom Media Plugin through the VDI channel to the VDI and then connects to the Zoom Cloud.

Installing

Time needed: 15 minutes

Installing the Zoom VDI Installer in your base image and your client.

  1. Download the latest Zoom VDI Installer client and the corresponding Zoom Media Plugin

    https://support.zoom.us/hc/en-us/articles/360041602711
    You need to download Zoom Media Plugin based on the environment you have: Citrix, VMware, or WVD.

  2. Install the Zoom VDI Installer within your VDI

    This is basically a Next,Next,Finish installation.

  3. Install the Zoom Media Plugin on the Client

    To make sure you don’t receive an error when using Zoom, you need to run the installer as an Administrator.
    Close all Citrix, VMware, or WVD clients before starting the installation.
    Then follow the wizard.

Verifying Zoom installation

After installing the Zoom VDI and Zoom Media Plugin, the testing starts. I made a Zoom call with myself. As I’m not so photogenic, I will spare you these images. To give you a good view of how it’s working. First, I made a Zoom call (without video in my session as this isn’t working for Citrix https://support.citrix.com/article/CTX270931), you can see there is 9.7% CPU usage. This isn’t that much, but when there are more people on a VM and you share the CPU, this could be affecting the user experience of other users.

Zoom call without the plugin enabled

Below you can see that that the Zoom client shows it’s not connected to any plugin and is working as a regular Zoom client.

Zoom session without a working plugin
Zoom without VDI plugin connected

After the first test, I installed the Zoom Media Plugin (for Citrix) and made another Zoom call. This time I was able to use my local Webcam (Laptop integrated) and as you could see below the CPU usage is 0% within the Citrix Session (First Image) and uses 8.4% CPU on the local workstation (Second image).

Zoom Session CPU usage within the Citrix Session
Zoom Session CPU usage within the Citrix Session
Zoom Session CPU usage local workstation
Zoom Session CPU usage local workstation

When looking at the statistics within the Zoom client you now see that the Zoom VDI plugin is Connected. The rendering is now completely offloaded to the Clients workstation and this saves you CPU and Memory on the VM (SBCB\VDI) and gives the user a good user experience during the call.

Zoom Media Plugin Connected
Zoom Media Plugin Connected

Features

When using the Zoom VDI client and the Zoom Media Plugin, you don’t have all the features that are available when using the Full Desktop version of Zoom. As this changes with every new release, I advise you to look at the comparison matrix that Zoom created and hopefully kees updated: https://support.zoom.us/hc/en-us/articles/360031441671-VDI-client-features-comparison

Tips and Tricks

I will update the Tips and Tricks section if new information is available or I needed to solve an issue that could help you as reader.

Registry Settings

Zoom has a list of Registry Settings that can help you troubleshoot or control the client. Here is a list of all available registry keys: https://support.zoom.us/hc/en-us/articles/360032343371

Error Zoom VDI

As mentioned in step 3, if you don’t install the Zoom Plugin on the local device, you could see an error. I only tested with the Citrix Media Plugin and can’t verify if the same happens with VMware or WVD. The error I received is:

Zoom.exe is using NTDLL.dll from an unknown publisher. Are you sure you want to run this software?

Error when Zoom Media Plugin is wrongly installed.
Error when Zoom Media Plugin is wrongly installed.

You could click on Yes as often as you like but that’s not going to work.

The need for a GPU in VDI/SBC environments

There are already many blogs about the pros and cons of using GPUs in VDI and SBC environments. I also would like to point out why I advise the usage of GPUs in these types of environments.

CPU vs. GPU

First, let me explain the difference between CPU and GPU. A CPU is designed to handle a few software threads at a time using its few cores and lots of memory cache. A GPU is designed to handle thousands of threads using their hundreds of cores.

In a production situation, the above difference means that when you have an application that wants to use a GPU but there isn’t one available, it will use the CPU instead. The downside of using a CPU is that it’s slower than a GPU and more expensive. A user can notice this when his programs are sluggish and not performing as expected. You can read more about the difference between a CPU and a GPU here.

VDI/SBC without GPU

In the early days, we had Windows 7 or Windows Server 2008R2 without a GPU and that would work perfectly. Users could do their work as if they were working on a local device. The architecture of the OS and most of the applications one was using, didn’t depend on a GPU. If they needed a graphics card and the server had none, Citrix offered a tool called OpenGL Software Accelerator and the VMware alternative was Soft 3D. This would work for some lightweight applications that required a graphics card and get fooled using the software.

In the last couple of years, we see increased usage of GPU accelerated programs like Chrome, Edge, Firefox, Dynamics, Teams, etc. Windows 10 is also more graphics-intensive than it was with Windows 7. A lot of applications already recommend or even require GPUs. They will function without a GPU but to get the best user experience (UX), you definitely should implement GPUs.

Maintaining the very best UX is exactly why I always advise the use of a GPU. I did several VDI projects where they were creating 3D drawings. In such cases implementing GPU is a no-brainer, because the 3D acceleration completely depends on it. You can read about one of those projects here. Those environments are not the ones that I would like to explain.

Looking back at projects starting with Windows Server 2012, users started to use more and more graphic intensive websites. They started consuming more video content like YouTube, online learning platforms, and websites that are more and more requiring GPU. Both Citrix and VMware developed solutions to keep giving the user the best UX without having to install graphics cards also. The downside of these software solutions is that everything is rendered using the CPU. That means you needed a faster (read: more expensive) CPU and when someone was using a graphics-intensive application, other users could start complaining that their system was slow. This all because the CPU couldn’t handle all requests fast enough.

VDI/SBC with GPU

And then came Nvidia, offering virtual GPUs (vGPU), where we could assign multiple VMs to one physical Graphics Card and give a portion of the card, this is called profiles. First, only Citrix supported this with Citrix Hypervisor (f.k.a. XenServer), and then also VMware supported this type of virtual GPUs. Nowadays we can create more VMs on the same physical server because we can assign more VMs to one GPU graphics card.

This all had a positive effect on the return on investment (ROI) for a VDI or SBC environment with GPU cards. Because of the profiles, we need both fewer physical servers and CPUs, resulting in less Rackspace, less maintenance, and fewer investments. Now we can start using the vGPU more often and help users get the best UX with the new and more graphics-intensive applications.

Almost always a GPU

Nowadays we almost always advise customers to buy GPUs for their VDI/SBC environment. We have a couple of reasons why we don’t advise GPUs:

  1. When they have a small number of VMs (≤ 4) and adding them to their already existing backend VMware environment. To use the GPU profiles, VMware requires the use of VMware vSphere Enterprise Plus. When customers don’t already have an Enterprise Plus license, it’s usually too expensive to add GPUs;
  2. That it’s just a temporary environment because they are migrating to local machines or the cloud.
  3. Usage, some customers are using the EUC environment just for some users or only when they have to work in remote offices. When this isn’t a daily situation, we can decide to only use CPU.

Conclusion

Based on the above pros and cons, you definitely need to consider GPUs when upgrading or migrating your VDI/SBC environment.

Looking back at 2020 and Work From Home

The end of 2020 is nearing so it’s the time to look back at this year, I can say it was a hectic year for me as a Consultant.

From PoC to production

We just started a Proof of Concept (PoC) with a new customer in February to show that working with a 3D modeling program on a VDI was an option for their clients. Then everybody started talking about some kind of virus in China and started reporting lockdowns. Just days before the Dutch first lockdown (23rd of March), the customer started asking to change the PoC to production and expanding from 10 people to 150 remote workers. We managed to achieve that. You can read more about this in my previous article: From PoC to Production during Covid-19

During the customer’s request to expand their PoC, all other customers requested more Work From Home (WFH) possibilities because of Covid-19. They requested the implementation of Microsoft Teams ASAP, my colleagues and I had to achieve this during the lockdown. Simply enabling Microsoft Teams or creating all the accounts in Microsoft 365 isn’t the hardest part. That came when they all wanted it on their sometimes outdated Citrix environments. 

Microsoft Teams

As mentioned before, many customers asked for Microsoft Teams. Luckily Citrix made its first Teams optimized for Virtual Apps and Desktops (VAD) available in June 2019 (VAD 1906). It’s a pity that most of the projects started before the availability of VAD 1906 and that they were all working with VAD 7.15 (LTSR). So before we could eventually start implementing Microsoft Teams optimized for Citrix VAD we had to upgrade all of these customers to VAD 1906 or newer. We prefer the use of the LTSR version, so we upgraded everyone to VAD 1912 LTSR. They also had to upgrade all of their clients with the latest Citrix Workspace App. 

The lessons we learned here is that we as an engineer didn’t like this kind of implementations. my colleagues and I all needed to find the best settings. Due to the stress and lack of time we all made different choices, we all reinvented the wheel. This leads to different experiences with Teams at some of our customers.

After the upgrade of the Citrix VAD and making all the necessary changes to get Teams optimized, we ran into some issues. The Clients!!! We faced a couple of issues:

  1. They didn’t have any Device Management solution in-place to upgrade all the Citrix Receivers to the latest workspace app;
  2. They have older (Thin) Clients that aren’t compatible with the latest Workspace App;

The lack of a compatible Workspace App resulted in the excessive CPU and Memory usage, which resulted in other users complaining about a slow User Experience. 

By now we have a default set of steps and requirements to get Teams working correctly, but it keeps hard to manage the CPU and Memory usage. 

Working from home as a consultant

Not only our customers needed to work from home, but us as a consultant also. I’m used to working at the customer’s location because that makes me part of the project. When on-premises you hear a lot of extra information that could help your project make it a success. Working from home removes all those benefits and makes it sometimes hard to understand the issue a customer is facing. When they try to explain an issue we as a consultant don’t understand it and vice versa.

Besides not understanding each other, the most thing I missed (and I think everybody is) is the personal contact with people. Microsoft Teams is making WFH possible and can give you some kind of personal contact, it’s not the same.

Looking at 2021

Now December is ending but the infection numbers are still going up, so we keep working from home. That means we keep helping our customers implementing and upgrading their environments to meet their WFH requirements. I just started a new project, where I can implement all the lessons learned.

For 2021 I set myself some goals, I’m not going to tell them because then I need to tell you next December I’ve not made my goals. 😊 But some nice projects coming up, personally and business-wise. We get vaccinated, meet others again, have a beer in the pub, enjoy a festival, and finally have a normal conference.

I hope you all are well and despite the current situation enjoy Christmas and have a happy new year.

See you in 2021.

From PoC to production during Covid-19

In the beginning of this year I was asked to look at a Citrix Virtual Apps and Desktops (VAD) environment which wasn’t performing great. Customer’s users were constantly complaining about performance and stability. The environment was set up in a mix of outdated software versions, which weren’t always compatible with each other.

I was asked to look into this matter and create a working environment with the current Windows 10 version on their existing hardware. We also offered a temporary server with the latest Nvidia T4 graphics card, to let them experience the enhanced performance with this new kind of graphics card. A Nvidia T4 card is a graphics card that offers GPU’s, so the VDI’s (or RDS) can perform more powerful tasks than integrated graphics adaptors do.

The Proof of Concept (PoC)

I started of with a simple single non High Available (HA) “design”, where we combined as much components to one server as possible. This was done from both a cost and management perspective.

I prefer to work with software I have experience with, so I installed Citrix VAD 7.15 LTSR. Yes, this isn’t the best version to show every new 3D HDX improvement, but it is a very stable version and we were sure we could provide the customer a much better experience than they were having before. We installed all the required applications within their new golden image, without optimizing them or check for Win10 compatibility.

After a week the PoC users were able to test their brand new workspace and informed us constantly how they experienced the new PoC desktop. The results were great. Everyone was enthusiastic and could do their work from anywhere. You can imagine what the experience could be after best-practice deployment and some finetuning.

Covid-19

And then came Covid-19 ☹
Just a few weeks after the PoC the news came that Covid-19 didn’t stay in China and was spreading around the world. Many countries announced (complete) lockdowns and everybody was obliged to stay home. Traditional office workers were in need of a remote workplace. Our customer asked if it was possible to move and upscale the PoC setup (sized for 10 users max) to a production environment suitable for approximately 100 users. We were able to use existing hardware and expanded the PoC to 30 users at first. By doing so we bought some time for our customer to order new hardware for the rest of the users.

Because of the urgency we offered two separate hardware options. Based on the delivery times they decided to order the hardware that could be delivered as soon as possible. But by then, they weren’t the only company that needed extra hardware rapidly. So in the meantime we provided them with rental hardware instead and we were able to give them the extra resources they needed.

The hardware was delivered in batches because of the poor availability and so in just a couple of days we expanded the PoC with an extra 30 users on their own hardware in a production environment with 100 users, at that time on rental hardware.

Production

During the current Covid-19 pandemic, many customers requested changes and extra resources within their current deployments. Our PoC is still in production, the upside of this is that we now know a lot of design changes which we could do better when creating a complete new design and implementation for this customer. Currently we are planning the design and implementation for a new solid and HA Citrix VAD environment.

Creating a Memory dump Citrix MCS/PVS and Hypervisor

When you need to debug some issues with your current deployment, you’re probably asked to create a memory dump. When you’re deployment is a traditional VM than that’s no issue, if you are using PVS than the memory dump is probably disabled because the PVS optimization disabled it or you used the Citrix optimizer which also disables the creation of memory dumps.

Just enabling the creation of a memory dump isn’t working, you need to specify a location for the memory dump to be created. Citrix created different articles on how to create a memory dump for PVS and MCS, CTX127871 and CTX261722

PVS

When using PVS you probably enabled “Cache on RAM with overflow on disk” and added a dedicated Write Cache Disk (WCD) for the vdiskdif.vhdx file. Because all the best practices tell you to change the location of the Page file and the Log files to a dedicated disk you probably use the WCD as alternative location.

When enabling Memory Dump creation you don’t want the space on the WCD got full because the DedicatedDumpFile is created on that disk. I recommend using a separate disk just for creating the Memory Dump, I make the size of that disk 1GB bigger than the assigned memory for that VM. Because I have situations were we have 40-60 VM’s, I don’t assign a Memory Dump disk to every device, because I do not have the storage available.

After the creation of the dedicated disk for the memory dump and you reboot your VM, you will notice that the vdiskdif.vhdx is located on the new disk. To make sure this doesn’t happen anymore you need to create a file with the following name and place it in the root of the new disk: {9E9023A4-7674-41be-8D71-B6C9158313EF}.VDESK.VOL.GUID See CTX218221.

Enabling the Memory Dump

I always set the location for the Memory Dump to the E: drive, for both PVS and MCS I create a dedicated disk as mentioned earlier.

To enable the creation of the memory dump just add the following registry keys as explained within the previous mentioned articles:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl

  • AutoReboot
  • CrashDumpEnabled
  • LogEvent
  • Overwrite
  • DumpFileSize
  • IgnorePagefileSize
  • DedicatedDumpFile
  • AlwaysKeepMemoryDump

I attached a TXT file that I always use to set it correctly, it’s at the end of this article. When you downloaded it, you need to rename it to .reg and you can merge the settings.

Creating a memory Dump

Because we mostly work with the Citrix Hypervisor (previous called XenServer), I have written down the steps to create a Memory Dump on this hypervisor. When I have to create a Memory Dump on VMware or Hyper-V I will write this down here to.

Ok, we now have a situation where you need to create a memory dump of the VM. After Identifying the VM, write down or copy the UUID.

Getting the UUID of the VM

Then run the following command within the console of the Hypervisor, where VM_UUID is the UUID you just written down or copied:

list_domains | grep -i <VM_UUID>

Then you get a Number which you have to write down as you can see in the following screenshot.

Receiving the Domain ID (## 239)

Then you run the following command, where Domain ID is the number you got from the previous step.

xen-hvmcrash [domain ID]

For this example that would be: xen-hvmcrash 239

Making the VM crash 🙂

Now the machine will crash and you can look at a BSOD within the console and after the machine rebooted you can find the Memory Dump on the E: drive. You can copy the memory dump and upload it or analyze it yourself.

When you have questions regarding these steps, please let me know.

Below the reg file, it’s a txt file within a zip file. You have to rename the txt to reg.

Remote DPI scaling for Windows

With the release of Windows 2016 Microsoft changed the way users could change their DPI size in a Remote Desktop Session, it inherited the setting from the end point. This would be the Laptop/Desktop or Thin Client the users are using to connect to the Remote desktop. When users needed to had a larger text, they need to change it on the local device, as this works for dedicated workplaces, it’s not working for the user that changes to different workplaces throughout the week.

As you can image, they connect and see they have the wrong size, have to sign out, change it on the local device and have to connect again to their remote session. That’s not very user friendly.

Citrix Workspace App

Citrix added an option to their Workspace App to change the DPI size in the current setting, after you change it you need to resize your window and everything is bigger or smaller depending on your choice. But also this is a setting a users need to change every time they connect from another device.

Setting the DPI size within a session
Setting the DPI size within a session using the Workspace App

The downside of this is that you also need the have the correct setting within your Workspace App locally. within the Advanced PreferencesHigh DPIScale the session for high resolution, you need to set it to YES. This is also a device settings, which means that you have to set it on every device you connect to or need to have a GPO that sets this.

Text Size

To change the DPI size to a different size you just need to change a registry key, which is a per user setting. Below are the registry values:

HKEY_CURRENT_USER\Control Panel\Desktop\
DWORD: LogPixels (create if it doesn’t exist)
Value: see below in Decimal:
96   –   Smaller 100%
120   –  Medium 125%
144  –   Larger 150%
192   –   Extra Large 200%

This is nice for Admins or users that are familiar with the registry, and know how to change a value without breaking their user settings.

Text Size App

Because most users don’t know how to change the registry or the Admins don’t want users to open Regedit. I created an App (basically a script which runs like an App) that lets users change their DPI size and sets the correct registry value. After setting this, the users gets asked to Sign out and sign-in again to get the changes to work.

I created the app in Dutch and English (based on the system local), but if requested I can add different languages to the App.

Changing the DPI size using the Text Size app

You can download the app for free and use it everywhere you want. Please let me know if you have any questions or just let me know that you are using the app.

Upgrading Nvidia firmware

During the last couple off days I needed to update the firmware from the Nvidia Tesla T4 card in our servers. When following the installation steps provided by HPE I ran into some issues, so I decided to create a step by step guide on how to update the firmware.

  1. Download the latest firmware from your vendor
  2. Upload the RPM file to /usr/local/bin using Winscp or your favorite tool
  3. Connect using SSH to the host
    1. Browse to cd /usr/local/bin
    2. unpack the RPM file using the following command: rpm –ivh ./Tesla_T4_90.04.96.00.01-1-0.x86_64.rpm
      The RPM file name can be different when upgrading a newer version or other Nvidia card.
    3. Go to the folder where the RPM file is extracted for now this is the Tesla_T4_90.04.96.00.01 folder: cd /usr/local/bin/Tesla_T4_90.04.96.00.01/
    4. Change the permissions of the file
      chmod +x Tesla_T4_90.04.96.00.01.scexe
    5. Make sure all nvidia kernel modules are removed
      init 3
      rmmod nvidia
    6. When you get the following error :
      ERROR: Module nvidia is in use
      run the following command:
      service xcp-rrdd-gpumon stop
      and then run:
      rmmod nvidia
    7. Now we can upgrade the firmware using the following command:
      ./Tesla_T4_90.04.96.00.01.scexe -f
      The SCEXE file name can be different when upgrading a newer version or other Nvidia card.
      Choose -i if you would control the upgrade for every card in the host.
  4. When all the cards are upgraded you need to reboot the host and continue to the next host.

Good luck with upgrading, as you can see it’s easy.