Automating FAS Authorization Certificate renewal

As I deploy more and more Citrix FAS Servers for customers who intend to utilize Entra ID (formerly known as Azure AD) as their Identity Provider (IdP), I have observed that the FAS Authorization Certificate requires periodic renewal. To ensure that expired certificates do not inconvenience my customers and to maintain a level of automation and security, I have developed a simple script that notifies them when a new FAS Authorization Certificate is needed and mails them to issue it through their Active Directory Certificate Services (ADCS).

The current FAS Renewal script is designed to run locally on the FAS Servers. In the future, I plan to enhance it to allow for remote execution from a management server or similar setup. However, for the current version, you must follow these steps on all your FAS servers individually.

Automation and Security Considerations

In today’s technology landscape, automation is fundamental to organizational efficiency. However, maintaining a degree of control over critical components like your Certificate Authority is equally important. To strike a balance, I have designed this script to handle the certificate renewal process up to a certain point and then notify you via email, granting you the authority to issue the certificate from your ADCS.

Script Functionality

When I set out to create this script, my primary objective was to automate the entire certificate renewal process. The script follows a logical sequence of actions:

  1. Checking the Expiry Date: The script evaluates the expiration date of the current FAS Authorization Certificate.
  2. Requesting a New Certificate: If the current date approaches the expiration date, the script initiates a request for a new certificate.
  3. Connecting to ADCS: The script establishes a connection to your Active Directory Certificate Services (ADCS), ensuring that the certificate issuance process is orchestrated securely.
  4. Updating FAS Configuration: Once the new certificate is obtained, the script updates the Citrix FAS configuration to utilize the freshly issued certificate seamlessly.

Enhanced Security

During my research on certificate issuance using scripts, I came across discussions highlighting the importance of maintaining control over such processes. I agree with this perspective. The FAS Authorization Certificate is a Certificate Request Agent, which implies that it can potentially be used to request certificates for all users within your environment. In the event of compromise, this could pose a significant security risk, allowing unauthorized access to your organization’s resources. To mitigate this risk, I made a deliberate choice to implement a user-based approach. This ensures that the issuance of certificates is controlled and limited to authorized personnel, reducing the potential for misuse and bolstering the security of your organization’s infrastructure.

Prepare Mail User

In order for the script to successfully send emails to a predefined contact, it is crucial to ensure that the sent emails are delivered without being flagged as spam. To achieve this, authentication against the mail server is necessary. When sending unauthenticated emails through Office 365, there is a high likelihood that such emails will be routed to the recipient’s spam folder, resulting in them going unnoticed. While I have utilized my personal Office 365 account for this demonstration, it is possible to configure the script to work with an on-premises Exchange server as well. However, there are certain prerequisites that must be met to ensure successful user authentication for email transmission.

Account Setup

To establish a connection to a user’s mailbox, it is essential that the user account be enabled. Typically, shared mailboxes are configured as disabled users, but an enabled user account is required for this script to function effectively. Given that most organizations have password expiration policies, it is essential to use an Application Password to prevent script failures due to password changes. To create an Application Password, please refer to the provided link and ensure that you assign a meaningful name to it for identification purposes; in this example, “FASRenewalMail” was used.

App Password

Additionally, it is crucial to verify whether the user is permitted to connect to “Authenticated SMTP” by following the instructions provided in the linked resource.

Mail Configuration and Testing

Once the user meets all the specified prerequisites, we can proceed with configuring the mail functionality on the FAS servers. To ensure the best security and prevent passwords from being stored directly within the script, especially since scheduled tasks can run under different user accounts, we adopt a secure approach by creating an encrypted text file. This encrypted text file is the most secure method for password storage as it can only be accessed by the user who created it and exclusively on the machine where it was generated. Copying the file to another location or machine does not grant access to its contents.

To create the encrypted text file, please utilize the provided code snippet:

PowerShell
Read-Host -Prompt "Enter your tenant password" -AsSecureString | ConvertFrom-SecureString | Out-File "C:\Scripts\cred.txt"

File Location and User Considerations:

The encrypted text file has been saved to the directory C:\Scripts. If you prefer a different location, you have the flexibility to make that adjustment. However, it’s important to note that if the scheduled task is created under a different user account, you must initiate a PowerShell session under that specific user. Failure to do so will result in the script’s inability to retrieve the password, leading to script execution failure.

Testing the mail:

PowerShell
$TestUser = "[email protected]"
$Pass = Get-Content "C:\Scripts\cred.txt" | ConvertTo-SecureString
$cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $AdminName, $Pass

Send-MailMessageFrom $TestUser –To $TestUser –Subject "Test Email" –Body "Test SMTP Relay Service" -SmtpServer smtp.office365.com -Credential $cred -UseSsl -Port 587 

Once the encrypted text file has been successfully created, you can verify its functionality by conducting a test email transmission. Use the provided code snippet below for sending a test email, and remember to replace the $TestUser variable with the designated user you are using for this test:

If this testing phase proceeds without issues, you can proceed with the FAS Renewal Script implementation.

FAS Renewal Script:

The script relies on several configurable variables that must be set before execution to ensure its proper functioning. One critical variable to adjust is “ThresholdDays,” which should be tailored to your specific requirements. I recommend setting it to 30-60 days to ensure renewal well before the certificate expiration date. During testing, I used a value of 730 days, as newly created certificates are valid for 329 days. This extended period allows for multiple tests until you are satisfied.

Additionally, ensure that the “CredentialFile” variable points to the same location where you saved the encrypted text file in the previous step.

PowerShell
# Configuration
$ThresholdDays = 730
$AdminEmail = "[email protected]"
$ToEmail = "[email protected]"
$CredentialFile = "C:\Scripts\cred.txt"
$SmtpServer = "smtp.office365.com"
$SmtpPort = 587 
PowerShell
# Load Citrix FAS snap-in
Add-PSSnapin Citrix.A*

# Configuration
$ThresholdDays = 730
$SleepTime = 10
$AdminEmail = "[email protected]"
$ToEmail = "[email protected]"
$CredentialFile = "C:\Scripts\cred.txt"
$SmtpServer = "smtp.office365.com"
$SmtpPort = 587

# Get current certificate information
$CitrixFasAddress=(Get-FasServer)[0].Address
$DefaultCA=(Get-FasMsCertificateAuthority -Default).Address
$CurrentCertificate = Get-FasAuthorizationCertificate -FullCertInfo
$DaysDifference = ($CurrentCertificate.ExpiryDate - (Get-Date)).Days

# Load credentials from file
$SecurePassword = Get-Content $CredentialFile | ConvertTo-SecureString
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $AdminEmail, $SecurePassword


# Function to send email notifications
function Send-EmailNotification {
    param (
        [string]$Subject,
        [string]$Body
    )

    Send-MailMessage -From $AdminEmail -To $ToEmail -Subject $Subject -Body $Body -SmtpServer $SmtpServer -Credential $Credential -UseSsl -Port $SmtpPort
}

if ($DaysDifference -lt $ThresholdDays) {
    # Request a new certificate
    $NewCertificate = New-FasAuthorizationCertificate -CertificateAuthority $DefaultCA -CertificateTemplate "Citrix_RegistrationAuthority" -AuthorizationTemplate "Citrix_RegistrationAuthority_ManualAuthorization"
    
    # Send email notification about pending certificate
    Send-EmailNotification -Subject "Pending FAS Authorization Certificates | $CitrixFasAddress" -Body "There is a pending FAS authorization certificate. Please issue the certificate on server $($CurrentCertificate.Address) so that FAS keeps working. After issuing, everything is automated and you receive a mail if everything was succesful."
    
    # Wait for certificate to be issued
    do {
        # Get all FAS authorization certificates
        $Certificates = Get-FasAuthorizationCertificate
        
        # Check if there are certificates with a status other than "Ok"
        $CertificatesNotOk = $Certificates | Where-Object { $_.Status -ne "Ok" }
        
        if ($CertificatesNotOk.Count -gt 0) {
            # Wait for a 1 hour before checking again
            Start-Sleep -Seconds $SleepTime
            Write-Output "Waiting for certificate to be issued"
        }
    } while ($CertificatesNotOk.Count -gt 0)

    # All certificates have status "Ok"
    Write-Output "All Certificates are Issued."

    # Remove the old Certificate and update the Certificate Definition
    Write-Output "Removing the old Certificate"
    Remove-FasAuthorizationCertificate -Id $CurrentCertificate.Id
    $CertificateDefinition = (Get-FasCertificateDefinition)[0].Name
    Write-Output "Updating the Certificate Definition with the new Certificate"
    Set-FasCertificateDefinition -Name $CertificateDefinition -AuthorizationCertificate $NewCertificate.Id
    
    # Send email notification about certificate renewal
    $CurrentCertificateNew = Get-FasAuthorizationCertificate -FullCertInfo
    $DaysDifferenceNew = ($CurrentCertificateNew.ExpiryDate - (Get-Date)).Days
    Send-EmailNotification -Subject "FAS Authorization Certificates Renewed | $CitrixFasAddress" -Body "The FAS authorization certificate is renewed. It's now valid for $($DaysDifferenceNew) days until $($CurrentCertificateNew.ExpiryDate)."
} else {
    Write-Output "Certificate expiration date is $($CurrentCertificate.ExpiryDate)"
}

After configuring these variables, save the script to the C:\Scripts directory.

Create a scheduled task.

Configure the scheduled task with a trigger that aligns with your customer’s preferences. I suggest selecting a workday for scheduling and avoiding Mondays, as mailboxes may be full, potentially causing the FAS Renewal Mail to go unnoticed. Configure the task to execute as follows:

Program: powershell.exe
Arguments: -ExecutionPolicy Bypass -File C:\Scripts\FASRenewalScript.ps1

FASRenewalScript Scheduled Task, Powershell

Use the same account that was used for creating the encrypted text file; otherwise, the script will not function as expected.

Conclusion

With these steps completed, you are now equipped to receive timely notifications when an FAS Authorization Certificate is approaching its expiration date. If you have any comments or questions, please feel free to reach out for further assistance.

Troubleshooting Citrix FAS Event ID 124 and 123

After a recent deployment of a new ADCS (Active Directory Certificate Services) and FAS (Federated Authentication Service) servers, it became evident that users were not receiving certificates. Interestingly, the existing production FAS servers, which were running older versions, continued to successfully request certificates.

Upon reviewing the FAS documentation, it was revealed that certificates could be requested through the use of PowerShell. (https://docs.citrix.com/en-us/federated-authentication-service/current-release/config-manage/ca-configuration#fas-server) Using the below code, I could simulate the request.

PowerShell
Add-PSSnapin Citrix.A*
$rule = "default" # rule/role in your FAS admin console
$users = sjoerd@lab.newyard.online # test users
foreach ( $user in $users )
{
    $server = Get-FasServerForUser -UserPrincipalNames $user
    if( $server.Server -ne $NULL) {
        New-FasUserCertificate -Address $server.Server -UserPrincipalName $user -CertificateDefinition $rule"_Definition" -Rule $rule
    }
    if( $server.Failover -ne $NULL) {
        New-FasUserCertificate -Address $server.Failover -UserPrincipalName $user -CertificateDefinition $rule"_Definition" -Rule $rule
    }
}

I observed Event IDs 124 and 123 in the FAS server event log.

Event ID 124:
[S124] Failed to issue a certificate for [upn: [email protected] role: default] at [certifcate authority: <server>.lab.newyard.online\lab-<server>-CA] [exception: MicrosoftCertificateAuthority::SubmitCertificateRequest – the CA returned CR_DISP_DENIED (code 2)]. [correlation: f901946c-1d9b-4351-9575-2f75eb24094c]

[S124] Failed to issue a certificate for UPN. Citrix FAS Failed request for a Certificate.

Event ID 123:
[S123] Failed to issue a certificate for [upn: [email protected] role: default] [exception: The CSR failed at all configured certificate authorities] [correlation: f901946c-1d9b-4351-9575-2f75eb24094c]

[S123] Failed to issue a certificate for UPN. Citrix FAS Failed request for a Certificate.

I found no documented explanations for these error codes after consulting Citrix’s troubleshooting guide. Subsequently, I researched further and found an article indicating that these errors were related to permission issues. However, the Citrix documentation did not provide explicit guidance on the required permissions.

To address the issue, I proceeded to remove the Citrix templates from the Certificate Authority (CA) and redeployed these templates. Regrettably, this action did not resolve the problem. Lacking access to another client environment for reference on certificate permissions, I established a CA with FAS functionality within my laboratory environment.

Upon conducting a comprehensive analysis of permissions associated with all Citrix-related templates, I determined that the templates required the following permissions to operate effectively:

  1. Authenticated Users: Read
  2. Domain Admins: Read, Write, Enroll
  3. Domain Computers: Read, Enroll

This is for the following Templates:

  1. Citrix_RegistrationAuthority
  2. Citrix_RegistrationAuthority_ManualAuthorization
  3. Citrix_SmartcardLogon

After changing the permissions on the customer environment to the permissions of the Lab, all started to work.