Renewing Expired xConnect Client Cert on Azure PaaS

Index

Legends have it that you're not a real Sitecore developer until you've dealt with an expired xConnect client certificate. If you're not a proactive and responsible developer, the story goes something like this:

Put into words:

1. Install Sitecore
2. Wait one year
3. Troubleshoot xConnect expired certificate
4. Join the rest of the elite Sitecore developers in Valhalla

When the xConnect client cert expires on an Azure PaaS XP install (as opposed to local), there is more work involved because certs and configurations are distributed across multiple app services.

• Be proactive when first provisioning an Azure PaaS XP environment to ensure that xConnect certificate expiration isn't going to be an issue
• Consider one or more of these solutions immediately before or after installing:
  • Use a self signed cert that never expires (and allow invalid certificates)
  • Disable xConnect certs
  • Automate and operationalize the cert update process (recommended approach for production environments)

If you're reading this, it's already too late. Let's dive in.

When your XC client cert has expired, you'll likely see these kinds of error messages (source):

Your Azure PaaS XP App Services will look similar to this:

• cd (SitecoreCD)
• cm (SitecoreCM)
• cortex-processing (SitecoreCortexProcessingEngine)
• cortex-reporting (SitecoreCortexReporting)
• ma-ops (MarketingAutomationOperations)
• ma-rep (MarketingAutomationReporting)
• prc (SitecoreProcessing)
• rep (SitecoreReporting)
• si (SitecoreIdentity)
• xc-collect (XConnectCollection)
• xc-refdata (XConnectReferenceData)
• xc-search (XConnectCollectionSearch)

We're going to need to dive into most of these, so let's get some tools in your toolbelt.

Yes, bookmarks. Bookmark the URLs in the next session for each App Service so you can move quickly.

The file system of any app service can be accessed via Azure Portal App Service Editor:

https://name-of-service.scm.azurewebsites.net/dev/wwwroot

Be careful in the App Service Editor. Changes to files are saved right away which could cause your App Service to restart.

App Service file systems can also be accessed via the Kudu interface which also offers other useful functionality such as PowerShell and file downloads:

https://name-of-service.scm.azurewebsites.net

As you work through the cert issues, it helps to have a way a quick look at the logs. navigate to:

https://name-of-service.scm.azurewebsites.net/DebugConsole/?shell=powershell

Run the following commands:

This is especially useful for the CM, CD, and PRC App Services.

According to this article, self signed certificates are fine to use for xConnect, even in production.

Self signed certs can be created with an expiry date far into the future, which saves the scheduling, work, risk, and downtime involved in needing to update the environments every x months.

Despite documentation and other blogs saying that self signed certs will not work in Azure environments, in my experience you CAN use a self signed cert for the xConnect client, as long as you configure your environment properly.

On your local windows machine, run the following command in PowerShell (run as admin):

Don’t worry about the DnsName — you can assign the cert to any host name

Let's find the cert on the Local Computer using mmc. See this guide for more info.

Open the Windows menu and type mmc

Go to Certificates --> Local Computer / Personal / certificates

Export the Certificate - Include the Private Key. Specify a password. Make note of the password. Save it as a .pfx file.

We want to get the self-signed cert added to the certificate store. This will make the cert available to all App Services.

For each App Service, navigate to the Configuration blade and ensure that the WEBSITE_LOAD_CERTIFICATES is set to * or to the thumbprint of the new certificate. This will make the certificate available to the App Services.

Make note of the old thumbprint as it will need to be replaced with the new thumbprint in numerous locations.

In the Kudu PS console of each app service, run this command in Kudu PS console to verify that the cert has been made available.

Go through each App Service. For all files below, replace any instances of the old thumbprint with the new thumbprint.

Any time you make a change, ensure that your deploy pipelines / variables / files are updated in order to prevent the updates being wiped on subsequent deploys. This is ususally the case for, but not limited to, CM and CD. However, any App Services that's being changed manually should probably have a deploy pipeline associated with it so this process can be more easily automated.

• dev-cd

  • If the AllowInvalidClientCertificates setting is present in the main Web.config, update it to True and update the DevOps variable if necessary.
  • https://dev-cd.scm.azurewebsites.net/dev/wwwroot/App_Config/ConnectionStrings.config
    • Update all instances of the old thumbprint to the new thumbprint
    • Append ;AllowInvalidClientCertificates=true to the end of all connection strings containing the thumbprints (if not already present)
    • Ensure that the Solr endpoint has the correct scheme (HTTP vs HTTPS). Currently, the scheme should be set to HTTP. Update the DevOps variable if necessary.

• dev-cm

  • https://dev-cm.scm.azurewebsites.net/dev/wwwroot/App_Config/ConnectionStrings.config
    • Update all instances of the old thumbprint to the new thumbprint
    • Append ;AllowInvalidClientCertificates=true to the end of all connection strings containing the thumbprints (if not already present)
    • Ensure that the Solr endpoint has the correct scheme (HTTP vs HTTPS). Currently, the scheme should be set to HTTP. Update the DevOps variable if necessary.

• dev-cortex-processing

  • https://dev-cortex-processing.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />
  • https://dev-cortex-processing.scm.azurewebsites.net/dev/wwwroot/App_Data/jobs/continuous/ProcessingEngine/App_Config/ConnectionStrings.config
    • Append this to connection strings wherever thumbprint is present ;AllowInvalidClientCertificates=true

• dev-cortex-reporting

  • https://dev-cortex-reporting.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />

• dev-ma-ops

  • https://dev-ma-ops.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Set AllowInvalidClientCertificates to True
  • https://dev-ma-ops.scm.azurewebsites.net/dev/wwwroot/App_Config/ConnectionStrings.config
    • Append this to connection strings wherever thumbprint is present ;AllowInvalidClientCertificates=true

• dev-ma-rep

  • https://dev-ma-rep.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />

• dev-prc

  • If the AllowInvalidClientCertificates setting is present in the main Web.config, update it to True and update the DevOps variable if necessary.
  • https://dev-prc.scm.azurewebsites.net/dev/wwwroot/App_Config/ConnectionStrings.config
    • Append this to connection strings wherever thumbprint is present ;AllowInvalidClientCertificates=true
    • Ensure that Solr is pointing to the correct endpoint. Previously it was set to <add name="solr.search" connectionString="https://localhost:8994/solr" /> which was almost certainly incorrect.

• dev-rep

  • If the AllowInvalidClientCertificates setting is present in the main Web.config, update it to True and update the DevOps variable if necessary.
  • https://dev-rep.scm.azurewebsites.net/dev/wwwroot/App_Config/ConnectionStrings.config
    • No changes should be necessary here.

• dev-si

  • https://dev-si.scm.azurewebsites.net/dev/wwwroot/Config/production/Sitecore.IdentityServer.Host.xml
    • Update the thumbprint

• dev-xc-collect

  • https://dev-xc-collect.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />

• dev-xc-refdata

  • https://dev-xc-refdata.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />

• dev-xc-search

  • https://dev-xc-search.scm.azurewebsites.net/dev/wwwroot/App_Config/AppSettings.config
    • Update appSetting value if exists <add key="AllowInvalidClientCertificates" value=“True” />

Now that you have made the necessary changes, do the following:

• Restart all App Services
• Deploy marketing definitions
• Populate solr managed schema (if applicable)
• Reindex

After doing all of the above, you can verify the fix:

• CM launchpad / dashboard should load with no console errors and the graph should load on the bottom left (may take a dat for data to populate after fixing)
• Should be able to view all analytics dashboard areas with no errors (Experience Profile, Experience Analytics)
• Ensure there are no related errors present in the logs, particularly for, but not limited to, these App Services:
  • cm
  • cd
  • prc
  • rep

• Do any of the App Services have relevant errors in their logs?
• Was the "Deploy marketing definitions" run in the Sitecore dashboard?
• Has the thumbprint been updated everywhere it needs to be?
• Has the cert been loaded in App Service Configuration everywhere it needs to be?
• Has AllowInvalidCerts been activated everywhere it needs to be? If you see that setting set to false in the error log, you missed a spot.
• Have all of the App Services been restarted?

• https://www.bramstoop.com/posts/2021/02/2021-02-17-replace-xconnect-certificate-sitecore10/
• https://sitecorecompetencies.com/2019/03/27/incompatibilities-of-xconnect-and-client-certificates/
• https://doc.sitecore.com/xp/en/developers/93/platform-administration-and-architecture/replace-a-sitecore-client-certificate-in-azure.html
• https://sitecore.stackexchange.com/questions/14569/azure-paas-reports-certificate-not-found
• https://learnsitecorebasics.wordpress.com/2020/02/06/cannot-start-analytics-tracker-exception-the-certificate-was-not-found-sitecore-azure-webapps/
• https://doc.sitecore.com/xp/en/developers/93/sitecore-experience-manager/walkthrough--deploying-a-new-sitecore-environment-to-the-microsoft-azure-app-service.html
• https://doc.sitecore.com/xp/en/developers/92/platform-administration-and-architecture/replace-a-sitecore-client-certificate-in-azure.html
• https://madhuanbalagan.com/xconnect-certificate-was-not-found
• https://learnsitecorebasics.wordpress.com/tag/xconnect-collection-certificate/
• https://sitecorestudmuffin.wordpress.com/2019/12/13/updating-xconnect-certificates-in-azure-paas/
• https://docs.microsoft.com/en-us/azure/app-service/configure-ssl-certificate?tabs=apex%2Cportal
• https://kamsar.net/index.php/2017/10/All-about-xConnect-Security/
• https://www.bramstoop.com/posts/2018/06/2018-06-13-replacing-the-sitecore-client-certificate-on-azure-in-5-steps/
• https://getfishtank.ca/blog/sitecore-9-xconnect-client-and-ssl-certificates-explained

Keep BUIDLing,

Marcel