Renewing Expired xConnect Client Cert on Azure PaaS

Cover Image for Renewing Expired xConnect Client Cert on Azure PaaS

Legends have it that you're not a real Sitecore developer until you've dealt with an expired xConnect client certificate. If you're not a proactive and responsible developer, the story goes something like this:

Installing XP thinking you don't need to do anything else (colorized)

Put into words:

  1. Install Sitecore
  2. Wait one year
  3. Troubleshoot xConnect expired certificate
  4. Join the rest of the elite Sitecore developers in Valhalla

When the xConnect client cert expires on an Azure PaaS XP install (as opposed to local), there is more work involved because certs and configurations are distributed across multiple app services.

Prevention is Key

  • Be proactive when first provisioning an Azure PaaS XP environment to ensure that xConnect certificate expiration isn't going to be an issue
  • Consider one or more of these solutions immediately before or after installing:
    • Use a self signed cert that never expires (and allow invalid certificates)
    • Disable xConnect certs
    • Automate and operationalize the cert update process (recommended approach for production environments)

If you're reading this, it's already too late. Let's dive in.

The Errors

When your XC client cert has expired, you'll likely see these kinds of error messages (source):


_10
Experience Analytics]: Failed to synchronize segments. Message: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.. Details: at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
_10
at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.GetAll(NameValueCollection readingPreferences)
_10
at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.GetAll(NameValueCollection readingPreferences)
_10
at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.GetSegmentsToSynchronize()
_10
at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsManager.SynchronizeAllSegments()
_10
at System.Threading.Tasks.Task.Execute()
_10
--- End of stack trace from previous location where exception was thrown ---
_10
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
_10
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
_10
at Sitecore.ExperienceAnalytics.Client.Deployment.SyncSegmentsProcessor.<process>d__4.MoveNext()


_11
Exception System.InvalidOperationException: The certificate was not found. Store: My, Location: CurrentUser, FindType: FindByThumbprint, FindValue: 83DCC21BBF54D76F71D7B67EA2319273BCDA8E10, InvalidAllowed: True.
_11
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
_11
at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
_11
at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
_11
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
_11
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
_11
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
_11
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
_11
at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
_11
at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
_11
...


_21
Exception System.InvalidOperationException: Ensure definition type did not complete successfully. StatusCode: 401, ReasonPhrase: 'Invalid certificate', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
_21
{
_21
Pragma: no-cache
_21
Cache-Control: no-cache
_21
Date: Thu, 02 Jan 2020 15:03:36 GMT
_21
Server: Microsoft-IIS/10.0
_21
X-AspNet-Version: 4.0.30319
_21
X-Powered-By: ASP.NET
_21
Content-Length: 0
_21
Expires: -1
_21
}
_21
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
_21
at Sitecore.ExperienceAnalytics.Core.Repositories.ReferenceData.ReferenceDataSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
_21
at Sitecore.ExperienceAnalytics.Aggregation.Repositories.AggregationSegmentReader.Get(IEnumerable`1 keys, NameValueCollection readingPreferences)
_21
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetReportSegments()
_21
at Sitecore.ExperienceAnalytics.Api.RequestTypeResolver.GetRequestType()
_21
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.GetModelFromBindingContext(HttpActionContext actionContext, ModelBindingContext bindingContext)
_21
at Sitecore.ExperienceAnalytics.Api.Http.ModelBinding.ReportQueryModelBinder.BindModel(HttpActionContext actionContext, ModelBindingContext bindingContext)
_21
at System.Web.Http.ModelBinding.ModelBinderParameterBinding.ExecuteBindingAsync(ModelMetadataProvider metadataProvider, HttpActionContext actionContext, CancellationToken cancellationToken)
_21
at System.Web.Http.Controllers.HttpActionBinding.<executebindingasynccore>d__12.MoveNext()
_21
...

Taking Stock of App Services

Your Azure PaaS XP App Services will look similar to this:

  • cd (SitecoreCD)
  • cm (SitecoreCM)
  • cortex-processing (SitecoreCortexProcessingEngine)
  • cortex-reporting (SitecoreCortexReporting)
  • ma-ops (MarketingAutomationOperations)
  • ma-rep (MarketingAutomationReporting)
  • prc (SitecoreProcessing)
  • rep (SitecoreReporting)
  • si (SitecoreIdentity)
  • xc-collect (XConnectCollection)
  • xc-refdata (XConnectReferenceData)
  • xc-search (XConnectCollectionSearch)

We're going to need to dive into most of these, so let's get some tools in your toolbelt.

Useful Tools

Bookmarks

Yes, bookmarks. Bookmark the URLs in the next session for each App Service so you can move quickly.

App Service Editor

The file system of any app service can be accessed via Azure Portal App Service Editor:

https://name-of-service.scm.azurewebsites.net/dev/wwwroot

App Service Editor

Be careful in the App Service Editor. Changes to files are saved right away which could cause your App Service to restart.

Kudu Console

App Service file systems can also be accessed via the Kudu interface which also offers other useful functionality such as PowerShell and file downloads:

https://name-of-service.scm.azurewebsites.net

Kudu Console

As you work through the cert issues, it helps to have a way a quick look at the logs. navigate to:

https://name-of-service.scm.azurewebsites.net/DebugConsole/?shell=powershell

Run the following commands:


_3
cd C:\home\site\wwwroot\App_Data\logs
_3
_3
gci . | select -last 1 | Get-Content -last 100

This is especially useful for the CM, CD, and PRC App Services.

Note on Self Signed Certs

According to this article, self signed certificates are fine to use for xConnect, even in production.

Self signed certs can be created with an expiry date far into the future, which saves the scheduling, work, risk, and downtime involved in needing to update the environments every x months.

Despite documentation and other blogs saying that self signed certs will not work in Azure environments, in my experience you CAN use a self signed cert for the xConnect client, as long as you configure your environment properly.

Generate a New Self-Signed Certificate

On your local windows machine, run the following command in PowerShell (run as admin):


_1
New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName "MySelfSignedCert" -FriendlyName "MySelfSignedCert" -NotAfter $([datetime]::now.AddYears(20))

Don’t worry about the DnsName — you can assign the cert to any host name

Let's find the cert on the Local Computer using mmc. See this guide for more info.

Open the Windows menu and type mmc

Go to Certificates --> Local Computer / Personal / certificates

Export the Certificate - Include the Private Key. Specify a password. Make note of the password. Save it as a .pfx file.

Add the Certificate to Azure

We want to get the self-signed cert added to the certificate store. This will make the cert available to all App Services.

SSL/TLS Settings

For each App Service, navigate to the Configuration blade and ensure that the WEBSITE_LOAD_CERTIFICATES is set to * or to the thumbprint of the new certificate. This will make the certificate available to the App Services.

Load the Cert

Make note of the old thumbprint as it will need to be replaced with the new thumbprint in numerous locations.

In the Kudu PS console of each app service, run this command in Kudu PS console to verify that the cert has been made available.


_1
Get-ChildItem -path cert:\CurrentUser\My

Update Thumbprints and Allow Self Signed Certs

Go through each App Service. For all files below, replace any instances of the old thumbprint with the new thumbprint.

Any time you make a change, ensure that your deploy pipelines / variables / files are updated in order to prevent the updates being wiped on subsequent deploys. This is ususally the case for, but not limited to, CM and CD. However, any App Services that's being changed manually should probably have a deploy pipeline associated with it so this process can be more easily automated.

Wrapping Up

Now that you have made the necessary changes, do the following:

  • Restart all App Services
  • Deploy marketing definitions
  • Populate solr managed schema (if applicable)
  • Reindex

Verification

After doing all of the above, you can verify the fix:

  • CM launchpad / dashboard should load with no console errors and the graph should load on the bottom left (may take a dat for data to populate after fixing)
  • Should be able to view all analytics dashboard areas with no errors (Experience Profile, Experience Analytics)
  • Ensure there are no related errors present in the logs, particularly for, but not limited to, these App Services:
    • cm
    • cd
    • prc
    • rep

Troubleshooting Questions

  • Do any of the App Services have relevant errors in their logs?
  • Was the "Deploy marketing definitions" run in the Sitecore dashboard?
  • Has the thumbprint been updated everywhere it needs to be?
  • Has the cert been loaded in App Service Configuration everywhere it needs to be?
  • Has AllowInvalidCerts been activated everywhere it needs to be? If you see that setting set to false in the error log, you missed a spot.
  • Have all of the App Services been restarted?

Resources

Keep BUIDLing,

Marcel


More Stories

Cover Image for Hello World

Hello World

> Welcome to the show

Cover Image for Super Fast Project Builds with Visual Studio Publish

Super Fast Project Builds with Visual Studio Publish

> For when solution builds take too long

Cover Image for Tips for Applying Cumulative Sitecore XM/XP Patches and Hotfixes

Tips for Applying Cumulative Sitecore XM/XP Patches and Hotfixes

> It's probably time to overhaul your processes

Cover Image for JSS: Reducing Bloat in Multilist Field Serialization

JSS: Reducing Bloat in Multilist Field Serialization

> Because: performance, security, and error-avoidance

Cover Image for Script: Boost SIF Certificate Expiry Days

Script: Boost SIF Certificate Expiry Days

> One simple script that definitely won't delete your system32 folder

Cover Image for NextJS/JSS Edit Frames Before JSS v21.1.0

NextJS/JSS Edit Frames Before JSS v21.1.0

> It is possible. We have the technology.

Cover Image for NextJS: Unable to Verify the First Certificate

NextJS: Unable to Verify the First Certificate

> UNABLE_TO_VERIFY_LEAF_SIGNATURE

Cover Image for JSS + TypeScript Sitecore Project Tips

JSS + TypeScript Sitecore Project Tips

> New tech, new challenges

Cover Image for Content Editor Search Bar Not Working

Content Editor Search Bar Not Working

> Sometimes it works, sometimes not

Cover Image for Tips for Forms Implementations

Tips for Forms Implementations

> And other pro tips

Cover Image for Symposium 2022 Reflections

Symposium 2022 Reflections

> Sitecore is making big changes

Cover Image for Ideas For Docker up.ps1 Scripts

Ideas For Docker up.ps1 Scripts

> Because Docker can be brittle

Cover Image for On Sitecore Stack Exchange (SSE)

On Sitecore Stack Exchange (SSE)

> What I've learned, what I see, what I want to see

Cover Image for Don't Ignore the HttpRequestValidationException

Don't Ignore the HttpRequestValidationException

> Doing so could be... potentially dangerous

Cover Image for NextJS: Access has been blocked by CORS policy

NextJS: Access has been blocked by CORS policy

> CORS is almost as much of a nuisance as GDPR popups

Cover Image for Add TypeScript Type Checks to RouteData fields

Add TypeScript Type Checks to RouteData fields

> Inspired by error: Conversion of type may be a mistake because neither type sufficiently overlaps with the other.

Cover Image for SPE Script Performance & Troubleshooting

SPE Script Performance & Troubleshooting

> Script never ends or runs too slow? Get in here.

Cover Image for Sitecore Symposium 2022

Sitecore Symposium 2022

> What I'm Watching 👀

Cover Image for On Mentorship and Community Contributions

On Mentorship and Community Contributions

> Reflections and what I learned as an MVP mentor

Cover Image for How to Run Old Versions of Solr in a Docker Container

How to Run Old Versions of Solr in a Docker Container

> Please don't make me install another version of Solr on my local...

Cover Image for Early Returns in React Components

Early Returns in React Components

> When and how should you return early in a React component?

Cover Image for Azure PaaS Cache Optimization

Azure PaaS Cache Optimization

> App Services benefit greatly from proper configuration

Cover Image for Year in Review: 2022

Year in Review: 2022

> Full steam ahead

Cover Image for Tips for New Sitecore Developers

Tips for New Sitecore Developers

> Because learning Sitecore can be hard

Cover Image for NextJS: Short URL for Viewing Layout Service Response

NextJS: Short URL for Viewing Layout Service Response

> Because the default URL is 2long4me