Last week, I got an email from my monitoring system that a certificate on a Server 2012 R2 Essentials machine was about to expire. I tracked it down to the computer’s local certificate, issued by the local Certification Authority (CA) almost 5 years ago. Since it hadn’t expired yet, it was a fairly simple matter to go into IIS, open Server Certificates, choose the expiring certificate, and click Renew.
Essentials Services Not Running
This week, I noticed that none of the Windows Server Essentials services were running. I didn’t even try to open the dashboard; I knew that wouldn’t work. Most of the services depend on the Windows Server Essentials Provider Registry Service. This article pointed me to the log patch, where I examined
C:\ProgramData\Microsoft\Windows Server\Logs\ProviderRegistryService.log
From the most recent error, the key lines are:
[1688] 201124.103151.9343: WssgCertMgmt: Found 0 matching certs without verification:
[1688] 201124.103151.9499: WssgCertMgmt: Collection Empty
[1688] 201124.103151.9499: IDENTITY: Local machine cert not found, trying to import the root cert backup to fix
I eventually found this 2012 thread, where Robert Pearman mentioned the registry key HKLM>SOFTWARE>Microsoft>Wwindows Server>IDENTITY. Sure enough, the LocalMachineCert value contained the thumbnail of the old certificate:
After manually replacing the LocalMachineCert value with the thumbprint of the new certificate (copied from the Personal store of Local Computer Certificates), I was once again able to start the Windows Server Essentials services and open the dashboard.
Remote Site Reporting .NET Warnings
While investigating the above issue, I also noticed multiple .NET warnings in the Application event log that started after the certificate renewal:
Log Name: ApplicationSource: ASP.NET 4.0.30319.0Date: 11/24/2020 11:08:48 AMEvent ID: 1310Task Category: Web EventLevel: WarningKeywords: ClassicUser: N/AComputer: MYSERVER.MYDOMAIN.localDescription:Event code: 3008Event message: A configuration error has occurred.Event time: 11/24/2020 11:08:48 AMEvent time (UTC): 11/24/2020 7:08:48 PMEvent ID: 748e9776d144445884b50993eba47579Event sequence: 2Event occurrence: 1Event detail code: 0Application information:Application domain: /LM/W3SVC/1/ROOT/Remote-4-132507185287384759Trust level: FullApplication Virtual Path: /RemoteApplication Path: C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\Machine name: MYSERVERProcess information:Process ID: 4036Process name: w3wp.exeAccount name: NT AUTHORITY\NETWORK SERVICEException information:Exception type: HttpExceptionException message: Exception has been thrown by the target of an invocation. (C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\web.config line 76)at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)Exception has been thrown by the target of an invocation. (C:\Program Files\Windows Server\Bin\WebApps\RemoteAccess\web.config line 76)at System.Web.Configuration.ProvidersHelper.InstantiateProvider(ProviderSettings providerSettings, Type providerType)at System.Web.Configuration.ProvidersHelper.InstantiateProviders(ProviderSettingsCollection configProviders, ProviderCollection providers, Type providerType)at System.Web.Security.Membership.InitializeSettings(Boolean initializeGeneralSettings, RuntimeConfig appConfig, MembershipSection settings)at System.Web.Security.Membership.Initialize()at System.Web.Security.Membership.get_Provider()at Microsoft.WindowsServerSolutions.Web.Security.AuthenticationHelper.add_PreLogOn(EventHandler value)at Microsoft.WindowsServerSolutions.Web.RemoteAccessSite.Global.Application_Start(Object sender, EventArgs e)Exception has been thrown by the target of an invocation.at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)at System.Activator.CreateInstance(Type type, Boolean nonPublic)at System.Activator.CreateInstance(Type type)at System.Web.HttpRuntime.CreatePublicInstanceByWebObjectActivator(Type type)at System.Web.Configuration.ProvidersHelper.InstantiateProvider(ProviderSettings providerSettings, Type providerType)Machine certificate is not found.Server stack trace:at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProductConfiguratorBase._CreateProxyEndpoint(Uri address, Type contractType, ProviderEndpointBehaviorAttribute endpointBehavior)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.ProviderFrameworkConfigurator.GetDuplexChannelFactory[T](ICollection`1 behaviors, ProviderInfo info, Object callback, NetworkCredential credential)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._CreateChannel(String targetComputer, IRegistryCallback callback, ProviderEndpointBehaviorAttribute endpointBehavior)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._GetFactory(String targetComputer, IRegistryCallback callback, ProviderEndpointBehaviorAttribute endpointBehavior)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1._CreateRealProxy()at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.AutoReconnecter`1._BeginConnect()at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ProviderRegistryConnectionMgmt`1.ConnectWhenAvailable(Int32 spinWaitMillis, Action operation, IRegistryCallback informOfProviderUpdateCallback)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.Internal.ProviderRegistryFacade.InitProviderRegistryProxy()at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory._InitProxy()at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.get__Proxy()at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.SetupConnector[T](ProviderConnector`1 providerConnector)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.GetConnector[T](String identifier, Object callback)at Microsoft.WindowsServerSolutions.Common.ProviderFramework.ConnectorFactory.GetServerConnector[T](String identifier, Object callback, NetworkCredential credential)at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.<>c__DisplayClass6.<CreateConnector>b__2()at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)Exception rethrown at [0]:at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)at System.Action.EndInvoke(IAsyncResult result)at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.CreateConnector(TimeSpan timeSpan)at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.Microsoft.WindowsServerSolutions.Users.Internal.IUserBackEnd.ConnectAsync(TimeSpan timeSpan)at Microsoft.WindowsServerSolutions.Users.Internal.UserBackEnd.Microsoft.WindowsServerSolutions.Users.Internal.IUserBackEnd.Connect(TimeSpan timeSpan)at Microsoft.WindowsServerSolutions.Users.UserMgmtManager.Connect()at Microsoft.WindowsServerSolutions.Web.Security.HSBSMembershipProvider..ctor()Request information:Request URL: https://mydomain.remotewebaccess.com:443/remoteRequest path: /remoteUser host address: 192.168.1.2User:Is authenticated: FalseAuthentication Type:Thread account name: NT AUTHORITY\NETWORK SERVICEThread information:Thread ID: 179Thread account name: NT AUTHORITY\NETWORK SERVICEIs impersonating: FalseStack trace: at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
In the middle of that huge event is this telling line: “Machine certificate is not found.”
Hmm. The Remote Web Access site is working fine at the moment, and that error hasn’t recurred since I updated the registry key (first part of this article). Maybe this was a symptom of the same issue. Will have to wait to see if it continues.
This all does leave me wondering what the “right” way is to renew an expiring certificate on Essentials. Maybe using the Anywhere Access wizard, even though we’re talking about the certificate issued by the local CA and not the Remote Web Access public certificate?