ASP NET Core HTTP Error 5025

We have been doing a round-robin-flu at work for the last few weeks, and this week it was my turn. It has been a frustrating and boring week in bed- but I’m finally recovering and I can start publishing the posts I’ve been working on since summer vacation.

We are doing a cold&flu round-robin at work and it’s my turn to enjoy a few days in bed 🤒
A post shared by Iris Classon (@irisclasson) on Sep 4, 2017 at 2:06pm PDT

As I’ve mentioned before we are migrating to ASP .NET Core. For the most it has been a smooth process, but a few weeks ago one of our Core services went down in Production. During lunch. Not good. Deploying at lunch is generally not a great idea if you don’t have green-blue deployment ((Not so) Stupid Question 292: How does blue-green deployment work?), but we had forgotten to update a certificate and had noticed during lunch so we hurried out a deployment. We use Octopus Deploy to manage our deployment pipeline, and since our certificate had been updated we had to create a new release. What we didn’t think about was that the new process steps for deploying new services had undergone some change that we hadn’t tested in AT and Prod (only Local, Edge and QA). We rushed out the deployment, all the steps were green. But you couldn’t log in. The Account service, a .NET Core service, was responding 500’eds. Not good. Being the one in charge for the pipe and anything ‘.NET Core’ I was tasked to drop everything and look into it.

In IIS, the Application Pool looked fine, as well as the service. At first glance might I add. But a simple call to our heartbeat controller returned 502.

I had a look in the eventlog and saw several errors.

The error:

Application ‘MACHINE/WEBROOT/APPHOST/KONSTRUKT.INTEGRATION’ with physical root ‘C:\Octopus\Applications\Konstrukt.net\PROD\Konstrukt.Service.Integration\0.8.2.282-release' failed to start process with commandline ‘“dotnet” .\ Konstrukt.Service.Integration.dll’, ErrorCode = ‘0x80070002 : 0.

The error code indicated that it couldn’t find a file. A similar error code that some got indicated access issues so based on experience I doublechecked access rights.

Here are a few basic things to check. Before you check the below go through the following more general steps:

Do a clean, compile and publish.
Recycle application pool, and reset IIS
Make sure that the startup is set up correctly (Startup.cs)
Go through the IIS deployment steps and make sure you’ve done each one
https://docs.microsoft.com/en-us/aspnet/core/publishing/iis
Make sure you have the right SDK and runtime version
Make sure you have the targeted .NET framework installed on the machine
Check that the *C:\Program Files\dotnet*exists in the path (run $env:path in Powershell )
Check Application pool identity access rights to assembly folder
Check Application pool identity access rights to dotnet.exe folder
Check your Web.config file, verify that the variables are correctly set*

* Example:

Application ‘MACHINE/WEBROOT/APPHOST/KONSTRUKT.SERVICE.INTEGRATION’ with physical root ‘C:\Users\msn\Documents\tmp16aug\Konstrukt\server\Konstrukt.Service.Integration' failed to start process with commandline ‘%LAUNCHER_PATH% %LAUNCHER_ARGS%’, ErrorCode = ‘0x80070002 : 0.

Means that the config file hasn’t undergone required transformation as the variables: %LAUNCHER_PATH% %LAUNCHER_ARGS% have not been set. Do a clean, compile and publish. Make sure Web.Config file looks right (should point to .dll or .exe depending on how you’ve chosen to publish the application- if it is self-contained or not).

Go through the common errors

I went through everything above, but the problem persisted.

I used started procmon and added a filter for the w3wp.exe process, and saw something interesting.

There is a lot of NAME NOT FOUND and PATH NOT FOUND. Nothing strange about that, the process is looking for the dotnet process in different directories until it finds it. Unless it doesn’t find it. Out of all the places it looked for the process, it never looked in the directory where it actually was. And the path was set at machine level. That made me think that it had to be a problem with accessing the environment variables. I went through the settings for the app pool again and was surprised to see that

‘Load User Profile’ was set to false. I’m not sure why. The pools and sites are set up automatically when we push a deploy with Octopus Deploy and all our sites are set up the same way except the Core services as we set the .NET CLR version to no Managed Code. This is optional by the way, but since the core services don’t rely on loading the desktop CLR it seems unnecessary.

To avoid this issue in the future I’ve written a PowerShell script module in Octopus Deploy, a simple function that set the Load User Profile property. We run this function as a part of our deployment (alongside other tweaks).

ASP NET Core HTTP Error 5025 - Process Failure

Comments