Getting odd behavior at one of our customer sites after switching them from ISAPI to .NET. We start the app, things work fine for a while, then pages start timing out with this error:

Unknown application error
Server aw3.acewebserver[4] failed to start in time.

Bringing up Task Manager from the server console shows an absurd number of active processes: There's no issue with COM server instances in the Handler Status Page: Though attempting to reload COM servers also throws an error.

It would appear the customer mucked something up when doing the .NET switch but I'm not sure what to have them look at. Any suggestions would be appreciated.

--stein

Unloading COM servers should release all instances, including ones that aren't currently active as it does a force reload after servers have been released via COM reference unload.

If that doesn't work here's another trick to unload all instances:

• Briefly switch to File Mode
• Go click on Unload Servers

This will unload all EXEs that are loaded for your server.

If you have this many instances it might be that there is perhaps another configuration being accessed and launching in file mode perhaps?

Orphaned instances either come from file mode launches, or from hard crashes of the IIS Worker process (the Application Pool) that prevents Web Connection from shutting down server instances as the server shuts down. If the latter is the case you should probably see some errors in the Web Connection server log (in the temp folder).

+++ Rick ---

So when I asked the tech to check wcerrors.txt in C:\temp\WC he reported that there was no such folder. I had him create a folder with the proper permissions so we'd at least have a log file to check. But as soon as that was done, we stopped seeing those extra processes.

Evidently each attempt to write a log entry to a nonexistent location was generating an orphaned instance. After enough of those the system would bog down and we'd have to clear them. (BTW, that tip about switching to File Mode to unload the servers worked great - though hopefully we won't need to use it anymore.)

Thanks,

--stein

This is in .NET Mode?

I would expect an exception in the handler, but not a crash that would fail and hang if the file can't be written to. Odd... But - the temp folder access is crucial for error handling so it's definitely a requirement.

WWWC will always try to create the file and folder if it doesn't exist, but if there's no permissions to do that that's not going to work obviously 😄

+++ Rick ---

Unfortunately I spoke too soon - the customer initially told us things were working but in fact they are still getting extra processes. Now at least we are seeing entries in the wcerror log, but I can't really make sense out of them. These are typical:

2022-05-23 10:17:28.759 - 775106401063943_53 - Com Server Execution Error: wwscripting.renderaspscript c:\inetpub\aceweb\classes\wwscripting.prg Error in line 691  -1073741819 - /wconnect/redapple/XLogon.awp1?Target=xwelcome.htm
2022-05-23 10:35:56.872 - 895677912454365_35 - Com Server Execution Error: wwscripting.renderaspscript c:\inetpub\aceweb\classes\wwscripting.prg Error in line 691  -1073741819 - /wconnect/cancelreg.awp
2022-05-23 10:41:12.471 - 927204948944429_35 - Com Server Execution Error: wwscripting.renderaspscript c:\inetpub\aceweb\classes\wwscripting.prg Error in line 691  -1073741819 - /wconnect/awJSON.awp?target=wkshpmst%3bwmtitle%2ccadesc%2cNiceDate(wmdate%2c%2c3%2c8)+AS+wkdate%2cwmtime%2cwmfeeamt%3bwmcrse%3d%27CAMP22SU160B%27%3bwkdate%2cwmtime%3bINNER+JOIN+catalog+ON+UPPER(wmcatcode)%3dUPPER(cacode)&_=1653320469091

By far the majority are rendering errors from wwscripting but the methods being executed are not script pages, nor do they contain calls to Response.expandscript. So I'm confused as to what's going on.

Yes, this is in .NET.

--stein

Looking at the code in wwScripting, it points at the point where it's trying to run the compiled PRG file.

Two things:

• This shouldn't fail: Make sure you're not running in Debug mode
• if that's not it, the FXP file is not there and it's a hard failure

This presumes this is a recent version of wwScripting, that lines up with my current line numbers, but my guess is you're running in debug mode and the file is not there or has compilation errors.

+++ Rick ---

I assume my source matches yours:

   TRY 
	  *** Just execute the PRG File
 	  DO (lcFxpFile)  && <=== Line 691
   CATCH TO loException

How would I figure out if I'm running in Debug Mode?

That code shouldn't be firing if debug mode is off. DebugMode is a flag in the yourApp.ini file (for v6+).

+++ Rick ---

Well, it should not be on - this is from their INI file:

[Main]
AdminEmail=
AdminMailserver=172.17.32.162
AdminSendErrorEmail=On
Logtofile=On
ComReleaseurl=https://comed.sf.k12.sd.us/wconnect/wc.wc?_maintain~Release
Debugmode=Off

I really don't get what's going on. We're running this same app at 100+ other sites and are not seeing any of these issues...

Looking at your code that actually is the right line of code that should be firing which is the exception handler that then then re-throws the exception. In my code though (which might be more current) it points at the code above that doesn't have the exception handler.

The latter shouldn't cause an error at that location though because it's wrapped in an exception handler. That handler should always handle the exception even if it gets re-thrown later so I'm not sure what the issue is, other than some sort of problem with loading the FXP file which might be a fatal error that causes to bypass the normal error handling chain.

+++ Rick ---

Just a note in case anyone has been following this thread...

We were never able to pinpoint anything in either the code or configuration that was causing these extra process instances to remain in memory. It was noted that the problem seemed to appear after we moved them to the .NET handler, so just for the heck of it I reverted their config back to ISAPI. That appears to have resolved the issue - it's been 2 days and we have not seen any unwanted processes.

So the issue seems somehow related to .NET, though they were running a recent release (4.7.2) under Windows Server 2019 which should have been stable. We've suggested to the client that they roll out a second server that we can work without impacting the production box. Maybe all it will take is a fresh install of Windows, .NET and IIS.

--stein

Stein,

Not sure why there would be a difference between the ISPAI and COM interfaces - at the end of the day we're instantiating COM objects and releasing them. I suspect the problem is that due to the failure we identitified somehow the COM server hangs, but doesn't release which then causes a timeout and a hung instance.

As mentioned during our debug session:

• Make sure that UnattendedComMode=On in YourApp.ini
  This will ensure that your server never hangs on some sort of dialog like an open file dialog if a file is not found etc. Without this servers can hang.

Additionally, I took a look at the code today and one difference between COM and ISAPI is that the ISAPI extension explicitly tries to kill the server if it's still running after releasing. IOW, it tries to release, then waits a bit and sees if the process is still running and if it is kills it.

I've added similar functionality to the UnloadServer() functionality in the .NET Module now as well.

If you want to try this out grab the updated DLL from:

• Web Connection Experimental

+++ Rick ---