Hi Rick,

Great to see you at SW Fox. After our discussion, I'm trying to upgrade webconnect from v 6.21 to 8.0. The actual 'update' seemed to go fine with no errors when I ran setup.exe, but I tried to simply rebuild my .exe after (haven't changed any source code) and I am getting some errors. Not sure if you there's an easy fix to these or maybe I missed a step during the update?

Compiling c:\source\wconnect\classes\wconnect.prg #IF WWC_LOAD_WWLOCALEINFO Error in line 112: Preprocessor expression is invalid.

Compiling c:\source\wconnect\classes\wwhtmlhelpers.prg #IF BOOTSTRAP_VERSION < 4 Error in line 284: Preprocessor expression is invalid. #IF BOOTSTRAP_VERSION < 4 Error in line 323: Preprocessor expression is invalid.

Compiling c:\source\wconnect\classes\wwserver.prg #IF WWC_LOAD_WWLOCALEINFO Error in line 1061: Preprocessor expression is invalid.

Never mind. my local deploy folder a wconnect.h that didn't get update when I ran setup.exe. Simple enough to copy it over and it builds now.

Scott,

One thing that might be useful (depending on your application) is that there's a whole bunch of new infrastructure built into Web Connection 'projects', and it might be useful to move an existing application by recreating a new project and then attaching the old process classes back into the new project. It's relatively straight forward to do, and it buys you access to the new tooling (Launchers, LiveReload, Deployment scripts, live updates, and a predictable folder structure).

FWIW, here's is a write up of the discussion we had re: >16mb files along with the fix. The fix is in an unreleased update available for download.

• Making Web Connection Work with Response Output Greater than 16mb

+++ Rick ---

Rick,

Thanks for the info for the article.

A note on the Comm servers loading. I have put version 8 on our alpha testing server and am having a problem with the comm servers loading. Here's what I've been able to observe. We do a lot of SQL connections and not a lot of processing (for the most part) and so we tend to run 'comm heavy' if you will in regards to CPUs. Some of our production boxes (we use Vultr) have 2 or 4 vCPUs and our CPU sits at 20-40% and request times are usually <1 second so it seems to work fine. We currently run 15 comm servers per server since that's the max we could do with wc 6.21 and the IIS app pool recycle and still have them come back up. To simulate this, our alpha testing server (hosted at our site) has 2 vCPUs and I just put 20 comm servers in the count to test the new v8 comm server loading.

Here's what I've been able to observe (both through watching task manager and web connect admin page) with the comm servers. Note that this is with active requests (approx. 1-2 per second) hitting the server.

• If it's set to 20 comm servers and you restart the server (vm), everything comes up great.
• If you recycle the app pool, it struggles a little bit but ends up pulling out of it and has the 20 comm servers
• If you change the comm server count and click 'Update' on the web admin page, it goes haywire, shows 0 comm servers are running (doesn't say none are running, it's just an 'empty' space) but if you switch to 'File server' mode you see 30-40 servers and it keeps trying to do bring up more. You can also see the exe's in task manager growing. After you restart the VM, it comes up correctly
• If you programmatically update the exe (i.e. call UploadExe.wc & then UpdateExe.wc), it behaves the same as changing the comm server count and starts going crazy. Again, if you restart the server, it comes back up correctly

In summary:

• If you recycle the app pool or it's launching for the first time from a server reboot, it seems to do well
• If you 'programmatically' do something with the comm servers (either update the count or stop them to do an update) it never starts working right until you restart the server.

Hopefully that's detailed enough for you to be able to reproduce this.. Our biggest issue with webconnect is comm servers not coming back up after we do an update or the app pool recycles so hoping you can find a fix...

Can you make sure you get the latest version of the webconnectionmodule.dll? The earlier release versions had some issues.

• Web Connection Experimental

Latest version is v8.1.0...

+++ Rick ---

I'll give that a shot. Based on the admin page, it says I'm running: Web Connection Handler 8.0.1.1 (.NET Handler) • .NET 4.7.2

I can't remember because I went through quite a few iterations with updates with Richard Kaye. It's a post release update, but I don't think that was the final one. The current one definitely has the latest changes.

FWIW this is what it should look like if it's all working:

If you still have issues I'd be happy to take a look...

A few other things to consider:

• Take a look at the error log (wcerrors.txt)
• Make sure there is no or very little code in OnInit() - all startup code should be in OnLoad()
  (so that servers can instantiate and report errors if there are any and not fail during instantiation)
• Make sure it all works with single and a few servers before throwing 15 servers at it 😄

+++ Rick ---

Hi Rick,

So like I mentioned, I'm testing on our alpha server (didn't even know you could run the comm on the dev box 😉) and so I don't see the flash bring the servers. I'm more watching the exe's in task manager / on the admin page.

Unfortunately, it's still not working. I had it set at 20 from my test yesterday and restarted the server after I put the new files on it. It started up fine. I tried to switch it to 6 comm servers and it crashed hard.

I don't think I really have any code in either: OnInit() - or in OnLoad() (what prg are these in?), we put most of our code in the actual method calls since our 'startup' code is really establishing SQL connections to different DBs and is different per request...

The wcErrors.txt is littered with this just repeated:

2024-10-04 07:05:07.857 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #2: Server cpmobile.cpmobileServer[2] failed to complete in time.
2024-10-04 07:05:07.857 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #6: Server cpmobile.cpmobileServer[6] failed to complete in time.
2024-10-04 07:05:07.920 - 2302002747773_24 - Server Execution Error
Server cpmobile.cpmobileServer[6] failed to complete in time.
---
   at Westwind.WebConnection.ComServerManager.LoadServers()
   at Westwind.WebConnection.ComProcessor.LoadServers(Boolean initializeOnly)
   at Westwind.WebConnection.ComProcessor.ProcessRequest(Boolean recursive)
   at Westwind.WebConnection.ComProcessor.ProcessRequest(Boolean recursive) - /cpwc.wc?prg=cpGetUsers&proc=isWCUp&custcode=TESTEXT&token=
2024-10-04 07:05:12.971 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #3: Server cpmobile.cpmobileServer[3] failed to complete in time.
2024-10-04 07:05:12.971 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #6: Server cpmobile.cpmobileServer[6] failed to complete in time.
2024-10-04 07:05:12.974 - 230796904892_13 - Unhandled Exception (Mainline): Server cpmobile.cpmobileServer[6] failed to complete in time.
   at Westwind.WebConnection.ComServerManager.LoadServers()
   at Westwind.WebConnection.ComProcessor.LoadServers(Boolean initializeOnly)
   at Westwind.WebConnection.ComProcessor.ProcessRequest(Boolean recursive)
   at Westwind.WebConnection.WebConnectionHandler.ProcessRequest(HttpContext context) - /cpwc.wc?prg=cpGetUsers&proc=isWCUp&custcode=TESTEXT&token=
2024-10-04 07:05:18.007 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #3: Server cpmobile.cpmobileServer[3] failed to complete in time.
2024-10-04 07:05:18.008 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #4: Server cpmobile.cpmobileServer[4] failed to complete in time.
2024-10-04 07:05:18.008 - Failed to start COM Instance `Instance.Run()`: cpmobile.cpmobileServer #6: Server cpmobile.cpmobileServer[6] failed to complete in time.
2024-10-04 07:05:18.009 - 231298212114_24 - Unhandled Exception (Mainline): Server cpmobile.cpmobileServer[6] failed to complete in time.
   at Westwind.WebConnection.ComServerManager.LoadServers()
   at Westwind.WebConnection.ComProcessor.LoadServers(Boolean initializeOnly)
   at Westwind.WebConnection.ComProcessor.ProcessRequest(Boolean recursive)
   at Westwind.WebConnection.WebConnectionHandler.ProcessRequest(HttpContext context) - /cpwc.wc?prg=cpGetUsers&proc=isWCUp&custcode=TESTEXT&token=

Do you have time today to take a look or sometime early next week?

It's working better, I've been able to jump the COMM count around from 2 - 4 - 8 - 12 - 16 - 20 and then back down to 6 but it still 'randomly' (I can't quite tell the pattern yet, might be changing the comm count from 20 back to down to 6ish) crashes and once it does, it never comes back up. Restarting IIS doesn't help, have to restart the server. What I see is a bunch of our exes in task manager (43 in this case)

the web admin page looks like:

If I click the 'json' link you have, it shows:

{
    "Time": "/Date(1728049364635)/",
    "Configuration": {
        "MessagingMechanism": "Com",
        "ServerCount": 6,
        "AutoStartServers": true,
        "ComServerProgId": "cpmobile.cpmobileServer",
        "TempPath": "c:\\svn\\webconnect\\deploy\\temp\\",
        "TempFilePrefix": "WC_",
        "ExeFile": "c:\\svn\\webconnect\\deploy\\Cpmobile.exe",
        "UpdateFile": "c:\\svn\\webconnect\\deploy\\Cpmobile_Update.exe",
        "TimeoutSeconds": 90,
        "AdminAccount": "ANY",
        "LiveReloadEnabled": false,
        "LogDetail": false,
        "LiveReloadExtensions": ".wc,.wcs,.html,.htm,.css,.js,ts",
        "UseWebSockets": false,
        "WebSocketProcessScriptMap": "wwd"
    },
    "Servers": []
}

The config file you're showing is not for the Web Connection module - for that you need to look at web.config. But I don't think that's the issue there.

From the log it looks like you have requests that are timing out and then are getting shut down. It's ok if this happens occasionally, but if that happens a lot the COM system will collapse as these unloads will leave behind COM debris of an essentially 'crashed' instance. Get enough of these and the COM system will not load new servers or fail to basically respond to any operations. Same if servers crash. So if you have a lot of this sort of thing happening that's a problem.

If you have a server that's processing and taking a long time, while you are restarting that server will not release through COM. WWWC will hard kill servers after a second small timeout and that is one of the bigger improvements in the new version - it hard kills servers more consistently than the older versions did.

I'm a bit at a loss for why you end up with so many instances - the new version of the DLL is hard killing instances when you unload or reload. The only way that might be happening I think is if permissions are missing to do the hard kill operation (it requires Admin rights of IIS Application Pool), or something is preventing servers to be Task killed in some way.

Hmmm... this is actually something I have not checked. If you're running the App Pool with a non-admin account (ie. Network Service or a custom account) I wonder if the release cycle starts failing more.

+++ Rick ---

Ok, so I did some more experimenting and I think I can duplicate the behavior you're describing based on IIS Application Pool Crashes with a non-Admin account running in IIS.

Everything works just fine as long as the COM servers are not crashing or hanging - even with non-Admin accounts. In that scenario where servers are happily running servers immediately unload and reload. Great.

If servers are hanging or taking too long (long requests) during recycles/restarts, the COM release fails (not possible to release a COM object that is still processing) and at that point the server is effectively hung.

That's where the post release cycle hard kill operations kick in - but they only work if the application pool is running as an Admin account.

So I tested this with the following scenario:

• Run 10 instances under load
• 1 fast request, 1 very slow (5 second) request
• Under load this pretty much slows the server to the 5 second requests
• Recycle

In this scenario an Unload/Load Servers works but it takes a bit - it's not instant as WWWC waits for a bit (but not the full timeout necessarily) for servers to complete. If servers are finishing in time, servers unload and all's well.

For the second test I did the same as above but I explicitly killed the IIS Application Pool. This would be the equivalent of an Application Pool crash or a shut down that doesn't trigger the Web Connection/ASPNET shutdown routine.

At this point all servers stay loaded even though IIS has released the application pool.

Now depending on whether you use an Admin account or not the behavior varies:

• Non-Admin: The existing instances are not unloaded because Task Kill doesn't have rights to kill those instances
• Admin: The existing instances are unloaded because Task Kill can kill them before loading the new instances

So I think the answer here is that if you want reliable operation of many COM servers in a busy server pool you have to run the Application Pool with an Admin account to ensure any hung/timed out instances get cleaned up.

As I was telling Richard when we originally looked at this, the behavior shouldn't be any different than it was for older versions, except perhaps that the timeout to release the COM objects is shorter because the new version relies on the Task Kill operation to clean up any hung/timed out instances.

+++ Rick ---

Rick,

Appreciate your continued looking into this. I'm not sure this is actually correct.. If I remember correctly, 'LocalSystem' as the identity is the same as 'admin' correct? If so, we are running with admin rights and still seeing this behavior. If LocalSystem does not mean admin, I've been doing it wrong and need to know what the admin really is.

Hmmm... yes SYSTEM is an Administrator level account so that would definitely have rights to kill instances. Does it show SYSTEM at the bottom of the server admin page for Server Account?

So then I'm not sure why you would see multiple sets of instances. As you Load Servers (or when you recycle which is really Unload/Load) it should kill all exes running first, then launch the new ones.

Is it possible for me to take a look at your setup? (send me an email if you want to do that)

Also - when using the ASP.NET module it's no longer necessary to use 32 bit mode - you can run the server in 64 bit.

+++ Rick ---

So after a screen sharing session with Scott, we were able to pin down what's been causing the problems that Scott's been describing.

The short of it is that the problem was due to COM server's timing out while loading. Specifically it was due to the timeout for the cumulative loading of servers which was hard coded at 5 seconds which simply was not enough.

In recent versions of WWWC I have parallelized the server load sequence and at the time hardcoded a timeout of about 800ms per server with a max of 5 seconds to load all servers. So the idea is that we allow server count X 800 or 5 seconds whichever is smaller.

In my testing with local servers that's been plenty of time, since parellel loading of servers should have servers loading in parallel and so not take a long time. But... it looks like for Scott this was not happening and server load in general appeared to be taking quite a bit of time. Between 8 instances loading and parallelization not doing what I would expect it do - perhaps due to limited resources on the server - the load time was exceeding the timeout. This caused servers to fail to loadand then attempt reloading while the original server was still trying to load ending up with extra orphaned instances.

The quick fix was to bump numbers for individual servers and the cumulative number up and that immediately fixed the load problems.

The permanent fix now is to allow waiting for the full per server load time and having the instance load time configurable as a configuration value in the web.config or WebconnectionWebServerSettings.json file:

<add key="ComServerPerServerLoadTimeoutMs" value="2000" />

or

respectively. The default is 2000 which should definitely be enough to load servers - if you are slower than that you can tweak this number.

This update is found in the Web Connection Experimental updates file with version 8.1.2 of Web Connection Module and the Web Connection server.

• https://west-wind.com/files/WebConnectionExperimental.zip

This seems to have fixed the issue for Scott and may also account for the issues we ran into with Thierry Nievelet and initially Richard Kaye when servers failed to load.

A few thoughts on Load Timing

I found it interesting that in Scott's scneario the servers seemed to load very slowly. Servers weren't particularily large, and they did not have anything in the OnInit() code that fires during COM server instantiation (ie. server INIT()). Yet each server seemed to take about a 1 second to come up. Further, servers didn't appear to load in parallel which is also odd.

It's possible that the machine is resource constrained - it's a VPS machine running 2 virtual CPUs and with us sitting there watching the Remote Desktop connection chews up a lot of resources and threads. When servers were loading it almost seemed like parallelization had no effect at all with servers loading exactly one after another. It's possible that this is due to limited available threads for paralellization by the .NET code using the

+++ Rick ---

So after a screen sharing session with Scott, we were able to pin down what's been causing the problems that Scott's been describing.

The short of it is that the problem was due to COM server's timing out while loading. Specifically it was due to the timeout for the cumulative loading of servers which was hard coded at 5 seconds which simply was not enough.

In recent versions of WWWC I have parallelized the server load sequence and at the time hardcoded a timeout of about 800ms per server with a max of 5 seconds to load all servers. So the idea is that we allow server count X 800 or 5 seconds whichever is smaller.

In my testing with local servers that's been plenty of time, since parellel loading of servers should have servers loading in parallel and so not take a long time. But... it looks like for Scott this was not happening and server load in general appeared to be taking quite a bit of time. Between 8 instances loading and parallelization not doing what I would expect it do - perhaps due to limited resources on the server - the load time was exceeding the timeout. This caused servers to fail to loadand then attempt reloading while the original server was still trying to load ending up with extra orphaned instances.

The quick fix was to bump numbers for individual servers and the cumulative number up and that immediately fixed the load problems.

The permanent fix now is to allow waiting for the full per server load time and having the instance load time configurable as a configuration value in the web.config or WebconnectionWebServerSettings.json file:

<add key="ComServerPerServerLoadTimeoutMs" value="2000" />

or

<ComServerPerServerLoadTimeoutMs>2000</ComServerPerServerLoadTimeoutMs>

respectively. The default is 2000 which should definitely be enough to load servers - if you are slower than that you can tweak this number. I've also removed the max wait value for the timeout so now the max value is simply serverCount x ComServerPerServerLoadTimeoutMs.

Note the wait value only kicks in if all servers have not loaded yet - once all have loaded code continues. Where this can potentially cause problems if a few servers fail or hang causing the entire pool to now have to wait for the full timeout. This was what the fixed value was supposed to help prevent initially, but this is perhaps too volatile - better to wait longer on single server fails than to fail with servers timing out.

This update is found in the Web Connection Experimental updates file with version 8.1.2 of Web Connection Module and the Web Connection server.

• https://west-wind.com/files/WebConnectionExperimental.zip

This seems to have fixed the issue for Scott and may also account for the issues we ran into with Thierry Nievelet and initially Richard Kaye when servers failed to load.

A few thoughts on Load Timing

I found it interesting that in Scott's scneario the servers seemed to load very slowly. Servers weren't particularily large, and they did not have anything in the OnInit() code that fires during COM server instantiation (ie. server INIT()). Yet each server seemed to take about a 1 second to come up. Further, servers didn't appear to load in parallel which is also odd.

It's possible that the machine is resource constrained - it's a VPS machine running 2 virtual CPUs and with us sitting there watching the Remote Desktop connection chews up a lot of resources and threads. When servers were loading it almost seemed like parallelization had no effect at all with servers loading exactly one after another. It's possible that this is due to limited available threads for paralellization by the .NET code using the

+++ Rick ---

Thanks Rick for your help with this. I've been doing some more exhaustive testing and over-loading the server even more (i.e. 150 simulataneous requests with 20 comm servers on a strapped 2 vCPU while I'm watching it), because why not 😉 and every test I've run so far has been successful. There might be something else I run into once we roll this out to production, but, as of right now, all seems to be working and it pulls out of it. There's the UI delay you and I noticed but other than that it's recycling, destroying, bringing up Com servers as expected.

That's great.

I think the timeout was what was causing some of the issues I ran into with others who were testing the improvements when the parallel load and explicit shutdown was rolled out. I'm surprised that servers are taking so long to load since they are supposed to load in parallel. I figured 5 seconds at the time should have been plenty when all servers are loading side by side, but it seems that parallelization is not guaranteed to work especially on VPS that are CPU/thread starved.

I haven't found anything better than potentially having to wait for the full 1 server x timeout wait time if there is a problem. I suppose this should be rare so we can probably live with it - well, we have to 😄

+++ Rick ---

Hi Rick,

We put V8 on one of our production servers and the comm loading is working well. However, I checked the logs this morning and I'm seeing 2 weird errors that I don't understand. I'm not sure if they're related to v8 or something else but since I never saw them in version 6 we were running before, going to say yes 😉

inside of wcErrors I'm seeing several errors of:

*** Com Server ProcessHit Error: Server not available, trying to reload: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) (800706ba) Time: 0.00ms - /<<querystringhere>>

It always has the same hresult even if it's different requests we're trying to process. Do you know what would cause this error?

The other error I'm seeing is not in wcErrors but is happening in wwJSOnService.prg I modified wwJsonService a while back (and copied my changes over to the v8 wwJSONService.prg) to send us an email when an error like this happens so we can fix it. Normally it's our code but in this case doesn't look like it. This is the code I modified in wwJSOnService just so you know:

IF llError
	*RETURN this.ReturnExceptionObject(loException.Message)
	* Let us know that it crashed. SRR 02/20/2024
	
	IF VARTYPE(g_production) = 'L' AND !g_production
		* Debugging. Hit an error. Stop here so we can look at it. SRR 02/22/2024
		LOCAL m.runline
		m.runline = 'SET ' + 'STEP ' + 'ON'
		&runline
	ENDIF 
	
	TRY 
		LOCAL oError
		oError = NEWOBJECT('cpError', 'cpError.prg')
		oError.sendWC2Support(this.ErrorMessage) && doesn't send it if !g_production
	CATCH 
	ENDTRY 
	
	RETURN this.ReturnExceptionObject(this.ErrorMessage)
ENDIF	

this.ErrorMessage is:

File 'jsonencodestring.prg' does not exist.

When I do a gofish on that, it seems to be inside of wwIPStuff.dll and a few of your prgs are trying to use a method called jsonencodestring. What doesn't make sense to me is looking at our logs, we got this error pretty consistently for about 2 hours yesterday approx 3pm to 5pm but if I re-run the queries that caused it, it works right now. Our app pool recycled just after midnight so that doesn't explain why it stopped around 5 but I do know I was getting the Comm server processHit error roughly around the same time as this error, although I have no idea if the 2 are related or separate errors. Thoughts?

That error means that a server has died and cannot be retrieved from the server pool. Most likely the server crashed or hung up and is not responding to COM commands and Web Connection captures that error and creates a new instance.

Error reporting has improved quite a bit since v5 especially in the last go around for the v8 refactoring of server loading so you'll see more descriptive errors and potentially more of them than before. That error typically means that the server failed to be accessed but a new one was created and the request run so in theory it's not an error that affects the end user.

You do want to be careful of these though - too many of these means there will be trouble. When servers die you end up getting COM inconsistencies. Occasionally this is OK, but if it happens frequently you eventually run out of STA slots to create new server instances. The only fix at that point is an App Pool recycle.

+++ Rick ---

Sounds good. Just as a follow up on the loading in parallel. I put wc v8 on 2 servers that both have 2 vCPUs and they load synchronously / 1 comm server at a time. Just put it on a server with 4vCPUs and it loaded in parallel / 2-3 comm servers at a time (just watching task manager to see this). The one with 4vCPUs also has more memory so not sure if that plays in at all but it loaded multiple comm servers at once with the 4 vCPUs.

Yeah I think 2 vcpus while connected with RPC is taxing the processors and there might just not be enough thread resources available to actually run in parallel. Add to that the STA gets lower priority in the system and it might just be that the threads are not being made available to load new STA thread instances in parallel. 4 vcpus gives a lot more room to run.

On my local machine (it's an i9 laptop so lots of threads) I can load 40 servers in about 3 seconds. But even there I see the 'staggered' loading of about 5 servers at a time. OTOH, on the local machine loading that many servers is pointless because Windows client only allows for 10 active connections and I actually only see 8 ever getting requests (there must be something else unrelated to IIS holding those other 2).

The updates have now shipped with v8.1.

+++ Rick ---

Another question on the comm servers loading. Just to test load capacity / what we should set the comm server count to on our 4 vCPU box, I tried to set the Comm count to 40. I tested with a computer in the office and set it to 100 earlier just for kicks and it worked great. However, on this VM with 4 vCPUs, it will not load more than 32 comm servers. I upped the Comm server loading timeout setting you added and that didn't make a difference, max is 32. Inside of WC Errors I can see a bunch of the following error repeated:

2024-10-10 14:21:10.041 - COM Instance thread initialization failed (`Instance.Run()`): cpmobile.cpmobileServer #36: Retrieving the COM class factory for component with CLSID {E9BAB4D7-AE90-4A2D-BC08-D33E72DC285C} failed due to the following error: 80080005 Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)).

Is this just some hard limit on the virtual machines or is something else going on? The CPU is fine (around 20% even with me being connected in, it's got 8GB of ram, I'm not sure it's a 'resource' issue per-se but who knows. Thoughts?

It's not a good idea to run so many instances for a number of reasons - slow startup time, resource usage and as I mentioned in my last message there are only so many STA threads available for processing and they are de-prioritized by the system. General rule is 2 instances per processor, possibly more if most of your processing wait time is for I/O (ie. waiting on SQL Server or some other DB). If you're CPU bound, extra instances are going to be doing very little good for you.

Best way to see how far you get is load up a given number of instances that larger than what you think you'll use and see how many instances are actually getting hit (in first available mode - not round robin). As long as you're not CPU bound you can use the count up to the one that has steady hits that fill the view. Any beyond that is just waste of resources.

So basically, run under load (with something like West Wind WebSurge or similar) add servers until one of these is true:

• Add servers until they barely get hit (or not at all)
• Add servers until CPU pegs and stays over 50% then back off

+++ Rick ---

Hi Rick,

Understand but here's where we're coming from. With v6.21 we were able to run 15 comm servers per VM. Our CPU usage is relatively low and every request talks to SQL so they do a lot of waiting and there's not a lot of processing for the majority of the requests we process. Currently we have it running a VM with 2 vCPU with 20 comm servers and that seems to be going well. On a VM with 4 vCPU I have the COMM count set to 20 and It keeps erroring out when the app pool recycles. These are the error(s) that first show up in the log when the app pool recycles.

2024-10-11 06:20:38.107 - COM Instance thread initialization failed (`Instance.Run()`): cpmobile.cpmobileServer #8: Retrieving the COM class factory for component with CLSID {E9BAB4D7-AE90-4A2D-BC08-D33E72DC285C} failed due to the following error: 80080005 Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)).
2024-10-11 06:20:41.140 - 440338312741_7 - Unhandled Exception (Mainline): Retrieving the COM class factory for component with CLSID {E9BAB4D7-AE90-4A2D-BC08-D33E72DC285C} failed due to the following error: 80080005 Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)).
   at Westwind.WebConnection.ComServerManager.LoadServers()
   at Westwind.WebConnection.ComProcessor.LoadServers(Boolean initializeOnly)
   at Westwind.WebConnection.ComProcessor.ProcessRequest(Boolean recursive)
   at Westwind.WebConnection.WebConnectionHandler.ProcessRequest(HttpContext context) - /cpwc.wc?<<params>>

What would cause the (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)) error?

Our hope was to utilize more CPU performance with a 4 cpu box compares to a 2 cpu box so we don't have to bring up as many servers since once the Comm servers are humming along / load we can handle the load and CPU sits around 25-30%. That may not be possible but it seems wrong to pay for more VMs when the VMs we have have resources to spare, we just can't spin up enough comm servers. That may be the reality and nothing we can do about it but hopefully that at least explains what we're trying to accomplish. Right now though I need to understand this error I'm getting even with a smaller Comm count.

Not sure - it sounds like the server is running out of resources.

FWIW, I gave this a shot on the message board here - I set up a WebSurge test that fires the homepage under load and then loaded and unloaded 20 servers repeatedly.

Here's what that look like:

This is running on a 4 vCPU / SSD Vultr instance with 15 other sites running on the same server.

One difference is that my requests are very quick - everything runs in a fraction of second (less than 1/10th of a second) so there's little to no waiting for servers to complete before shutting them down. This would slow things down a bit as Web Connection waits on the COM servers to shut down for a bit of time (but not the full timeout of the request).

+++ Rick ---

No wonder you were giving me the stink-eye with my half-second average request time. I'm in slapback delay land and you're flanging everything... 😉

FWIW, the actual server execution and scheduling hasn't changed between any version. The only thing that was changed is how servers are loaded during initial startup. So performance related issues should be no difference beyond the initial server loading.

Took a quick look at this related to the CO_E_SERVER_EXEC_FAILURE which apparently has something to do with desktop and user profile access:

https://stackoverflow.com/questions/22062284/server-execution-failed-exception-from-hresult-0x80080005-co-e-server-exec-fa

If you recall when we were poking around your server I think I made you turn off the Use User Profile option - you might want to turn that back on. It's possible your application is referencing something that requires access to a user profile (registry, system configuration, some other COM services perhaps)

+++ Rick ---

Okay. I'll play around with those settings and see if it makes a difference (fyi, we had it turned off and you had us turn it on so I just turned it back off to see if that makes a difference). You also had us change enable 32 bit apps to false and I turned that back to true for now, just grasping at what could be causing this.

Hi Rick,

Just curious, what are the advantages / disadvantages of setting the app pool to 32 bit mode vs 64 bit mode? Since VFP is 32 bit, I'm guessing the 64 bit mode has something to do more with your .Net code that's managing the comm servers but not entirely sure? Just want to know the implications / ramifications of my messing with this setting 😉

I don't think it matters a lot, but IIS natively and by default uses 64 bit, so since we don't have to be running 32 bit, I figure go with what the default is - one less thing to configure and most likely the most optimized path in the application.

VFP is 32 bit, but we can run the Application Pool in 64 bit since we are loading out of process COM servers - they are not running inside of IIS or the IIS host process.

+++ Rick ---

Hi Rick,

I read an article but it didn't give a lot of details. I was wondering if you could either point me in the right direction (i.e. a more detailed article) or explain to me the benefits of the Comm vs file server modes?

I'm toying with the idea (may be a bad one 😉) of trying out one of our servers in file mode but just curious what the pros and cons are? The reason for this is the generic comm errors I'm seeing in the wcError log. It's possible our server is overtaxed as well and that's what's causing it but would just like to be able to make an informed decision. i.e. how much slower is file vs comm? can you still programmatically update the exe via web calls by putting requests on hold and then updating the exe? Do the file servers start up automatically when the computer reboots or do you only get that with comm? etc...

I’ve seen this happen when constants like WWC_LOAD_WWLOCALEINFO and BOOTSTRAP_VERSION aren’t defined or updated properly. You might need to check your wconnect.h file and make sure those constants are defined.

Hi Rick,

Not sure yet what's 'causing it' but I was finally able to track down a major problem we've been having since updating to v8. I was getting a slew of errors of "File 'jsonencodestring.prg' does not exist".

Inside of wwJSONSerializer you have:

FUNCTION WriteString(lcValue)
LOCAL lnK, lcResult, lcValue,lcOutput

IF THIS.TrimStringValues
   lcValue = TRIM(m.lcValue)
ENDIF

IF EMPTY(lcValue)
	this.cOutput = this.cOutput + [""]
	RETURN
ENDIF	

*** Optimized for perf with C code in wwIPStuff.dll
*** JsonEncodeString
LOCAL lcOutput, lnBufferSize
lnBufferSize = LEN(lcValue) * 5 + 3
IF (lnBufferSize < 16750000)
	lcOutput = REPLICATE(" ",lnBufferSize)
ELSE
    *** Set to max string size and hope for the best!
    *** The calculated value is overly pessimistic so
    *** as long as the output is under 16 megs it'll still work
	lcOutput = REPLICATE(" ",16750000)
ENDIF	

PUBLIC __JsonEncodeStringAPI
IF !__JsonEncodeStringAPI
	DECLARE INTEGER JsonEncodeString ;
	        IN wwipstuff.dll ;
	        string  json,string@  output
	__JsonEncodeStringAPI = .T.
ENDIF	


lnPointer = JsonEncodeString(lcValue,@lcOutput)

And __JsonEncodeStringAPI was still around and set to .T. but the reference to JsonEncodeSTring inside of wwIpStuff.dll is gone and so it crashes. I put a band-aid on my code at the end of every request I release __JsonEncodeStringAPI and that seems to be helping... Again. Not sure yet why it's losing the reference to wwIPStuff and the method in there but it's causing our Comm servers to hang / crash because of it. I got 1800 errors last night from it and was finally able to track this down as quick and dirty fix.

Have you ever run into this?

Double checked and those are defined in my wConnect.h Thanks for the thought. Just to clarify, what have you seen happen if they aren't around? There's a few things going on in this thread we've talked about so not sure which one you are referring to?

Hi Rick,

We're still testing but it's looking promising but I found a legacy spot of code that was issuing a 'Clear DLLs' and it was breaking your stuff where you had the public variable set to say you had already declared it but then VFP didn't know about it. Hoping this resolves the comm problems we've been having...

Ugh - yes CLEAR DLLS will cause all sorts of issues in Web Connection I think - there are lots of API calls that are pre-loaded and cached. More recent ones check for status, but older ones don't and just assume that things are loaded.

I have to ask: CLEAR DLLS in code - why? 😄

That should never be necessary - more for dev environment.

+++ Rick ---

Looking at it, it was trying to resolve a dynamic DNS name that constantly changes and was doing a bunch of DECLARE statements and was trying to be environmental friendly / clean up after themselves but rather than release each one individually just did a CLEAR DLLS.

So far so good. Last night was the most stable night we've had since implementing V8 so I think we might be on to something here.

I'd still be interested the breakdown of comm vs file servers. I think the comm is working well now but I'd like to understand it better vs the file server mode.

Re: DECLARE caching

PUBLIC __JsonEncodeStringAPI
IF !__JsonEncodeStringAPI
	DECLARE INTEGER JsonEncodeString ;
	        IN wwipstuff.dll ;
	        string  json,string@  output
	__JsonEncodeStringAPI = .T.
ENDIF	

The more I think about it the more I think this sort of thing is not worth it. Running a few tests on this with a million iterations resulted in a fractional percentage performance increase.

Instead I added an Init() and initialize the DECLARE in there. That way at least the serializer will recreate on the next request and as a bonus the code doesn't have to check the variable for a slight performance boost.

+++ Rick ---

Re: COM servers vs. File based

Short story - don't use file servers in production. You lose a lot of functionality and performance. And unless your servers are 100% reliable and NEVER crash (unlikely with FoxPro in general), you're likely to run into unresponsive servers that are not restarting eventually especially in high volume environments.

There are lots of reasons for using COM servers.

Advantages of using COM

• Auto-loading
  COM servers automatically load when first accessed and if individual servers are not running or have crashed they will automatically restart. File servers can only auto-start on application start or manually and explicitly and there's no server monitoring for crashed or hung instances.

• Automatic Crash Recovery
  If a server crashes, COM can usually detect that and either catch the exception directly and restart, or if it hard crashes start a new instance on the next request if the server can't be retrieved or fails to be available on the request. This is not possible in file based and probably the biggest reason you should use COM.

• COM is faster
  COM is a direct 1-1 interface that fires immediately when a request is made. File based checks for requests files on disk and there's a delay between requests (the timer interval basically). There's also additional overhead for writing output to file and for the module to read that file. It's not exactly slow but in very busy scenarios this overhead can make a difference. Also for very busy file servers the volume of temp files written and read can become a problem at the file system level.

• COM Server Management
  COM Servers are tracked by the module - each server instance has request stats and you can individually manage and shut down each instance. If you kill an instance it's automatically restarted. Shutdown directly shuts down either individual instances or all instances - file based can only interact via Process Kill operations.

• Server Live Updates
  COM servers can be automatically updated with a simple script or manually which handles server shutdown, server updates, and then restarting. Possible with file mode manually, but it's not reliable and requires manual restart.

Advantages of use File Based Servers

• No COM registration required
• It's possible to run the server interactively and from within Visual FoxPro (ie. perfect for debugging/development)
• You can run the server on separate machine from the Web Server by pointing the temp folder to a network location
• Possible to run Web Connection Web Server on Linux and run FoxPro on Windows looking for semaphore files on Linux

+++ Rick ---

Hi Rick,

That's very interesting on the declare not having a real performance degradation. Good to know. Taking out our clear dlls has fixed the problems we were having. (imagine that 😉 )

Appreciate your detailed breakdown of Com vs file.

FWIW, I've also updated the following topic, which has had most of this information I detailed earlier in it:

• When to use COM

+++ Rick ---