Hi all,

We face a difficult issue when loading a large FoxinCloud/WC app in COM mode : the 10th COM server chokes with one of these VFP errors occurring when instantiating a class :

• when instantiating a form class : Error #43 "There is not enough memory to complete this operation"

or

• when instantiating an imageList OLE class : Error #1426 "OLE error code 0x80004005: Unspecified error"

We get the exact same behavior using wc.dll (4.67) or the WebConnectionModule.dll .net handler (6.7.11.33730)

Conversely, we can load any number of instances of the same exe standalone without any error, just by repeatedly clicking the exe file from within Windows Explorer.

This screenshot shows 2 instances started in COM mode through wc.dll ('snoTest.exe' in blue lines), and 15 instances started as standalone executables ; both types use approx. the same amount of memory (in Working Set column) : 40-60 MB per instance.

This screenshot shows the result of starting 12 instances of the server in COM mode : only 10 have started, the 10th server (process ID 5532) has around 10 MB less memory than the other instances, probably because the startup procedure did not complete because of the error mentioned above.

WebConnectionModule.dll is aware of the issue as it displays Server snoTest.snoTestServer[11] failed to load in time.
wc.dll provides no feedback, displays the 10 instances loaded (the last with error) and just stops loading the 2 last instances.

Both cases of error occur on an instruction requiring a GDI handle. Each instance loads about 30 form classes, 300 for 10 instances, very far from the Windows limitations larger than 10k.

And again this issue occurs in COM mode only; we can load any number of instances as a standalone exe.

Are you aware of any limitation specific to the COM system? Could not google anything pertinent on this subject.

Server is: Windows 2022 64 bit running on 12 Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz RAM 32 GO (We never experienced the same issue with Windows Server 2012 R2 64 bit).

We have : Sys(3050, 1, 1024*1024*1024) and Sys(3050, 2, 1024*1024*1024) (1 GB)

Thanks in advance for your experience.

IMPORTANT UPDATE

I could instantiate 12 COM objects from within the VFP IDE without any error; the problem may not lie in the COM subsystem.

(First word of reqID is the process ID)

If you're testing on a client version of Windows the reason you potentially see issues is that Windows has some weird limiter that only allows 10 simultaneous request threads for IIS. Since it fails at 10 I bet the issue has to do with the limiter initiating a request and then killing the server resulting in a memory leak and using up some of the local storage that's used by Windows to keep the COM servers loaded.

Can you try this on a Windows Server version and see if you have a different result?

FWIW I gave this a try here locally and while I can load more than 10 servers and they run OK, any servers over 10 don't get any traffic (unless I go round robin) - even with explicitly slow requests only 10 will fire up and run simultaneously.

+++ Rick ---

Hi Rick,

Thanks for your reply.

All elements provided yesterday were taken from the server, except the COMtest.prg that I ran on my desktop, and today on the server (after installing a VFP IDE with an exe and a couple of dlls)

Obj01 = CreateObject('snoTest.snoTestServer')
Obj02 = CreateObject('snoTest.snoTestServer')
Obj03 = CreateObject('snoTest.snoTestServer')
Obj04 = CreateObject('snoTest.snoTestServer')
Obj05 = CreateObject('snoTest.snoTestServer')
Obj06 = CreateObject('snoTest.snoTestServer')
Obj07 = CreateObject('snoTest.snoTestServer')
Obj08 = CreateObject('snoTest.snoTestServer')
Obj09 = CreateObject('snoTest.snoTestServer')
Obj10 = CreateObject('snoTest.snoTestServer')
Obj11 = CreateObject('snoTest.snoTestServer')
Obj12 = CreateObject('snoTest.snoTestServer')

suspend

m.obj01.Destroy
m.obj02.Destroy
m.obj03.Destroy
m.obj04.Destroy
m.obj05.Destroy
m.obj06.Destroy
m.obj07.Destroy
m.obj08.Destroy
m.obj09.Destroy
m.obj10.Destroy
m.obj11.Destroy
m.obj12.Destroy

I got the same result on the server as on my desktop : all 12 instances instantiate without error, with 25 forms loaded and about 20 MB of memory per instance, 234 MB total.

I can start more instances by reducing the footprint upon startup and lazy load the forms whenever a user needs one.
For example, if I instantiate no form at startup, I can load 24 instances of Server without error (about 15 MB/instance), but the memory error occurs later when loading a form.

Based on COMtest.prg above, there seems to be a difference whether the COM call originates from within an IIS application pool and elsewhere.

FWIW, the IIS application pool and COMtest run under the same administrative account.

Yesterday I mentioned Windows Server 2012 R2 64bit, here is an application running on such a server : 10 instances for 420 MB total. this app. never had this kind of memory error.

I can't tell precisely at which level of memory the error occurs : can be less than 200 MB as well as over 300.

I don't understand how the amount of memory that the VFP process can get from the system may depend on the COM calling process…

Do you know any change in Windows Server 2016/2022 and / or IIS that could explain this behavior? any registry setting to change it?

I will send you by PM a link to a Google Sheet with my various measurements and observations.

Thanks in advance,

The issue isn't whether you can instantiate that many servers at a time - you can instantiate 50 and that will always work from FoxPro. You're doing it one after the other. From IIS these get instantiated side by side each on their own dedicated thread (that's what Web Connection provides) so that's different.

If failures occur due to COM and memory those failures will manifest when the server loads and immediately at the call site (ie. the COM method call). That's when you get an error from the Web Connection connector, not from FoxPro. The memory usage of COM operation should have no effect on FoxPro's internal memory usage - FoxPro gets its memory of the global Windows heap. The exception to that may be ActiveX controls which use some shared COM memory from a special pool.

If the problem occurs inside of FoxPro the issue is very unlikely to be COM related but related to something else. Honestly I'm not sure why you would have memory constraints inside of your application with this much memory available, but FoxPro error messages often are misleading. Could be something like an image failing to load. A million things can go wrong when loading a Form especially if hte forms contain ActiveX controls. If there are any issues they will be exaggerated by running under load and in COM.

FWIW - I just tried putting the message board running with 12 servers and then putting it under load. That worked fine, although I'm also seeing that the server load is cutting off at 10 servers and this is on Server 2022. But in this case it looks like it might just be the natural request busy cut off (since 10th only has 5 out of thousands).

I'm running on Server 2022 but I don't think that would matter. IIS and COM are pretty much in maintenance mode in Windows so there are haven't been any major changes there. The only thing that might be changing is the security environment/protection. I would also check anything anti-virus related and temporarily turn that off to see if that has any effect.

AV can effect how memory is used by applications and it often doesn't cause a problem until things are under load.

+++ Rick ---

Hi Rick,

Thanks a lot for your help.

From IIS these get instantiated side by side each on their own dedicated thread (that's what Web Connection provides) so that's different.

I understand, however wc.dll and wcm.dll instantiate the COM objects one after another, right? IOW waits for one to complete before starting the next one?

That's when you get an error from the Web Connection connector, not from FoxPro.

On my dev. machine (windows 10), with 12 instances, whenever the VFP error occurs (same as on Win 2022 Server "Error #43 "There is not enough memory to complete this operation"), wc.dll gives an error message :

• either "An internal exception occurred in the ISAPI application", and wc.dll gracefully unloads servers
• or "An internal exception occurred in the call to LoadServers (COM)", wc.dll keeps the loaded servers alive (9/12), and unloads them gracefully whenever required to do so (eg. when switching back to file mode).

So apparently either wc.dll is aware of the error in the VFP process, and/or the VFP error contaminates the w3wp/wc process.

update: the error does affect the w3wp process because, just after it occurs, the admin page is no longer accessible:

update: after stopping the COM servers (…/wc.wc?_maintain~Release), the admin page is back to live.

if the forms contain ActiveX controls

App startup does instantiates 2 forms containing a Treeview/ImageList, I've tried to remove these activeX from the startup procedure, without success.

FWIW, the startup procedure instantiates these COM objects :

• Scripting.FileSystemObject
• msxml2.domdocument.6.0
• msxml2.domdocument.4.0
• Shell.Application
• WScript.Shell

update: tried to remove these objects from startup, same error.

I'm also seeing that the server load is cutting off at 10 servers

Yes, that's how we know that we have enough servers to match the user requests … when the last instance is idle full-time.

It's strange that requests average response time increase with the rank of the COM server; we observe the same behavior while we never start more servers than the number of processors.

IIS and COM are pretty much in maintenance mode in Windows so there are haven't been any major changes there

Same belief here

We'll try to test this app. on a Windows 2012 R2, with limited hope however

anti-virus

Would you turn "real time protection" off?

update: tried with "real time protection" off, same error

Hi again Rick,

Though it's hard to believe, our very same app. works like a charm on Windows Server 2012 (high res. video)

Update : the test server is a Windows Server 2012 standard edition (IIS 8.0), not Windows Server 2012 R2 like I wrote earlier.

• no memory error
• 12 COM instances instantiated through wc.dll
• 12 instances with all resources loaded (170 forms, activeX objects, COM objects, etc. total under 90 MB)

What do you think?

• limitation starting with later versions of IIS / Windows Server that we can overcome with some registry setting?
• new feature that you're not aware of?
• regression in w3wp.exe?

Windows Server 2012 support ended 10-oct-2023….
Does any one know until when it'll be purchasable? (Vultr still sells it)

More details… errors logged by IIS / Windows 2022 :

Faulting application name: w3wp.exe, version: 10.0.20348.1, time stamp: 0x7feb3edd
Faulting module name: ntdll.dll, version: 10.0.20348.2031, time stamp: 0x6f2d20bf
Exception code: 0xc0000005
Fault offset: 0x0006234f
Faulting process id: 0x2258
Faulting application start time: 0x01da26b04ae07842
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: d6080da6-c01f-4ed2-ba2c-ad3b180ad16f
Faulting package full name: 
Faulting package-relative application ID: 

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> 
- <System> 
<Provider Name="Application Error" /> 
<EventID Qualifiers="0">1000</EventID> 
<Version>0</Version> 
<Level>2</Level> 
<Task>100</Task> 
<Opcode>0</Opcode> 
<Keywords>0x80000000000000</Keywords> 
<TimeCreated SystemTime="**2023-12-04T13:49:07.5163008Z**" /> 
<EventRecordID>22487</EventRecordID> 
<Correlation /> 
<Execution ProcessID="0" ThreadID="0" /> 
<Channel>Application</Channel> 
<Computer>guest</Computer> 
<Security /> 
</System> 
- <EventData> 
<Data>w3wp.exe</Data> 
<Data>10.0.20348.1</Data> 
<Data>7feb3edd</Data> 
<Data>ntdll.dll</Data> 
<Data>10.0.20348.2031</Data> 
<Data>6f2d20bf</Data> 
<Data>c0000005</Data> 
<Data>0006234f</Data> 
<Data>2258</Data> 
<Data>01da26b04ae07842</Data> 
<Data>C:\Windows\SysWOW64\inetsrv\w3wp.exe</Data> 
<Data>C:\Windows\SYSTEM32\ntdll.dll</Data> 
<Data>d6080da6-c01f-4ed2-ba2c-ad3b180ad16f</Data> 
<Data /> 
<Data /> 
</EventData> 
</Event>

Faulting application name: w3wp.exe, version: 10.0.20348.1, time stamp: 0x7feb3edd
Faulting module name: wc.dll_unloaded, version: 4.6.7.4, time stamp: 0x426a2377
Exception code: 0xc0000005
Fault offset: 0x0000dd3a
Faulting process id: 0x2258
Faulting application start time: 0x01da26b04ae07842
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: wc.dll
Report Id: 92661673-c490-4986-ad13-e426f8a0d88d
Faulting package full name: 
Faulting package-relative application ID: 

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> 
- <System> 
<Provider Name="Application Error" /> 
<EventID Qualifiers="0">1000</EventID> 
<Version>0</Version> 
<Level>2</Level> 
<Task>100</Task> 
<Opcode>0</Opcode> 
<Keywords>0x80000000000000</Keywords> 
<TimeCreated SystemTime="**2023-12-04T13:49:07.8444011Z**" /> 
<EventRecordID>22489</EventRecordID> 
<Correlation /> 
<Execution ProcessID="0" ThreadID="0" /> 
<Channel>Application</Channel> 
<Computer>guest</Computer> 
<Security /> 
</System> 
- <EventData> 
<Data>w3wp.exe</Data> 
<Data>10.0.20348.1</Data> 
<Data>7feb3edd</Data> 
<Data>wc.dll_unloaded</Data> 
<Data>4.6.7.4</Data> 
<Data>426a2377</Data> 
<Data>c0000005</Data> 
<Data>0000dd3a</Data> 
<Data>2258</Data> 
<Data>01da26b04ae07842</Data> 
<Data>C:\Windows\SysWOW64\inetsrv\w3wp.exe</Data> 
<Data>wc.dll</Data> 
<Data>92661673-c490-4986-ad13-e426f8a0d88d</Data> 
<Data /> 
<Data /> 
</EventData> 
</Event>

A process serving application pool 'snoTest' terminated unexpectedly. The process id was '8792'. The process exit code was '0xc0000005'.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> 
- <System> 
<Provider Name="Microsoft-Windows-WAS" Guid="{524B5D04-133C-4A62-8362-64E8EDB9CE40}" EventSourceName="WAS" /> 
<EventID Qualifiers="32768">5009</EventID> 
<Version>0</Version> 
<Level>3</Level> 
<Task>0</Task> 
<Opcode>0</Opcode> 
<Keywords>0x80000000000000</Keywords> 
<TimeCreated SystemTime="**2023-12-04T13:49:09.7973635Z**" /> 
<EventRecordID>125059</EventRecordID> 
<Correlation /> 
<Execution ProcessID="0" ThreadID="0" /> 
<Channel>System</Channel> 
<Computer>guest</Computer> 
<Security /> 
</System> 
- <EventData> 
<Data Name="AppPoolID">snoTest</Data> 
<Data Name="ProcessID">8792</Data> 
<Data Name="ExitCode">c0000005</Data> 
</EventData> 
</Event>

Notes:

• the process ID 8792 does not show up in task manager
• {3D55DFD4-9B41-4500-8531-6DCA47EA2FAF} is the GUID of my VFP COM server.
• VFP memory errors start AFTER the above process has aborted. (GMT +1) -- see below

04/12/2023 14:50:26
Awapphost.formlaunch(tcForm: 'menugen.scx', tlSubForm: .F., m.tcSuffix: '') - Process ID 9848
Method or procedure formlaunch_()
Error #2005 ("Error loading file - record number 3. menugen . Loading form or the data environment : There is not enough memory to complete this operation") at code line #2 783

The server {3D55DFD4-9B41-4500-8531-6DCA47EA2FAF} did not register with DCOM within the required timeout.

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> 
- <System> 
<Provider Name="Microsoft-Windows-DistributedCOM" Guid="{1B562E86-B7AA-4131-BADC-B6F3A001407E}" EventSourceName="DCOM" /> 
<EventID Qualifiers="0">10010</EventID> 
<Version>0</Version> 
<Level>2</Level> 
<Task>0</Task> 
<Opcode>0</Opcode> 
<Keywords>0x8080000000000000</Keywords> 
<TimeCreated SystemTime="2023-12-04T13:50:56.7509479Z" /> 
<EventRecordID>125063</EventRecordID> 
<Correlation ActivityID="{caf4feaa-5f5a-4551-a5ec-c731008e176a}" /> 
<Execution ProcessID="488" ThreadID="6032" /> 
<Channel>System</Channel> 
<Computer>guest</Computer> 
<Security UserID="S-1-5-21-3128523506-2161658953-3285229700-500" /> 
</System> 
- <EventData> 
<Data Name="param1">{3D55DFD4-9B41-4500-8531-6DCA47EA2FAF}</Data> 
</EventData> 
</Event>

So, with wc.dll, event sequence seems to be:

1. 13:49:07.5163008: one COM object raises a C000005 error to ntdll.dll
2. 13:49:07.8444011: exception propagates to wc.dll_unloaded
3. 13:49:09.7973635: IIS (WAS) is aware that a process depending on app. pool 'snoTest' has aborted
4. 13:50:26 (GMT) : memory errors occur when loading next COM server
5. 13:50:56.7509479: The server {3D55DFD4-9B41-4500-8531-6DCA47EA2FAF} did not register with DCOM within the required timeout.

With WebConnectionModule.dll, we only get this single error message:

The server {3D55DFD4-9B41-4500-8531-6DCA47EA2FAF} did not register with DCOM within the required timeout.

not always the same scenario, on some startup we simply have messages from DCOM and COM+:

Log Name:      Application
Source:        Microsoft-Windows-Complus
Date:          04/12/2023 17:06:38
Event ID:      4689
Task Category: (8)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      guest
Description:
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in com\complus\src\comsvcs\threads\staactivity.cpp(812), hr = 8000ffff: CSTAActivity: Failed to enqueue work.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Complus" Guid="{0f177893-4a9c-4709-b921-f432d67f43d5}" EventSourceName="COM+" />
    <EventID Qualifiers="49154">4689</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>8</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-12-04T16:06:38.4159098Z" />
    <EventRecordID>22528</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>guest</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Error in com\complus\src\comsvcs\threads\staactivity.cpp(812), hr = 8000ffff: CSTAActivity: Failed to enqueue work.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft-Windows-Complus
Date:          04/12/2023 17:06:38
Event ID:      4689
Task Category: (8)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      guest
Description:
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in com\complus\src\comsvcs\threads\activityworkqueue.cpp(364), hr = 8000ffff: CActivityWorkQueue: Failed to bind queue.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Complus" Guid="{0f177893-4a9c-4709-b921-f432d67f43d5}" EventSourceName="COM+" />
    <EventID Qualifiers="49154">4689</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>8</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-12-04T16:06:38.4159098Z" />
    <EventRecordID>22527</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>guest</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Error in com\complus\src\comsvcs\threads\activityworkqueue.cpp(364), hr = 8000ffff: CActivityWorkQueue: Failed to bind queue.</Data>
  </EventData>
</Event>

Log Name:      Application
Source:        Microsoft-Windows-Complus
Date:          04/12/2023 17:06:38
Event ID:      4689
Task Category: (8)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      guest
Description:
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in com\complus\src\comsvcs\threads\stathreadpool.cpp(1087), hr = 8000ffff: CSTAThreadPool: Unable to get bind thread.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Complus" Guid="{0f177893-4a9c-4709-b921-f432d67f43d5}" EventSourceName="COM+" />
    <EventID Qualifiers="49154">4689</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>8</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2023-12-04T16:06:38.4159098Z" />
    <EventRecordID>22526</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>guest</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">Error in com\complus\src\comsvcs\threads\stathreadpool.cpp(1087), hr = 8000ffff: CSTAThreadPool: Unable to get bind thread.</Data>
  </EventData>
</Event>

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          04/12/2023 17:06:38
Event ID:      10010
Task Category: None
Level:         Error
Keywords:      Classic
User:          GUEST\Administrator
Computer:      guest
Description:
The server {3D55DFD4-9B41-4500-8531-6DCA47EA2FAF} did not register with DCOM within the required timeout.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DistributedCOM" Guid="{1B562E86-B7AA-4131-BADC-B6F3A001407E}" EventSourceName="DCOM" />
    <EventID Qualifiers="0">10010</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2023-12-04T16:06:38.1971597Z" />
    <EventRecordID>125155</EventRecordID>
    <Correlation ActivityID="{213bbf38-3a35-4ab4-a1ec-297038481ee2}" />
    <Execution ProcessID="488" ThreadID="7968" />
    <Channel>System</Channel>
    <Computer>guest</Computer>
    <Security UserID="S-1-5-21-3128523506-2161658953-3285229700-500" />
  </System>
  <EventData>
    <Data Name="param1">{3D55DFD4-9B41-4500-8531-6DCA47EA2FAF}</Data>
  </EventData>
</Event>

What's in your Web Connection error log for the handler?

did not register with DCOM within the required timeout

What account are you running the IIS Application Pool Under? Are you using a different account for DCOM than for the Application Pool? If not consider not using DCOM Configuration at all (ie. pass-through) and setting the security at the IIS Application Pool Identity level. This sounds like it could be caused by the Identity transition from one security context to another.

Also make sure the server is registered with all of its interfaces (ie. latest version that is actually running is COM registered).

+++ Rick ---

Thanks for your help

What's in your Web Connection error log for the handler?

No error registered, just

2023-12-5 10:20:24:308   Forced Release of Server - 0
2023-12-5 10:20:25:317   Forced Release of Server - 0
2023-12-5 10:20:26:324   Forced Release of Server - 0
2023-12-5 10:20:27:331   Forced Release of Server - 0
2023-12-5 10:44:12:887   Forced Release of Server - 0
2023-12-5 10:44:13:895   Forced Release of Server - 0
2023-12-5 10:44:14:902   Forced Release of Server - 0
2023-12-5 10:44:15:911   Forced Release of Server - 0
2023-12-5 10:44:16:918   Forced Release of Server - 0
2023-12-5 10:44:17:926   Forced Release of Server - 0
2023-12-5 10:44:18:934   Forced Release of Server - 0
2023-12-5 10:44:19:942   Forced Release of Server - 0
2023-12-5 10:44:20:949   Forced Release of Server - 0
2023-12-5 10:44:21:956   Forced Release of Server - 0
2023-12-5 10:56:58:479   Forced Release of Server - 0
2023-12-5 10:56:59:488   Forced Release of Server - 0
2023-12-5 10:57:00:495   Forced Release of Server - 0

What account are you running the IIS Application Pool Under?

• Named account member of Administrators
• With LocalSystem: same behavior
• Tried AppPoolIdentity (snoTest), modified accordingly directory security settings and DCOM launching permissions, same behavior.

Are you using a different account for DCOM than for the Application Pool?

No, haven't tampered DCOM for years, always using the default settings, eg. launching user

the server is registered with all of its interfaces

AFAIK, interfaces only matter when trying to process requests, not at COM object startup time (late binding). Exact same behavior with ISAPI wc.dll and .Net wcm.dll.

I got that dump analysis from DebugDiag -- may reveal a clue

In w3wp__snoTest__PID__18652__Date__12_05_2023__Time_10_44_22AM__20__Second_Chance_Exception_C0000005.dmp the assembly instruction at ntdll!RtlpWaitOnCriticalSection+8f in C:\Windows\System32\ntdll.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to write to memory location 0x00000014 on thread 22

The bottom line remains :

• the exact same files (just 7-zipped, copied and extracted) work just fine on Windows Server 2012
• starts fine as a plain exe or COM object from anywhere but w3wp -- this excludes causes from VFP like corrupted memo or image, driver or activeX exception.
  Nonetheless, packed memo on all tables, removed everything dealing with printers (eg. foxyPreviewer), and tried without forms holding activeX.
• excluded app. and site directories, and snoTest.exe, from Windows Defender
• Max Worker process = 1
• reviewed and applied IIS best practices
• …

My conviction is that, in Windows Server 2022 (and probably 2016 as some of our customers have experienced -- occasionnaly, once in a while -- the same error with Windows Server 2016), w3wp somehow influences memory allocation to COM objects.

came across GCHeapCount in IIS best practices, can it have an influence?
(could not find the proper XML syntax to add a <configuration> <runtime> section in web.config)

The DCOM COM system is separate from IIS. You don't want to jump to conclusions based on circumstantial evidence. We don't know anything based on the error messages you have other than memory is being corrupted most likely because some internal pool in Windows is exhausted.

You can't duplicate this problem outside of a running application - because there are a number of special circumstances when running COM objects off multiple threads and especially so with Visual FoxPro. From the sounds of it it's Visual FoxPro that's failing so ultimately the problem lies there which is not very surprising since FoxPro was never designed to run under this environment. You're also pushing the limits with heavily running FoxPro forms inside of invisible COM servers and that adds to that stress.

The fact that it works and doesn't fail on first hits indicates that the most likely culprit is a memory leak. I would start by looking at memory usage of servers under load and see if they stabilize after a short bit of load - they shouldn't be using more and more memory.

If they are you have a memory leak - which BTW is easy to do with FoxPro if you don't release objects (on forms, nested refs etc. - I ran into that with Web Control Framework controls a bit and it was important to explicitly release). I'm sure you must have something in place in FIC but checking the actual memory usage is probably key.

FoxPro has unique rules about how it gets heap memory and it can run out of memory long before it hits VFP Process limitations.

Also see if you can pinpoint the actual error and request(s) that are causing the problem. Is it a specific page/request? Does it occur after a common number of requests/time? And again memory/thread usage may be an indicator...

There are no canned answers here. I certainly can't corroborate the Server 2022 assertion in my own apps that run 24/7 with constant load, and several apps that I've seen running on customer servers who I work with. So while that may be an issue in your specific case, it's not a universal issue.

I'm poking in the dark here as much as you are. You can keep posting your findings here and I will try to think of things that you can look at, but there aren't going to be any 'fixes' that can address this issue if the failure occurs within FoxPro itself.

+++ Rick ---

Hi Rick,

Thanks for your insights.

COM and w3wp are indeed independent; I can see it in a dump by DebugDiag.

Issue is not related to memory leak or any specific request; it occurs when server attempts to load a form into memory, always when reading record 3 of the *.scx where the form object is defined:
Error #2005 ("Error loading file - record number 3. adrtype <or one of its members>. Loading form or the data environment : There is not enough memory to complete this operation").
(it even happened when loading wwServerForm)

In fact the issue happens only if server starts as a background process.
If it starts in the foreground, everything loads fine: I can create 12 instances of the COM server, each with 150 forms loaded.

When executing in the foreground, the COM server loads these extra 3 dlls:

• C:\Windows\System32\MSCTF.dll loaded at 0x77020000
• C:\Windows\SYSTEM32\textinputframework.dll loaded at 0x743d0000
• C:\Windows\system32\uxtheme.dll loaded at 0x744b0000

All other dlls (90) are loaded in the same order and at the exact same address.

Windows 2012 Server does not have this behavior, it probably appeared with 2016.

When using interactive user in DCOM identity setting, wc.dll can load 12 COM servers with 150 forms instantiated and no error:

Interactive User requires having a user session open, which is not sustainable in production.
Do you know any other way to force a COM object to execute as a foreground process?

Note: we have Sys(3050, 2) = 536 870 912

¡¡VICTORY!!

After struggling for 2 1/2 weeks,

I could have wc.dll instantiate 12 background instances of my COM server with 150 forms loaded, without any error !

Just by using this user in DCOM config … and specifying the very same user as in the IIS application pool!

Champagne this evening!

Hmmm... that's what I was asking about to use the same user for IIS and DCOM. What were you doing differently? Explicitly setting the user?

By using this user DCOM passes the context through as is. IIS may be doing additional things to the user account that may not be assigned if DCOM is explicitly designed. If you specify a user - even if it's the same user it resets the environment because effectively the COM system is separate from the host process.

I'm not sure why this would cause a problem, and in theory it should not, but well, it's Windows and it's FoxPro ... 😄

+++ Rick ---

Although VFP is the usual ideal culprit, he was just a victim in this case; any other COM server with any other coding tech would have had the same issue.
Instantiating 150 forms means about 2,000 objects for a total of 90 MB RAM (including code), very far from the known limits of 60,000 objects and 2GB RAM per 32-bit process. I never saw a FoxinCloud server above 300 MB -- no memory leak here.

DCOM configuration documentation vaguely mentions the 'resources' for both identity settings :

• Launching User is the highest-level security protection identity setting. However, there are finite limits on resource consumption.
• Specified User: this type of server has a primary token and can access remote resources where a server that has the launching-user identity might not be able to. For more information about impersonation and access tokens, see Impersonation Levels and Cloaking.

That requires skills for reading between the lines…

What were you doing differently?

Like described earlier:

1. specify an administrative user in the IIS App. Pool and, according to WC documentation recommendation, use the DCOM default 'Launching User'. -> 'not enough memory' VFP error
2. explicitely set the same user in IIS App. Pool and DCOM config. -> no more error.

big relief…

Good find, Thierry. That would have never occurred to me.

I'm surprised that this is the case. Reading the docs you pointed at I would have read this, that it can be a problem of memory constriction due to use of a limited account.

I suspect the actual issue has something to do with Profile data. By using an external user (ie. not hte passthrough user) a whole new user context gets created and it gets a full profile, while IIS may or may not have a profile (depending on settings in the AppPool) or the profile may be more limited in storage usage.

IAC - great that you figured this out. I think I'll add a FAQ entry regarding this into the documentation.

Added to the Docs in the FAQ section:

Out of Memory Errors for High Load FoxPro COM Servers

+++ Rick ---

Hi Rick,

Thanks for sharing this issue with other WC users.

Just one last word… in IIS App Pool advanced settings, setting Load user profile to true did not help in my case.

However I see some discussion about the scope of 'user context' (CLSCTX enumeration) passed as third parameter dwClsContext to:

HRESULT CoCreateInstance(
  [in]  REFCLSID  rclsid,
  [in]  LPUNKNOWN pUnkOuter,
  [in]  DWORD     dwClsContext, // CLSCTX
  [in]  REFIID    riid,
  [out] LPVOID    *ppv
);

The CLSCTX enumeration has 2 parameters CLSCTX_DISABLE_AAA and CLSCTX_ENABLE_AAA referring to EOLE_AUTHENTICATION_CAPABILITIES that might be worth exploring.

The ISAPI module has been deprecated in terms of modifications and there haven't been any changes made to it for some time now. I'm not looking to updating the module any further, so what you see is what you get. It's been working over so many years so I am certainly not mucking with it now after it's been retired. 😄

The .NET interface uses standard COM instantiation .NET APIs which don't expose those flags (I think - I have to check actually) so they use whatever is considered the sensible default that's generally used which is the pass through context. The problem with some of the other options is that you have to actually provide the context meaning it has to be created from an account and that's a big problem in terms of getting the credentials etc. which is a big security risk...

The ISAPI module has been deprecated

Sure, however .net being a wrapper around Windows APIs, I'd guess there's something equivalent there.

Just an after thought, these missing 'resources' along a launching user DCOM invocation happen only when the target object runs in the background, not in the foreground…
eg. createObject('COM.class') in VFP command window has no restriction.

Hi,

The fact that setting the sole and same User solved your problem seems to indicate that your issue could be caused by the Desktop Heap Size !

See these pages about it :

• https://www.saotn.org/inexplicable-com-error-messages/
• https://web.archive.org/web/20170430180247/https://blogs.msmvps.com/alvin/2008/04/07/com-the-run-time-environment-has-detected-an-inconsistency-in-its-internal-state/
• https://learn.microsoft.com/en-us/archive/blogs/ntdebugging/desktop-heap-overview

Hope this helps? Alex

Didn't we have this very same discussion just a few weeks ago? Can you re-read that thread and your solution?

The issue is not that you can or cannot instantiate your COM instantances the same way from VFP - those are not loaded off of different OS threads - they're all loaded on the same UI thread or completely new processes loaded from the OS itself. That's not the same as getting launched of an IIS server thread.

As I mentioned in your last thread - what you really should look into is:

• Create a simple server and load that
• See if you can load more than 10 instances
• I know you can load many more instances than that although I've not seen servers over 10 get much traffic due to the way the COM system handles the thread marshalling inside of IIS
• If it works with the simple server (no activeX, keep it simple in your FIC stuff) then the issue likely is related to something inside of your application interfering with COM. My guess would be activeX controls or other COM activity that has to be marshalled to the UI thread from within the FoxPro code.

As before I can't really help here as I can only ensure that servers load. If they crash in your particular usage scenario it's up to you to try and pinpoint where the code actually crashes as hard as that may be (especially if it only occurs ten servers in).

As I said before key to get better debug information is to ensure the server does as little as possible during startup in the Init() phase used for server instantiation. Keep the server load simple and likely this problem goes away. You may have it later when requests go through but you have a better chance of capturing the problem.

+++ Rick ---

removing any code you can from the Init() of the server so the server can instantiate - I think what's happening most likely is that you're running into race conditions on your forms with ActiveX controls on them. It would probably be fine once the server is launched and properly instantiated but while in the loading state the thread marshalling likely hasn't occurred y

+++ Rick ---

Hi Alex,

Thanks for this contribution, however there were some significant differences between our context and the one you refer to:

• different error: not enough memory -vs- inconsistency in internal state
• the articles you mention are before 2010 while Windows Server 2012 works fine in our test case (problem was identified in Windows Server 2022 only)
• we are using COM (isolated server process) and not COM+ (in-process server)

Thanks anyway

Hi Rick,

AlexLaforge bumped the thread, not me… Never mind…