Hi Rick,

This is a two part question. Last night I had an incident in my production environment where I was getting out of memory errors from VFP. The error appears to be related to Craig Boyd's regexp.fll which I've been using for years in both my desktop and WWC applications. This library is loaded on demand as needed in the context of a specific process request.

A little background first on my environment; I have two IIS servers deployed in AWS behind an AWS load balancer. Each server has 10 COM instances. When there's an application error I'm using SendAdminEmail to notify a distribution list. Pretty much out of the box WWC-type error handling.

To recover I basically:

• took one server out of the LB using a PShell script from my fine DevOps team
• switched to File mode using the standard WWC administration page
• unloaded the instances
• switched back to COM
• redeployed to the LB
• Rinse and repeat

I realize that switching to File mode was probably unnecessary but better safe than sorry as I was fire-fighting at the time.

From when it first started to the time it stopped was approximately an hour. This morning I queried the wwrequest table and during this interval I counted ~6500 requests. There were ~450 error emails related to the memory error so ~93% worked as normal. After I had a chance to look at some logs it looked like the errors were being generated from one server. This leads me to the theory that one or more instances on that server ended up with a corrupted memory space, and that any requests handled by that instance returned the out of memory error.

So that's the setup to the first question which is:

Is there a recommended automated way to proactively recognize that a COM instance has an unstable/bad application memory space and kill it? Obviously I can add something with the VFP error handling beyond sending me an email when an out of memory condition occurs but how that can be tied to the bad instance is the harder part. And, of course, that assumes my theory is correct that the error was not being thrown by all instances.

Second question: While Craig's library has served me well for decades now, would you recommend refactoring to use the wwRegEx class instead?

TIA

I don't think there's anyway to detect corrupt memory beyond that Windows itself should detect memory overruns and bad heap access errors and hard C5 the application. If that doesn't happen there's nothing in VFP itself that can tell us that something is wrong. And the COM pool can't either.

If there's a memory corruption that can happen either in COM or file mode EXE. After all a COM server is just a FoxPro EXE except it's not loaded by the shell, but through COM. So I don't think there's anything that can be done about this once it's in flight and happening.

As to RegEx - wwRegEx has its own set of problems - it's the old VBScript reg ex engine that's COM based. It works, but since it's based on a system library it's likely very stable. I can't verify how well it works under load - I use it for some things but it's mostly light weight stuff.

I would probably recommend using .NET with regEx expressions now. In fact I probably should look into fixing wwRegEx and use wwdotnetbridge for this. Main issue is that some RegEx might break because the RegEx syntax for VBScript is old and might be slightly different than the .NET/JavaScript (much more standard) versions. Probably worth the risk though, so we can have an up to date reg ex implementation. :smile.

Thanks for the reply. As this happens so rarely I can wait for that wwRegEx refactor. 😃

Alas I had another one of these events last night so I'll be interested in that .Net refactor of wwREgExp sooner rather than later. 😃

I've also got my team chasing me about other ways to capture and log these events to make them alertable and "webby". At the moment I'm just using the basic framework stuff to send an admin error email. Is there a particular place in demos or docs you might point me at so I can do my homework? For example, if I get a VFP out of memory error, to use this particular situation, I should be returning a 5xx error from the request as opposed to a 200 with a sorry we had a problem page. (Starting to feel like the old giant error handling CASE structures where each type of VFP error can have its own response but in the context of WWC.)

To continue the conversation, would you recommend setting VFP memory usage in the context of WWC via SYS(3050)?

To continue the conversation, would you recommend setting VFP memory usage in the context of WWC via SYS(3050)?

Yeah, but just don't set it too small or it will probably make out of memory errors worse. But memory restrictions will VFP behave better and not allocate rampant memory until the system is under stress.

There's not much you can do about memory leaks other than trying to find them. Only way to do that that I know of is log memory usage as you go. VFP has functions that tell you how much memory is using - it actually gets logged in the MemUsed field. But realistically you need something more granular that tracks memory going in and memory coming out.

Maybe this is something that WWWC could even log - but I hate changing the log. This invariably breaks a lot of people...

+++ Rick ---

Years ago, I recall a thread where Christof (I think, although it could have been w00dy) recommended setting foreground and background to ~1/2 GB, and that's what I've tended to do in my dev environment. The WWC admin page rarely shows memory usage going above 60-70MB and the averages hovers around 40MB. With that context, do you have any thoughts on what would be "too small?" Also, what do you think about calling SYS(1104) at end of each process request? The VFP help seems to indicate that function also clears cached memory for things besides tables. Or implementing something like Scott's garbage collection routine at the end of each process request?

I'm also working on a refactor to use the wwRegEx class in place of Craig's FLL. It looks like that will be pretty straightforward for me but I'll be looking forward to the .Net implementation when you decide to get to it. 😃

I put together a little page that echoes memory stats using what I think are the relevant VFP memory functions. (Based on the docs a lot of them are really useless in a post-640k world. Anybody remember EMS?) I loop through it with a little sleep in between. This server has no SYS(3050) constraints. Of course, this is just for this single request. The trick as you said is to figure out what call is causing the memory leak in the first place.

sample output:  
REPORT 1 FOR STG-IES-AWS
Virtual memory pool size is 1,073,741,824 bytes
VFP memory handle count is 2,205
User object memory allocated is 1,838,012 bytes
Foreground memory allocated 1,466,695,680 bytes
Background memory allocated 366,739,456 bytes
REPORT 2 FOR STG-IES-AWS
Virtual memory pool size is 1,073,741,824 bytes
VFP memory handle count is 2,209
User object memory allocated is 1,838,428 bytes
Foreground memory allocated 1,466,695,680 bytes
Background memory allocated 366,739,456 bytes

In the thread with Scott I see you mentioned increasing the frequency of IIS pool recycling and I'm going to investigate that. I believe I've got it at the default 27 (?) hours. OTOH I've found that restarting Windows seems to be the only reliable way to truly clear the problem.

As to more granular memory usage logging, I assume I can sub-class wwServer.LogRequest to track the appropriate stats, or otherwise roll my own.

Thanks again for any insights you have to share.

I'm looking at the wwrequest table for more clues and have noted that all the out of memory errors can be lined up with error = 1 and remoteaddr is Local. Presumably by looking at the last request prior to the first memory error might offer some clues, but without verbose logging mode I can't tell much more. Until I can figure this out I guess I can/should enable level 3. Good or bad idea?

Checking memused, the range of the failed requests are between 46,653,612 and 47,070,784 but I also have successful requests that used more memory.

Yeah the detail logging is meant to be used when you're trouble shooting - now's the time 😄

OTOH not sure if that necessary will reveal more - the main thing that logging will tell you is, which request may be causing a problem. The extra data may or may not be helpful as it's all request data that won't provide any further help.

The full logging tracking is primarily so you can capture the POST data so you can try to 'replay' specific requests potentially in order to get a debuggable scenario.

In the case of memory errors i think you're mainly hunting for patterns in URLs that preceed a failure, or errors that occur around that time.

+++ Rick ---

Am I better off doing something similar to Scott and logging memory use at the beginning and end of each process request to my own logging mechanism instead of relying on wwrequestlog in verbose mode? I'm a little concerned about blowing up that table as I already average well over 100k requests a day during the work week across all my clients.

Also, I think I asked this in that conversation. (Apologies to Scott for thread hijacking...) You showed 2 examples of where to put a SYS(3050) call (OnLoad or wwserver.Init) and my question was; is one strategy preferable to the other?

Also, I think I asked this in that conversation. (Apologies to Scott for thread hijacking...) You showed 2 examples of where to put a SYS(3050) call (OnLoad or wwserver.Init) and my question was; is one strategy preferable to the other?

No - but the latter will be added back into Web Connection. Not sure why that wasn't in there, since there's a config key for it. I suspect I pulled it out in one of the migrations when code went to the internal OnInitInternal/OnLoadInternal calls some time ago.

+++ Rick ---

So control it via the INI when that's released, correct? And I noticed you have the command to set background memory (SYS(3050,2)) but not foreground memory (SYS(3050,1)). Is that intentional?

Yes you don't want to really mess with main memory to limit primary processing usage. What you want to control is how much cached memory VFP is using.

Again - I'm not sure how necessary this is these days where we have machines with 16gb or even more. This will make VFP more aggressive about reducing memory, but if you're leaking memory that's only going to make the problem worse as memory becomes scarce quicker.

What is useful for is limited memory environments or if you're running a lot of instances in the Web Connection pool. If you're running 30 instances keeping memory down might be somewhat useful.

+++ Rick ---

Ah, that makes sense, Rick. Thanks, I will ignore using SYS(3050) for now. So I'm circling back to trying to determine root cause on the memory leak. In the meantime I'm also moving forward with refactoring to use WWRegEx so at least that will eliminate Craig's FLL as the source.

This blew up on me again last night. I've got a lot of angry Aussie clients. This is really puzzling as I don't know what of significance has changed in my app/environment that this is now happening with disturbing frequency.

Just noticed that in my app INI there is an explicit Memusage value which must have come from the default create project script. And it's set to the wwserver default of 8176. FWIW it's been running this way since I launched 5 years ago

I am making my way through that regex refactor to determine if there's a meaningful performance impact. I'm also adding some more garbage collection stuff like sys(1104) and explicit release library calls to my OnProcessComplete method.

It may be consultation time again soon...