Rick – one of my servers is having its app pools crash every hour or so several times a day. It started about a week ago. The fix is to recycle the app pools manually. I set the app pools to recycle automatically every 2 hours 'just to save me' from having to do it manually, but even that is not helping.

At one point, one of the sites on the 'bad' server was not serving any pages (it would hang with no errors) and I did an iisreset /stop and noticed that all but one of the app exe’s would not close from task manager. When I tried to 'end task' the OS fires up multiple instances of the ‘zomby’ instance and does not serve pages. The only way to clean it up is to reboot.

I'd like to hire you to come take a look at the server.

Ping me on skype please when you are avail.

For those lurking, we identified a few issues on the server. THe server was running with a multiple multiple worker processes enabled which resulted in a large number of potential instances getting created. Also because of the way that IIS works this puts a number of edge case issues up for Web Connection that were never tested when operating under the ISAPI dll.

Ultimately the suggestion for improved overall stability was:

• Run with the .NET Module not wc.dll (ISAPI) The .NET module is designed from the ground up for newer versions of IIS and the full feature set of IIS, so it can safely handle many more scenarios than the ISAPI dll. The .NET Module also has many more checks and guards to check for application failures and can recover from many more scenarios without an Application Pool restart than ISAPI. It's not always possible to use the .NET Module - especially in some multi-tenant scenarios (at least not a different design approach).

• Try to keep Applications Isolated Multi-tenant applications often require many domains using the same Web application/server and this can be problematic as you potentially need to run a large pool of COM instances. Web Connection runs best with 1-2 instances per CPU core and having many separate applications that run 1-2 (or even multiples of that) instances is better than having one pool with as many as 20 instances due to the way IIS works with the STA threads required to run FoxPro COM servers.

In Michael's moving to .NET Modules wouldn't be an easy case but with 20-30 sites per physical server it's totally doable to run 3-4 instances per site (on average) given a typical 8+ gig machine set up. Web Connection servers running typical medium data loads tend to clock in around 20megs of memory usage and even 100 instances would amount to only around 2 gigs of memory - so lots of room to spare.

I think for new applications this isn't really a discussion - there are always ways to make multi-tenant work even with many individual sites, but for older sites that weren't designed with that in mind this is a more difficult task to retro-fit to.

There's a bit more info in the Help file re: multi-tenant that you might find helpful:

• Multi-Tenant Web Connectin Applications

+++ Rick ---

In the first 24 hours it 'seems' as if the crashing has stopped. A huge relief if that is the case. As Rick points out, we were assigning multiple worker processes in multiple application pools. A busy site might get as many as TEN instances assigned. As the traffic heated up, IIS would fire up new instances. This strategy was easier to manage visually by less experienced system admins, because they could use the OS UI. We assumed that this would make it possible to 'hard code' n instances to a given site and therefore isolating those instances to that site.

Rick noticed that in our wc.ini we had only a single server and we immediately replaced it with ten and reduced the 'worker instances' to 1 (the default) in each pool.

In case you are wondering, the site is a marketplace of web stores. Each seller has a sub domain. The WWWC application is setup in multi tenant fashion, but it is sort of like MySpace vs Facebook in its config. The marketplace allows each seller to have their own design, and their own business rules (MySpace) as opposed to a fixed UI for each seller and a fixed business rule set (Facebook).

Thanks Rick for being there 'almost on demand'. The time saved paying you for support is well worth it. Long live West-Wind!

Hi guys,

Thanks for sharing your findings. I'm sure it's appreciated by many.