Ok, I think I might know what caused most of the errors, well most of the errors on the 2003 box. Just before the time we started getting this errors we had an email problem. This problem was more of a global problem that made it impossible to send emails via TCP. To rectify the problem I had to edit the metabase data. I did this with IIS Metabase Explorer.
I had to add under LM/SMTPSvc/1 a dword, value 36997 (SmtpUseTcpDns) with a value of either 2 or three. These settings made it possible to send mail. I could not get this key added to the 2000Box and had to route the mail through the 2003 box.
I now changed the setting back to 1 (Tcp). Its now been 3 days and not once I had to restart IIS.
The 2000 box is still giving some problems but seems to be more related to einter ftp users not deleted where the folder has been deleted or the site configured to still run also where the folder does not exist.
Regards
Willie
Willie,
You can tell based on the URL in the error log what requests are failing. Maybe you can duplicate the environment or retry the request with that same info? I suspect though the issue may be the POST data from SPAM bot or something along those lines.
With 500 sites it might become difficult to pinpoint problems though I can see that :-)
+++ Rick ---
<s> yes, the box is about 10 years now, with more than 500 active websites. I dont think it is hardware although nothing is impossible. Problem is if I get a new box it means that I have to redo all of those 500 + domains as well as all other data related on spesific places. The other problem is that the 2003 box is also still "sick".
Fidler, all the 500 + sites is running WC and there is a lot of traffic. The IIS problem do not occur regularly so to trap those is basically impossible.
Must say, last night went much better, only had it falling over once.
I guess if you're getting these input data can't be parsed errors it'd be nice to capture the input data (ie. the POST data and querystring values/headers etc) with something like Fiddler and replay them to see if the failure is repeatable.
If it's not then I would suspect some hardware/software issue with the server itself (bad memory maybe)... If it's a Win2000 server the box has got to be well over 10 years old <s>...
+++ Rick ---
:) Yes it is some crazy stuff but it is working, all got to do with otimazation. It is not that though. been there for over two years.
No not consistent. Today I removed .Net framework 2 on the 2000 box bec it was one of the things we have added receintly, before the shit started. Had IIS falling over 4 times after that so it is not that too.
This is, IIS does not stop totally, well, not in the begining but then evetialy no websites is working and all that remoains is to reset IIS.
The way I see it is I have one of 2 options, 1. delete every IIS entry and redo it or 2. buy a new server with Win 2008. Problem with option 2 is that it is expensive and not planed nor in the budget since a new box will be in the range of $9,000
Can you try to find a consistent repro? It looks like your query string has some crazy stuff in it there. Is that consistently failing?
+++ Rick ---
Hi Rick
I am still batling with the IIS on both 2000 and 2003.
Here is an extract of one of the WC error log files on the 2000 box
Started here:
2010.08.1 21:14:23:093 - n/a - 2948_BF8AF58CAB09 - An exception occurred in WC.DLL: &Display=20&Sortby=4&PropTypeSel=Vacant%20Land&Prov=gauteng&city=&searchstr=randburg&Page=2
Unloading all servers and reloading... - /Property_for_sale_*_South_Africa_**_gauteng.prs?&Display=20&Sortby=4&PropTypeSel=Vacant%20Land&Prov=gauteng&city=&searchstr=randburg&Page=2 - 0 - 2948 - Com:
2010.08.1 21:14:23:359 - n/a - An error occurred writing form/server variable: - n/a - 122 - 2820 - Com:
2010.08.1 21:14:23:359 - n/a - An error occurred writing form/server variable: - n/a - 122 - 2820 - Com:
2010.08.1 21:14:23:359 - n/a - An error occurred writing form/server variable: - n/a - 183 - 2820 - Com:
2010.08.1 21:14:23:359 - n/a - An error occurred writing form/server variable: - n/a - 1413 - 2820 - Com:
2010.08.1 21:14:23:359 - n/a - An error occurred writing form/server variable: - n/a - 183 - 2820 - Com:
2010.08.1 21:14:23:375 - n/a - Error reading server variables from Web server - n/a - 259 - 2820 - Com:
2010.08.1 21:14:23:375 - n/a - 2820_5852E0082A44 - An exception occurred in WC.DLL:
Unloading all servers and reloading... - /Property_to_Rent_*_south_africa_**_gauteng0--edenvale---greenstone_hill.prs? - 0 - 2820 - Com:
2010.08.1 21:14:26:093 - n/a - Error reading server variables from Web server - n/a - 87 - 2964 - Com:
2010.08.1 21:14:26:218 - n/a - An error occurred writing form/server variable: - n/a - 87 - 3432 - Com:
2010.08.1 21:14:26:218 - n/a - Error reading server variables from Web server - n/a - 259 - 3432 - Com:
2010.08.1 21:14:26:218 - n/a - 3432_97422FDF5213 - An exception occurred in WC.DLL: &searchid=Fast
Unloading all servers and reloading... - /showpropertySC2012555395.cp?&searchid=Fast - 0 - 3432 - Com:
2010.08.1 21:14:26:343 - n/a - An error occurred writing form/server variable: - n/a - 1413 - 3600 - Com:
2010.08.1 21:14:26:343 - n/a - Error reading server variables from Web server - n/a - 259 - 3600 - Com:
2010.08.1 21:14:26:390 - n/a - An error occurred writing form/server variable: - n/a - 1413 - 5328 - Com:
2010.08.1 21:14:26:390 - n/a - An error occurred writing form/server variable: - n/a - 122 - 5328 - Com:
2010.08.1 21:14:26:390 - n/a - An error occurred writing form/server variable: - n/a - 1413 - 5328 - Com:
2010.08.1 21:14:26:390 - n/a - An error occurred writing form/server variable: - n/a - 183 - 5328 - Com:
2010.08.1 21:14:26:390 - n/a - Error reading server variables from Web server - n/a - 259 - 5328 - Com:
This caries on until I restart IIS.
I am now at a point where I either need to solve the problem or replace the box and load 2008. Moneywise the first option will be prefered.
Willie
I'm not sure how to help you frankly since this isn't something I've seen or can duplicate. File based mode operation is so basic that it's hard to see what could be crashing things.
Another option on Server 2003 is to use the .NET managed module and see if that fares better. This requires you set up ASP.NET on the box/virtual and the Web.Config file. It's an alternative that completely bypasses the ISAPI extension in favor of a .NET component. Obscure errors like what you're seeing are much less likely with the .NET extension, but as I said this behavior you describe is not something that I've seen before or have even heard of before...
s
+++ Rick ---
Hi Rick
Ok, after replacing the wc.dll on the Win2k server it is 100% stable. We had no crashes today on that server. The 2003 box though is a different story. It crashes regularly.
What did help today is when I changed the application pool settings, Recycle worker process (in minutes), changed that to 60 and the next option, Recycle worker process (number of requests) to 15,000. IIS do however still keep crashing.
Corruption: When all this started I did check the domain and ftp entries, check to see if the folders exist as a number of errors were pointing to invalid folders. I did found numerous entries but fixed them all. What I also did is to update Foxpro to service pack 2 as ADDPROPERTY gave an error.
Is there any tools available you know of I can use to do some testing or logging? I really need to sort this out as I have a few very unhappy customers.
Willie,
I'm not sure what to tell you honestly as I've never seen this behavior before. I'm not sure what would be so different between versions to cause this epecially on straight through requests. Maybe there's some specialized case somewhere with some data that causes corruption specifically?
+++ RIck ---
We are running file mode. We have switched back to a WC.dll in version 5.25. Had to replace all the DLL's. On both, Win2k and 2003 it is at least a 1000 times beater. We do still get errors on both but by far not as bad. On the 2003 the following error occur about once every 4-5 hours:
Event Type: Information
Event Source: Application Popup
Event Category: None
Event ID: 26
Date: 11/07/2010
Time: 11:07:48 PM
User: N/A
Computer: XAVIER
Description:
Application popup: w3wp.exe - Application Error : The instruction at "0x7c82a774" referenced memory at "0xc7ecc2b4". The memory could not be "read".
Click on OK to terminate the program
Click on CANCEL to debug the program
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
After this error the following normally follow:
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1010
Date: 11/07/2010
Time: 11:22:51 PM
User: N/A
Computer: XAVIER
Description:
A process serving application pool 'DefaultAppPool' failed to respond to a ping. The process id was '7176'.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
When this happens some websites are unavailable. When I refresh the application pool it seems to be fine again.
Like I said, this is now much better. We were getting these errors sometimes more than 10 times an hour.
I suspect there might still be a new WC.DLL file somewhere that is causing the occasional errors.
On win 2000 box the errors has dropped to about 3 in 12 hours from also about 10 per hour. I did found a new wc.dll file on one of the folders which I now replaced as well so hopefully this box will now run without any IIS errors. The following is a typical error on the W2k box.
Event Type: Error
Event Source: WAM
Event Category: None
Event ID: 204
Date: 11/07/2010
Time: 21:44:58
User: N/A
Computer: MONALISA
Description:
The HTTP server encountered an unhandled exception while processing the ISAPI Application '
ntdll!RtlFreeHeap + 0x227
wc + 0xB0E9
wc + 0xAD25
wam!DllCanUnloadNow + 0x636
wam!DllCanUnloadNow + 0x20C
w3svc!HTTP_HEADERS::FindValue(char const *,unsigned long *) + 0xE2
w3svc!STR::Copy(char const *,unsigned long) + 0xC71
w3svc!STR::Copy(char const *,unsigned long) + 0xB49
w3svc!STR::Copy(char const *,unsigned long) + 0x9A2
w3svc!CLIENT_CONN::OnSessionStartup(int *,void *,unsigned long,int) + 0x642
w3svc!HTTP_HEADERS::Reset(void) + 0x1CA
w3svc!STR::Copy(char const *,unsigned long) + 0x16EF
ISATQ!CDirMonitor::RemoveEntry(class CDirMonitorEntry *) + 0x13A
+ 0xECA5F167
'.
For additional information specific to this message please visit the Microsoft Online Support site located at: http://www.microsoft.com/contentredirect.asp.
We also have another W2k server with only about 10 sites configured. This server is running fine with the new DLL file. The other two servers run over 1000 websites between the two.
Not sure. We have lots of customers running on Win2003 R2 without issues, so I'm positive the DLL itself is fine. If you're runnning in COM mode try running in File Mode to see if the issue goes away. If you're running in COM mode and there's a problem it might be server COM permission issues.
+++ Rick ---
HI Rick,
Looking at event logs i get the following on the server
Source:WAM
The HTTP server encountered an unhandled exception while processing the ISAPI Application '
wc + 0x11100
wc + 0x187E
+ 0xB11AF4E0
'.
So i believe it is the wc.dll causing the crash.
Running Windows server 2003 and iis 5
Any help fixing this error would be appreciated
Regards
Robert
Hi
We recently upgraded from 5.3 to 5.51. I have replaced all the wc.dll (for those sites running on the new version) as well as wwipstuff.dll files. I have 2 boxes, the one with Windows 2000 SP4 and the other Windows 2003 SP 2.
After the update we are experiencing IIS crashes on both servers. On Windows 2003 box where a website was working 100% we get HTTP 500 error in IE and "A dynamic link library (DLL) initialization routine failed." in Chrome browser. We then have to restart IIS in order to fix the problem.
Any idea where to start looking?
Regards
Willie van Schalkwyk
http://www.cyberprop.com
