ICUHost.Net: News

Hello news file, been a while. Power outage 6/19/2009 ~4:00pm until next day ~12:30pm. Systems brought back up by phone. MySQL server needed massaging. Most main services restored after the power was back on, but other problems did creep up. A Cisco 2924 didn't make it and went faulty within 2 days. A replacement was bought offa eBay and installed yesterday. Originally ~$300.00, replacement was $25 so bought 3. The FreeBSD servers were affected and all uptimes were zeroed.

Upgrades to a Windows server yesterday from PHP4 to PHP5 resulted in unexpected hangs of the web service. The PHP5 installation MSI was used to make the upgrade. It is suspected that this is the culprit, so today the installation was removed and then performed manually. If the hangs persist, we'll be forced to remove the open source tool from the proprietary monster. Anyone residing on a Windows server that doesn't use any Windows technologies is advised to switch to a FreeBSD web server running Apache. The TTL of IIS isn't looking up.

9:44-49am : Rebooted FreeBSD Firewall - Kernel upgrade. Network unavailable during those 5 minutes.

9:15am : Rebooted FreeBSD Mail Server - Kernel upgrade.

A Windows server containing many sites failed. The cause was the display adapter overheating/halting the server. The video card was replaced and the server has been running for hours.

Horrible mail server failure. <= It gets its own page.

Transfer of the network from Savvis to Cogent took place overnight. The connections co-existed throughout last night and this morning. Once rDNS was in place, the change could take place immediately. At 8:08am the cable connecting us to Savvis was removed. No downtime should have been noticed, but there is sometimes the residual access at the precise moment an IP address is changing. Also, some registrars are not too quick on propogating their networks. We utilized the tools at DNSstuff.Com as well as another remote site and Cogent customer support. We apologize for any outages you may have experienced.

The virus detection component to the mail server failed late Tuesday night causing mail flow to cease but the mail server simply reported errors to those attempting to send mail. It was reported and fixed mid-afternoon today.

Firewall upgraded 06:04-06:32am. All traffic stopped for this period.

Power outage 10:35-11:30am.

Major upgrades across the board and no, they're not finished. The latest openssl port has been known to break things, so a regression to openssl-stable is suggested. Of all things, the major impact was on the mail server since it is heavily integrated in all those applications. This past Sunday night's update ran into Monday morning and inconvenienced some with mail delivery. Those with forwards or SquirrelMail access were running easily. POP3 was hindered and then POP3's daemon was changed for the better Monday night. When upgrading the mail server, preconfigured files shut it up tight and Spamhaus was limiting some to send mail. Spamhaus PBL was removed Tuesday night. FreeBSD 4.x machines are not able to run the latest and greatest PHP so some sites failed and had to be moved. This process is ongoing. Any issues, please email us.

Windows Server transfer complete. Please let us know of any issues.

The Windows server continues to fail. After replacing all parts and testing on those individual parts, the machine continues to randomly boot. No messages, no indications as to why. The machine was disassembled and reassembled down to the heat sink on the CPU. The resolution will be to move everything from this server to another server. Being that this is Windows, this makes this task extremely difficult. Throughout the course of the morning, all files from the current server will be moved to a new one. When it is complete, this news will be updated.

It has come to our attention that the main Windows server has been bouncing. Currently, the cause is unknown. A quick MEMtest was run this evening, but we didn't let it run to completion since sites are active on this machine. The initial test showed memory operating as expected. The power supply has just been replaced even though the current power supply is a newer one that seems to be functioning just fine. No one reported any problems with this server and that seems strange since its activity is the highest on the Windows platform. In contrast, FreeBSD servers report nightly their status and we're informed of impending doom. The logs are extremely helpful as well. In the case of this Windows server, there are no notifications and there is nothing in the Event Log except for "The previous shutdown was unexpected" ... Well, thanks.
UPDATE: It was not the power supply, so the original went back in. We pulled the machine from the rack and it is now sitting on our bench. More symptoms were found, and another avenue has been pursued. We will keep close watch on this machine and let you know the final resolution. We apologize for any inconvenience this may cause.

An upgrade to the FreeBSD mail server went longer than expected yesterday morning. Over 200 programs were evaluated and upgraded if necessary. Some programs needed to be upgraded due to reported vulnerabilities. While vulnerabilities are reported, it's not always necessary to upgrade the program immediately. What put a snag in this upgrade was that it wasn't automated therefor needing constant interaction, and a very important program would not compile. For most of the day yesterday, mail delivery stopped working. Much like the Eagles loss after a 17 point lead, we learned a lot from this incident. Sincere apologies to those that can't live without their email, especially on a Monday. Know that the mail server is running better than it was Sunday. All new programs in place to better serve you. A SPAM and Virus free inbox is the goal, and you have all the tools available to you at no additional cost to make that happen. Thanks.

Happy 4th! DNS was reloaded this morning to switch back to a database backend. In the midst of the AXFR transfers, NS records were not added to zone files making them not resolve. A script was written to load each zone domain and add the necessary NS records. This was completed by 8:15am EST.

About 4:00am heavy rains in the Philadelphia area led to ~4:30am rain water finding its way to the SmartJack. Third time is a charm and they finally moved it. Replaced and back up approximately 9:30am. Intermittent downtime when SmartJack was moved and intrusively tested.

The primary DNS Server failed Thursday 6/22/06 at approximately noon-time. While TTLs expired, the main effects were noticed on email and that's where the reports came in. Focused on email initially, it wasn't realized until poking into the mail server logs that DNS was the ultimate cause. Rebuilding a primary DNS name server would have taken too long so the secondary was aliased as the primary and resolution IP addresses were changed to get things back to "normal". The primary DNS server was rebuilt yesterday afternoon into the evening and put in place. The secondary was returned to its normal duties. These DNS servers were built in April of 2004.

Residual from the power supply failure left some things in limbo. For one, the stats hit a corrupted file that needed to be removed so a proper stats run could complete. Also, some mail files were corrupted at the same time leaving messages to the console to the affect of their corruption. Windows can mess up a file while the system is live, but it cannot fix them when the system is live. A CHKDSK that used to take a couple minutes back in the day runs much longer now due to the file sizes of new. It was unexpected that this would bring the system to a halt for so long this afternoon. Finding/Fixing corrupted files didn't take long at all. It was the checking of the indexes that took a lot of time. This did not affect any FreeBSD servers.

A power supply was replaced this morning at 10:00am in one of the Windows Servers. It had failed intermittently at 3am and finally 4am. System maintenance was applied this morning at 10:30am ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers.

The FreeBSD Mail server has been upgraded to the latest and greatest.

System maintenance was applied this morning at 1:00am ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers. The M4P MIME type is available on the IIS Servers now.

System maintenance was applied this evening at 6:15pm ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers.

System maintenance was applied this morning at 10:55am ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers. .NET Framework 2.0 was installed as well.

Firewall coldboot took all systems offline approximately 14:51 to 14:59. Certainly not planned but necessary.

FTP PASV (passive FTP) should be possible now due to an update to the firewall.

Firewall machine ceased into being. All traffic nailed to a stop. Temporary measures in place.

Many firewall changes throughout the night starting approximately 10pm lasting until 5:30am. Each firewall change required a reset of the rules which sometimes takes about 5 minutes to settle in. These changes included new rules as well as some bandwidth throttling imposed. If these changes affected you throughout the evening, we apologize for any inconvenience.

System maintenance was applied this afternoon at 4:15pm ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers.

System maintenance was applied this morning at 1:45am ET to the Windows Servers. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers.

FreeBSD Mail server sync'd for SCSI HDD swap 4:45pm. Cycle had immediate authetication problem due to permissions which was fixed within minutes.

6:00pm smartjack fry. 7:45pm called Savvis, escalated. 9:07pm Verizon called dispatching tech. 9:50pm Savvis called; 7:36pm CT Verizon dispatch. 11:40pm Savvis called, no update. 12:50am Verizon on site. 12:55am connection restored. Multiple users reported multiple outages on various networks through the course of yesterday. 'twas not a good Internet day.

Savvis had problems in the Philadelphia area. Their Master Ticket # is 465020. Our logs of the interface showed the following:
Jun 14 13:54:59 gw 41637: 5d18h: %LINK-3-UPDOWN: Interface Serial0, changed state to down
Jun 14 14:33:01 gw 41643: 5d18h: %LINK-3-UPDOWN: Interface Serial0, changed state to up
Jun 14 14:33:19 gw 41645: 5d18h: %LINK-3-UPDOWN: Interface Serial0, changed state to down
Jun 14 14:46:17 gw 41647: 5d19h: %LINK-3-UPDOWN: Interface Serial0, changed state to up
Jun 14 14:49:06 gw 41650: 5d19h: %LINK-3-UPDOWN: Interface Serial0, changed state to down
Jun 14 15:01:29 gw 41651: 5d19h: %LINK-3-UPDOWN: Interface Serial0, changed state to up

FreeBSD Mail server cycled for SCSI HDD install 9:40-9:55pm.

Power outage 6:58-7:45pm.

System maintenance was applied this afternoon at 3:30pm ET to the Windows Domain controller. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect any FreeBSD servers.

System maintenance was applied this afternoon at 4:30pm ET to the main Windows server. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

Updated the support machine and got it working properly again since the Perl upgrade. SuSE works a bit different than FreeBSD and it took a little while to figure this out. Also moved the FAQ to the OTRS application and the link has changed on the main site. Also added the Online Support link so that peope can use it if they prefer web based support to email support ... which is basically about the same thing in regard to the OTRS system. If you have any concerns, questions or suggestions, don't hesitate to contact us.

System maintenance was applied this morning at 5:00am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

FreeBSD mail server upgrades taking place this afternoon. Some services may be quirky for short periods of time.

An unexpected system shutdown (a 6 second power outage) occurred at 3:53:20PM. The router did not recover well and needed to be restarted. Because the router failed, several systems ended up with an identity crisis not knowing who they were since they couldn't resolve. This took a little time to get everything in sync. The last to be rectified was the FreeBSD mail server at 16:29:21. We apologize for any inconvenience.

All servers have been upgraded to PHP 4.3.10 and Zend Optimizer 2.5.7 where applicable. Also most versions of phpBB2 have been upgraded to 2.0.11, again where applicable. If you have any issues, please contact us immediately.
Merry Christmas!

System maintenance was applied this morning at 6:25am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

Upgrades: Cacti, VegaDNS ... Some FreeBSD machines. Game server HD. :: Because of the unavailability of the game server, this caused the stats to backup. The 3 programs that queried the game server from the stats gathering machine weren't timing out, but rather filling the machine's memory until it just crapped out on all new processes. You can see this in the publicly available status graphs. Now that that machine is back in place and responding to SNMP queries, everything is A-okay again. In the process, the couple programs got updated and we finally got to upgrading VegaDNS.

System maintenance was applied this morning at 5:35am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

When it rains, it pours ... literally. Yesterday had to be the worst in history. After the outages, the Smart Jack fried. This was approximately 9pm Sunday evening. The first call for service was determined to be the next AM so we had to wait for it to be replaced. No servers were down (as the emails like to say), it was the connection and beyond our control. Frustration. Service man just left at 9:35AM and said "You're back." A "card blew" in the smart jack. Wish they'd leave spares.

We had two power outages this morning. The first lasted about 1 second. That was enough to fry the power supply in the main Windows server. Right after the power supply was replaced, another power outage. The second outage lasted approximately 30 minutes or so. If you experienced connectivity issues at this time, this is the reason and we apologize for any inconvenience. There goes our year and a half uptime ... =( ... darn that PECO ...

cs-server :: 3:01AM up 473 days, 18:20, 5 users, load averages: 0.08, 0.02, 0.01
stats :: 3:01AM up 437 days, 21:40, 1 user, load averages: 1.68, 1.01, 0.83
web001 :: 3:01AM up 437 days, 22:34, 1 user, load averages: 0.00, 0.00, 0.00
design01 :: 3:01AM up 436 days, 10:02, 1 user, load averages: 0.00, 0.00, 0.00
cswombat :: 3:01AM up 392 days, 22:46, 1 user, load averages: 0.00, 0.00, 0.00
mysql :: 3:01AM up 214 days, 22:19, 1 user, load averages: 0.00, 0.00, 0.00
host120 :: 3:01AM up 209 days, 6:47, 1 user, load averages: 0.00, 0.00, 0.00
host121 :: 3:01AM up 209 days, 6:50, 1 user, load averages: 0.00, 0.00, 0.00
ns1 :: 3:01AM up 167 days, 8:30, 1 user, load averages: 0.00, 0.00, 0.00
ns2 :: 3:01AM up 167 days, 8:11, 1 user, load averages: 0.00, 0.00, 0.00
host12 :: 3:01AM up 85 days, 8:03, 1 user, load averages: 0.00, 0.00, 0.00
host10 :: 3:01AM up 84 days, 12:57, 1 user, load averages: 0.00, 0.00, 0.00
mail :: 3:01AM up 75 days, 9:19, 1 user, load averages: 0.22, 0.14, 0.09

The router uptime was substantially longer, but that time isn't logged anywhere. This is just a sampling to show some of the FreeBSD uptimes. The Windows machines are booted often enough to apply maintenance (read: bug fixes) so uptime isn't anything to brag about.

System maintenance was applied this morning at 4:50am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

System maintenance was applied this morning at 4:45am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

After replacing the RAM in the mail server and testing each chip individually, it was found that a NEC 256M chip indeed did go bad. This morning all residual fallout from the upsets and crashes of various instances of different mail serving programs has been cleaned out and mail is flowing as usual. Again, we apologize for any inconvenience this may have caused you.

Testing this early AM showed RAM errors on the mail server. It was a comprehensive test, and if you're interested there is a FREE product called MEMtest86 where you burn a bootable ISO and it boots directly into memory testing. One pass depending on processor speed takes 15-20 minutes. Anyway, the mail server will be taken down tomorrow morning approximately 4 or 5am to replace its RAM. We're sorry that it took this long to diagnose, but the problem was attempting to fix the mail server without taking it down, and a hardware test is a hard down. It came to that point where we had to do it, and this showed the error.

We have been having some issues with the FreeBSD mail server that started with a corrupted virus database. That led to upgrading that virus software, and then that started crapping out. While upgrading some dependencies that this new release didn't document, the machine quit mid compile. This led to more and more clean up. If you've been affected by this inconvenience, we're certainly sorry, but this is not the norm of the mail server. You've surely come to realize its stability for almost a year now, and this has been the first major issue. Even while fixing it, mail flowed.

The appliance we were attempting to insert into our connection finally behaved as expected, so the connection will no longer be interrupted by our hands. Thank you for your patience.

Approximately 5:45 to 6:30am and 3:45 to 4:00pm we were messing with the connection so it was up and down at times. We apologize for any inconvenience this may have caused. This is to better the connection. This time it worked. We will keep you posted on the progress.

Approximately 6:00 to 7:00am we were messing with the connection so it was up and down at times. We apologize for any inconvenience this may have caused. This is to better the connection. We will keep you posted on the progress.

Approximately 6:45am we experienced connectivity problems. The added bonus of moving the DNS Servers to FreeBSD servers is that we actually take the same route as the rest of the Internet to resolve host names. So, when there is an outage, we can't even work locally. A call was placed into Savvis and Nick was happy to help us with our issues. Yes, there is maintenance going on. They are replacing some timing modules in the core router that is affecting us. Are we familiar with their weekly maintenance window every Saturday AM 5-9 EST? Well, no. We have never been affected before as far as connectivity goes every Saturday morning. I guess we were just the luck of the draw area this week. Service was restored a little after 7am.

Savvis performed emergency service to a core router that went out approximately 2:30am CST. From the graphs it looks as if connectivity was rerouted until between 5 and 6am and then connectivity ceased. Service was available approximately 7am.

The Windows DNS Servers had their ports closed to queries shortly after the FreeBSD DNS Servers were put in place. This morning we removed all of the entries from the Microsoft DNS servers. This should affect nothing, but there's always a remote chance that it could. If any issues arise, please inform us. Also, the DNS Servers that the Microsoft Mail servers point to are now the new FreeBSD servers. There shouldn't be any interruption or change in the flow of mail. Switch as soon as you can to the FreeBSD Mail server! Any questions, email us.

In reaching our goal for 100% stability, the DNS Servers for the network have been upgraded to FreeBSD Servers. The Name Servers link on this web site reflects this change. Any resellers that need the IP addresses explicitly, please contact us directly. The Microsoft DNS Servers will remain active while propogation progresses, although we have already seen much of the activity switch to the new servers. The decision to go this route came once we realized that MSFT DNS Servers are not RFC compliant, are easily hacked and go down frequently with operating system restarts either unexpected or for frequent maintenance to patch other bugs. FreeBSD Servers never go down. Any questions, email us.

The FreeBSD mail server went through a series of upgrades last night starting at approximately 7pm and running until about 12:30am EDT. During this time there may have been some quirkiness in the mail flow. While no mail was denied its destination, some people may have experienced the inability to send mail due to being on an RBL list. Mostly dnsbl.sorbs.net which blocks Comcast and Verizon dynamic IP ranges. While this is the "right thing to do", it prohibits connections from these networks. That RBL check has been removed and normal operation for those of you affected by this continues. The upgrades and tweaking done last night will enable features that can bring this RBL back into the list of RBLs checked. These RBLs block the main source of SPAM and malicious virus laden emails their entry onto the Internet. While you may have been inconvenienced not being able to send mail, it is in the long run not only for your protection, but for the amount of SPAM and viruses to cease getting into your mailbox. When we can sufficiently educate you and your users how to authenticate with the mail server with these blocks in place, we will enable all RBLs and lock the mail server up tight. Any questions, email us.

You may have experienced outages this morning due to network maintenance. We were informed of this maintenance earlier in the week from Savvis (as we always are) and they haven't always affected us. This morning we noticed a brief outage for about 2-3 minutes so figured we should post this for those concerned. As always maintenance is performed on off peak hours.

While in the server room 6:45PM last night working on a new rack, heard the tell tale sign of a boot, a floppy seek. Turns out it was, surprise, one of the Windows servers and a pesky bugcheck. This plagues the 2 Windows DNS servers, and if an IP address isn't cached, all DNS requests will fail for the time it takes that machine to boot as well as the web sites on those Windows machines are inaccessible. We are going to move all DNS services to FreeBSD machines. When this happens, we are also going to work on a customer interface (possibly only available to resellers) to enable you to change your own DNS parameters. We apologize for any inconvenience this may have caused. But again, we're making excuses for Microsoft ...

System maintenance was applied this morning at 2:35am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

The php.ini file was changed on a web server where magic_quotes_gpc was set to Off ... Setting it to On rectified some PHP issues. If you have any concerns, please email us.

The FreeBSD Mail Server has been updated over the course of the last week. The major update happened last night when the box was upgraded from FreeBSD 4.8 to FreeBSD 4.9. A lot of the internal workings of the mail server have been updated to allow for the better processing of email and the better recognition if an RBL is unavailable making mail deliveries slow. This also affects web pages that are trying to send an email from the web server. The mail server now "pings" the RBLs to make sure they are responding and if they don't, they are removed from checking. ClamAV has been upgraded to the latest release. The FreeBSD kernel has been optimized for better performance. Memory utilization, etc ... Initially after the upgrades, there were some issues with Comcast users unable to send mail. This is because the residential block of Comcast has been put on an RBL list. Essentially, this RBL has determined that Comcast users should be using their ISP's SMTP server and not connecting directly to one outside of their network. Mindspring has done this by blocking port 25 on their network. Realistically, this is the proper way to do things to cut down on unauthorized email, but it does cut into the virtual hosting environment. The way around this is through SMTP-AUTH which the Mail server does support, but is sometimes difficult to understand for the email impaired. (Basically you POP before you send). If you want to do SMTP-TLS or POP3-TLS, you will have to use the actual name of the mail server as that is how the certificate is signed. You can still use SSL IMAP via the https:// webmail interface. If you have any questions regarding the mail server, don't hesitate to shoot us an email.

System maintenance was applied this morning at 6:05am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

System maintenance was applied this morning at 4:00am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

System maintenance was applied this morning at 5:30am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the FreeBSD servers.

Memory added to the mail server approximately 12:30pm ET. You may have experienced inaccessibilty lasting no longer than 5 minutes.

System maintenance was applied this afternoon at 4:20pm ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux or FreeBSD servers.

System maintenance was applied this afternoon at 2:45pm ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux or FreeBSD servers.

System maintenance was applied this morning at 4:20am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux or FreeBSD servers.

System maintenance was applied this morning at 4:30am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux or FreeBSD servers.

One particular web server stopped responding to the Internet this morning. The first report of this came in at 8:44:30AM. The web server was back in service at 11:13:40AM. This affected anyone served by 64.243.181.157. There was nothing to suggest a problem internally. When we first approached the network, we were able to access all sites. We then went to a proxy site and couldn't get to the sites on this Windows web server. KVMing over to that screen showed no pop-ups or apparent problems. We did a WindowsUpdate which had 3 patches to apply, and booted. This made the sites accessible from the outside. In looking at the EventLog afterward, we found that this box was last booted April 25th. Guess the Windows boxes need booting more often than this, seems they should be booted at least once a month. Hope this wasn't a major inconvenience, we have no one to blame except for the Windows Server, and it's not pointing the finger anywhere either. The resolution to the problem was simply booting that server. The connection was fine and other servers were serving outside requests.

Changed a router last night around midnight. You should have expereienced inaccessibilty lasting no longer than 5 minutes. This router may not be permanent.

System maintenance was applied this morning at 7:05am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux or FreeBSD servers.

The latest Critical Update was pretty vague. We were seeing some weird network activity so we did a rare afternoon apply of the maintenance. Hope this didn't affect anyone too bad as it's understandable a midday outage (DNS failure) can throw the world into a frenzy. We apologize for any inconvenience and assure you that no outage lasted more than a restart which we always say is no longer than 5 minutes. This did not affect any Linux or FreeBSD servers.

Systems were down this morning due to some moving around. During this process, the router's power was removed so our uptime has changed.

Of course there have been updates that haven't made it to this page simply due to lack of time. But this had to go up immediately. We just got a phone call from Savvis. They informed us that they lost IP connectivity to us at approximately 10:45a (Don't know if that was EST or CST, forgot to ask). We noticed no outage, but they opened a ticket and will monitor it for 24 hours. The outage was said to have been for less than a minute. Just to demonstrate the difference in our new connection; The cable to Fast.Net was physically unplugged at 9:45a last Tuesday. (Mar 5 09:46:33.092 EST: %LINK-3-UPDOWN: Interface Serial0, changed state to down) We got Jason's concerned call from Fast.Net at 10:20a. It only took them 35 minutes. So, guess there really are providers out there that will see your connection hiccup before you will. Every outage with Fast.Net we called them. Well, except for the last one which will never be restored. We normally knew when the connection went down due to game server connectivity or Instant Messenger services. Nothing here caused alarm at all.

Gateway changes were started at approximately 5:45pm ET and should take no longer than 20-30 minutes. There should have been no disruption of service.

System maintenance was applied this morning at 5:45am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux servers.

System maintenance was applied this morning at 3:45am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux servers.

Approximately 3:20PM there was a line outage that lasted about 4-5 minutes. Of course we called them first cause they will know our line is down before we will, but that never seems to be the case. We were told that it was the "emergency reboot of a router" ...

System maintenance was applied this morning at 5:10am ET. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux servers.

System maintenance was applied this morning at 6:00am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux servers.

System maintenance was applied this morning at 6:30am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. This did not affect the Linux servers.

SCHEDULED OUTAGE: The line is scheduled to be stress tested again at 4:00am Thursday, October 3rd. An outage should occur not to last more than one hour. We're told it will take 30 minutes to an hour to complete the test. This is to test maintenance that will be applied at 2:00am, but no outage will occur at 2:00am. Rather than waste a day to see the results of the 2:00am maintenance, we suggested they go right into a stress test after the maintenance is applied.

Results of the first test showed packet loss prompting the router change. After this maintenance was applied and the results came back, the line is reported squeaky clean. These tests are in relation to the high pings we're seeing on the UDP based game server. HTTP, SMTP and FTP seem to be unaffected because of their error correcting protocols that will make the world spin on your browser, an email or a file transfer take possibly one second longer. On the game server, a second could mean life or death .. hehe. But, what is a concern is that if these seconds are added up over the many web sites and SMTP/FTP activity going on, overall we feel it is a major issue to be addressed. If you have any concerns, please email us.

SCHEDULED OUTAGE: The line is scheduled to be stress tested at 4:00am Tuesday, October 1st. An outage should occur not to last more than one hour. We're told it will take 30 minutes to an hour to complete the test.

System maintenance was applied this morning. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

An application called from a web page hung that server's W3SVC. A restart was necessary that may have affected other web servers on the network. This was done at a time thought best that most people would be in transit from work to home. (approximately 5:30pm) A mid-day restart was unavoidable. We apologize for any inconvenience this may have caused.

After getting the bright idea of doing a traceroute directly to our provider on a connection also based out of them (Princeton University), we saw latency on that connection as well. Someone working on a Saturday duplicated these results almost immediately and they have finally opened a ticket and have realized that there is a problem. We have offered to help them in any way we can.

In our ongoing problem with our upstream provider, scripts were written to ping 206.245.170.9 (Fast.Net) and 63.237.65.78 (QWest) every second. When we reached 100 excessive pings, an email was sent and CC'd to us. Shortly after this began, our upstream asked us to cease sending the automated emails. Even though these pings showed the problems that exist at these two IP addresses, they are again suggesting that we stress test the line with the Telco even though we already performed this test and have proven that it isn't the local loop. We have even had people from around the country perform traceroutes and send them in showing the problems at these two IP addresses. We will keep you posted as to the results of these tests and investigation into these addresses. Of course things such as HTTP, FTP and SMTP (which is what most of you use) will not see any considerable loss due to the fact that these protocols have automatic retries built into the protocol whereas UDP and ICMP pings will be the ones to suffer.

Power outage from approximately midnight to 5:00am.

System maintenance was applied this morning at 5:30am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

Spoke further with provider about outage. They say the local loop people had to contact Verizon. Verizon reported that the line was okay, but after their results, the line magically became active. We are left to draw our own conclusions.

Currently have a call in with our provider to see if we can get an answer as to why we were without service for 3 hours Friday afternoon from 3-6pm. Again, our end was fine, and the provider's end was fine. It was the connection between us that was down known as the "local loop". That is who we are hoping to get some sort of explanation from.

At approximately 3:00pm the line dropped ceasing communications. Support was immediately called and after verification that both our ends were functioning properly, a ticket was opened up with the local loop provider. While on the phone with support for the second time at approximately 3:15pm, the line came back up for about 20 seconds. This confirms that it is the local loop. Again at 3:40 connection was restored enabling us to browse outside and it was taken down again. 4:30pm the line appears to be functioning. We will press for a reason why this happened and keep you informed. 4:34pm ... down again. 4:35 up. 4:36 down. 5:00pm still down. After several attempts, it seems the line cannot stay up. Of course these attempts are only seen at this end by a little LED indicating status. It appears the line came up and stablized at 6pm.

System maintenance was applied this morning at 5:00am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

You may have experienced a brief outage early this morning due to a router configuration change. This outage should have been simply a blip in your daily activity.

The test on the line turned out "clean" results. The upstream is doing maintenance between Saturday, May 25th, 2002 10:00 PM and Sunday, May 26th, 2002 2:00 AM but this will not be affecting our services. They are moving some of their customers around. Don't know if this is related to the router troubles we turned up.

SCHEDULED OUTAGE: The line was taken a couple minutes past 3:00AM and seems to have been restored approximately 5:30AM making the total outage approximately 2.5 hours. Hopefully whatever the results will aid in the determination of the upstream router troubles. From the pings that were received back from customers, it doesn't seem to be within our network at all. Thanks for sending those in. Although, we will have to wait for the upstream to realize these issues.

SCHEDULED OUTAGE: The line is tentatively scheduled to be stress tested at 3:00am Wednesday, May 22nd. An outage should occur not to last more than one hour. We're told it will take 30 minutes to an hour to complete the test.

System maintenance was applied this morning at 3:30am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

Yet more system maintenance was applied this afternoon at 1:45pm EST. This was a mid-day apply of maintenance due to the nature of the security update. Seems Microsoft found 10 bugs in IIS that required immediate attention. Of course, with every bug Microsoft finds, it could result in unauthorized execution of malicious code. Because of the many different areas that this security update addressed, we felt that a mid-day maintenance event was necessary. We are sorry that Microsoft requires a system restart after the application of maintenance, but we don't think Microsoft cares much about the web sites running under IIS servers throughout the world. Sorry for our rant, but if you look below, Microsoft's buginess is really out of control ... You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. During this time, our Linux servers started to gather and wonder what was going on with the Microsoft servers. A spokesman for Bill Gates leaped out of one of the servers and told them to 'Move along, there's nothing to see here!'.

Yet more system maintenance was applied this morning at 6:00am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

System maintenance was applied this morning at 7:30am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes. .NET SP1 was also applied to those servers running the .NET Framework. In some cases, servers required a double boot. Ah, Microsoft ... keeping our servers running (except when you have to boot them.)

New installation of .NET Framework software on one of the web servers occurred at 3:45AM EST. The installation required several system restarts and the application of some system maintenance from Microsoft. This was all completed by 4:00AM EST. You may have experienced strange activity at this time not lasting beyond this 15 minute period.

System maintenance was applied this morning between 4-5:00am EST. You may have experienced inaccessibilty while the server you were trying to access had its maintenance applied lasting no longer than 5 minutes.

System maintenance was applied this morning shortly after 5:00am EST. You may have experienced inaccessibilty for no longer than 5 minutes at this time.

New system seems to be functioning properly. No errorneous messages in the Event Log and operations appear normal. We apologize for any inconvenience. If you have any questions or concerns, email us.

Norton Ghost to the rescue! All services were restored by 8:00pm EST. A new motherboard was installed in the server and Norton Ghost was used to image the drive. So we got the new motherboard and inadvertently ended up with a new hard drive as well. All DNS has been pointed back to the original server.

The problem has been diagnosed as the motherboard. This was a special motherboard ordered and cannot easily be replaced today. Because of the Windows 2000 operating system, those drives cannot be put in another machine and be booted. Still don't know what this feature is good for? Anyway, sites have been created on another server. As the evening progresses, we will see whether we'll have a solution to the existing hard drives, or if we have to start creating new accounts on another server. At this time FTP accounts were not created figuring that the original configuration will be restored. If you have any major problems, concerns or any questions, please email us.

We are experiencing a hardware failure on one of our servers. We are working to fix this problem as soon as possible.

There was an error on one of the servers approximately at 7:00pm EST. At approximately 8:00pm we received a phone call from one of our resellers reporting the problem. The machine was cycled and operation restored. The cause will be investigated and we will let you know what happened. Worse case scenario will be that the server will be replaced. Sorry for any inconvenience this may have caused.

There has been some concern as to whether or not our customers would benefit from us installing Exchange Server. If this affects you, please email us and let us know. Thank you.

Happy New Year! We remain extremely optimistic for the coming year's events.

Seasons Greetings! Several system restarts due to critical maintenance being applied. All were at wee hours so as not to affect the majority of users. Another Linux box has been brought up. We will soon network the two and start procedures to host several sites on the Linux setup. There will also be a domain registration system installed on there and resellers (and end users) will be able to utilize these services to economically register new domain names and transfer existing domain names to us as a registrar to save you money. We will keep you posted via the web site as a new design will be implemented soon.

Maintenance applied approximately 7:00am.

Yesterday our Internet bandwidth was increased.

Performed two system restarts this morning. One at 4:30am, the other at 6:00am EST. This was necessary for the installation of some new hardware. The 4:30am boot was simply a system restart whereas the 6:00am restart was from a complete system shutdown. Any questions, please email us. No restart or shutdown lasted any longer than 10 minutes. These restarts were unexpected. Even with these restarts, hardware did not respond as planned. This hardware does not affect the operation of your services. We apologize for any inconvenience.

8:00am EST all problems solved. Always wait for day shift. Please add to our resume complete knowledge of the LinkSys SVIEW04 KVM Switch. Two different support people had no ideas. We played with the dip switches. Must be the reason they call them dip switches ... If you use one of them, it's really simple. We were chaining two together, apparently no one else on the face of the planet has ever done this.

Maintenance was applied aproximately 5:00 AM EST to all servers. This maintenance was unrelated to the router maintenance. If you or any of your customers saw an outage at this time, it didn't last more than 5 to 10 minutes.

ANNOUNCEMENT: Scheduled maintenance to emergency core router between 2:00 - 3:00am Monday, October 29th. An outage may occur not to last more than 10 - 15 minutes during this window.

We started a list for newbie Linux admins coming from a Windows background. If you'd like to join this list, send an email to linuxadd@wombatsweb.com.

Updates applied from Microsoft at 1am. Again, you may have experienced outages at this time due to server restarts to apply the patches. These patches are necessary to avoid attacks on our servers such as Code Red or Nimda. Thank you for your understanding.

All machines were updated with Microsoft Critical Updates at approximately 4:00 AM EDT. If you experienced an outage at this time we apologize for the inconvenience. Each update brought each server down for about a 2 minute period.

Due to Microsoft's impending release of XP this 25th and the not-so-sure future of where this will lead Microsoft themselves, we have begun further development into the Linux platform. Since we have, you can monitor our progress on the "Linux Testing" link on the main page. We have added a last updated date to this link as well so you can see if there is any new news available. This does not mean that we are looking to replace all of our Microsoft servers by any means. Heck, we'd be lost without them and their VB interoperability! We are simply leaving the current network in place and looking to add further means of meeting customers' requirements. Believe it or not, there are benefits to running both operating systems within the same infrastructure and there are ways of making them talk to each other and work in concert.

WARNING: We have been bombarded with this new worm hundreds of times per hour today starting exactly one week and supposedly at the minute that the WTC tragedy occurred ... A 'net slowdown is the result with many many more infected systems to come. Please protect yourself. Once again ICUHost having the most up to date systems was not affected.

W32.Nimda.A@mm is a new mass-mailing worm that utilizes multiple methods to spread itself. The worm sends itself out by email, searches for open network shares, attempts to copy itself to unpatched or already vulnerable Microsoft IIS web servers, and is a virus infecting both local files and files on remote network shares. The worm uses the Unicode Web Traversal exploit. A patch and information regarding this exploit can be found at http://www.microsoft.com/technet/security/bulletin/ms00-078.asp. When the worm arrives by email, the worm uses a MIME exploit allowing the virus to be executed just by reading or previewing the file. Information and a patch for this exploit can be found at http://www.microsoft.com/technet/security/bulletin/MS01-020.asp. Users visiting compromised Web servers will be prompted to download an .eml (Outlook Express) email file, which contains the worm as an attachment. Users can disable 'File Download' in their internet security zones to prevent compromise. Also, the worm will create open network shares on the infected computer, allowing access to the system. During this process the worm creates the guest account with Administrator privileges.

More information can be found at:
http://www.symantec.com/avcenter/venc/data/w32.nimda.a@mm.html

The tragic events that are unfolding in New York City and Washington DC are disrupting communications throughout the country. The Internet is especially affected as family and loved ones try all means possible to get information. We are seeing massive latency and disruption throughout the eastern corridor. We will pass any additional information we receive throughout the crisis. Our hearts and prayers go out to all.

Reports of major communication failures in New York City are starting to come in, as facilities that were undamaged from the building collapses are still being overcome by spreading fire and power outages. Many operating, but inaccessible locations have been on generators that cannot be refueled. This will greatly affect reachability of overseas sites throughout the day, as well as domestic locations. To minimize the impact as much as possible, technicians are diligently working to procure additional routes to non affected areas. Currently one of our backbone providers is experiencing issues which is causing customers to experience additional connection and latency problems. Please revisit this site for updates.

We added a new domain to serve as the web design info ICUDesign.Net. A simple page has been generated there and we'll work on it as time allows. Part of the reason for this is that we started playing with style sheets on ICUHost's web site and have been seeing some strange results in Netscape browsers, so we wanted another site to "play" on ...

Applied system updates from 8:00 to 8:30am today. It is possible you may have experienced an outage within this half hour. Several updates were made to various components. Normally we would have waited until the wee hours, but this was right on the edge of the business day, and critical updates we tend to try to get on the systems as soon as possible.

Put together a web site for Drop Dead Sexy.

Another outage resulting from the Baltimore/Philadelphia UUNet backbone lines. Outage lasted about 5 minutes from about 2:40 - 2:45 pm. Periods of sluggishness beginning around noon time preceeded the actual outage. The previous outage in the wee hours of 7/31 were a non-event whereas this one certainly sent off bells and whistles from many customers relying on our servers for their daily interaction. Unfortunately we feel helpless this being outside of our control and question the reasons for outages when redundancy is in place and a work-around has been in operation without any downtime since the fire that took out the Baltimore/Philadelphia line.

More info from my upstream: Thursday, August 9th, 2001 - Currently customers may experience slowness with their internet connection. WorldCom is reporting a fiber cut in the Baltimore area. This fiber cut is affecting connectivity to the surrounding areas including Philadelphia. Construction crews have been dispatched to restore the fiber. There is no estimated time of repair at this time. Please check back for further updates.

Response to our email: This may go without saying, but neither fiber cut was a planned outage. It's unfortunate that both occurred in such a short time frame, but there's little we can do when a backbone provider has serious trouble. We do what we can to route around the issue, but secondary paths get saturated rather quickly when a major outage occurs. In cases like this, all packets to or from the Eastern seaboard will be sluggish.

Network status graphs depicting incident: click here!

Apparently a UUNet backbone line went out this morning and there was an outage from about 3:15 - 3:35 am.

A couple customers expressed concern about the recent Code Red attack that's been plaguing the Internet and IIS web servers. Rest assured that as all Critical Updates from Microsoft become available, we are notified immediately and we apply the updates. We were prepared for the Code Red attack, but unfortunately were victim to the brunt of all these DDoS attacks flying around yesterday. While they were all stopped here, it didn't stop the 'net from flowing like molasses.

The online customer interface is operational in that simple queries can now be performed and emailed. The Update functions have not yet been put in place, but this should come fairly quickly. If you know your userid and password, you will be able to access the online customer interface here.

Thursday evening we removed all of the /29 IP addresses from the network and now are completely switched over to the /28 network. We also completely changed the firewall structure as well. If you have any problems, please report them immediately. Thank you.

This news entry will go a little off topic since we lost a great entertainer today, Jack Lemmon. Yesterday would have been a birthday of a great 44 year old programmer computer techie. He would have loved everything going on in and around ICUHost.Net. But anyway, so we digress on the ways of the world and journey towards our future, whatever that may hold.

Since the last news of the name server problems with Network Solutions, the expected happened. Everything was straightened out at 9am Friday, 6/22. Come Monday morning, 6/25 ... still no name servers. Spent 2 hours and 15 minutes on the phone with Network Solutions providing the same information over and over. Being told to fax authorizations for things that were supposedly fixed that prior Friday. We were told that we were waiting for the "registry people" and no matter how many times asked, the term "registry people" has still not been defined. There is no way to contact the "registry people" and the "registry people" are part of Network Solutions, but they aren't. 'What does that mean?' ... There was never a clear answer ... We were asked 'Would you like my supervisor to call?' .. We said yes, that would be a great idea. We are still waiting.
So anyway, we figured, what the heck, we'll fax in our changes as well as waiting for the "registry people" to cover all bases. Wouldn't you know it, Tuesday we get an email saying that we didn't sign the fax. Horror of horrors, we signed the fax and sent it again. Wednesday, 6/27 at 5am changes were finally completed and verified. At 4:44pm 6/27 we received a reply to our email detailing all of the changes that needed to be done and our request to forward the email to Level 2 the top or whomever could take care of the problem. Our entire message was quoted with instructions on how to use the WHOIS database.
We dread the next time we have to deal directly with Network Solutions again.

Name server woes seem to have been straightened out at Network Solutions ... We will see in the 24-48 hours to come. Then we can move forward with about 15 domains that have had to remain in limbo due to this problem.

Complete TEST domain registration system put in place. Wee hours, second statement went out to customers. Tuesday will concentrate on updates to billing system. NetSol troubles still persist with the IP address conversion. As Mr. T makes his 1-800-COLLECT comeback, we pity the fool who registers with NetSol.

We've completed first phase of new billing system. More updates to come, but we were running late on June and had to get the bills out. Any questions, please email us. Thank you. - - - In other news, the IP update has so far gone as planned. We're still waiting on NetSol. (It's ALWAYS NetSol) ... They have not documented the addition of the new IP numbers for ICUHOST.NET and the new reseller names servers. We wait. Unscheduled cycle at 4pm today.

All IP addresses changed internally and changed at directNIC. Any domain registered through us has had all necessary changes made. The only thing remaining is for the NetSol database to be updated and all NetSol domains to have their host nameserver IP addresses changed. When this was changed on directNIC, all other records were updated. NetSol is too busy making money to implement such a nice feature.

Windows maintenance applied and boot of servers.

We applied for a new block of IPs and were approved. We have grown into requiring these new addresses and it will facilitate the "anonymous" reseller name servers. Please don't post these name servers anywhere on this site or forums so that they remain anonymous. The implementation of the new IP addresses will move forward in the coming days. Please inform us immediately of any problems you encounter, but these changes should go unnoticed.

ANNOUNCEMENT: Scheduled maintenance to core router and power backup system. Outages between 2:00 - 4:00am not to last more than 2 - 10 minutes.

Additions to the ICUHost site. Added online sign-up form as well as Terms of Service. Put the hosting plans back up and altered the layout of the main page. Microsoft wanted to see our "URL where our company is displaying a description of its Web Hosting Service offering on Windows 2000" ... We originally provided the URL to ICUHost.Net where it says WINDOWS 2000 WEB HOSTING but we guess they wanted to see the individual packages? So, as said, we put back up the plan page.

Added support forums to the web site. Hopefully this will be a great place for help amongst customers. Also registered a new domain name to be temporary and to be used for name servers. This will provide anonymity from us to resellers who may charge more for their services.

ANNOUNCEMENT: Scheduled work will be performed Sunday, April 29 2:00 - 4:00am. Type of work being done: Complete upgrade redundant core router at Philadelphia. Services Affected: Some sites may experience a brief outage as they are replumbed. Expected length of outage: 3-4 minutes.

Updates, updates ... is our work ever done? Taking a break from our hacker paranoia and hosting development, we had to do some much needed web site updates. In the past couple days we've updated the Wombat's Web site, Gfab, DebiJaye, Nelson, Monkey Bus, The Turnstyles, Out On the Town and probably some others. A lot of nifty ASP stuff. In fact, on some of the sites you won't even notice a change. We've also had 3 or 4 new sites come on board utilizing our hosting.

It's upsetting, but boots are effecting our uptime rating. Unfortunately, this is necessary to install all the Windows Updates that have come out and installing various components that customers require. At some point, we should come to a plateau and our 5 9's rating will once again rise. When ICUHost.Com is moved here, we will add that site to a different server to be able to show uptimes across the network. Also, with the addition of the ICUHost.Com domain, all temporary site names will use that domain since it will be flat file based rather than registry based as it is now with ICUHost.Net.

We have been so busy doing everything imaginable but update this news. Sites are being set up for hosting. The particulars of the handling of this is being changed for the better with each addition or change. Web sites are being updated across the servers with different content based on the changes that happen each month and basic maintenance. Not to mention the other jobs that we do in advertising and throughout the course of our normal (if you want to call them that) days. Rest assured, everything is happening for the better.

This news file was created along with the beginnings of the new ICUHost.Net web site. This news will probably replace the news that now exists on Wombat's Web. Incidentally, all servers were upgraded to the latest service levels and security updates according to the Microsoft Windows Update.