Our Yipes connection is bouncing up and down this morning. They have dispatched a tech to their Fort Collins node to investigate. No estimated time to repair.
When they go down traffic is failing over to our backup circuits, but it's causing some choppy performance. Expect reduced performance until this is resolved.
11:30AM MT --- Yipes reports the problem has been resolved, although we haven't seen any drops since 8:06AM.
We're having an issue with one of our web servers this morning. It looks like there may be hardware issues with the machine, so we are moving the sites hosted on that machine to a different server.
Update: All services have been restored.
We don't know the full story yet, but apparently there was a major power outage at PRPA (ironically the power company) where all of the fiber is terminated. If we get more information, like why a UPS didn't handle this problem, we'll post here.
A ticket is open and they are investigating. It appears to be affecting all of Fort Collins. We are still up but running very slowly over the backup circuit.
Tomorrow evening 4/21 at 6PM Pacific Time we'll be taking one web server offline to tend to a disk drive that is starting to have errors. We expect it to be unavailable for a few hours.
The ethernet card in our old mail server locked up again and required an emergency reboot of the machine.
We ran into some problems and were down longer than expected, but the ups has been replaced and all services are back online.
This Friday, February 29th, after 10PM Mountain time we will be replacing a defective UPS. This will require powering down several servers for approximately 15 minutes. All email and web hosting will be impacted. Network connectivity, colo customers, and VoIP should not be effected with the exception of Drake Park customers with a NAT'ed connection.
An unknown event caused BGP in both routers to drop, reconnect and eventually hang. There was a period of about 20 minutes of intermittent connectivity while this was diagnosed and resolved.
We had a power event in Fort Collins earlier today which caused a momentary loss of connectivity and several servers to reboot. Not all of the servers came back up cleanly. All services were effected and are back online as of this time.