Internet connectivity is provided by the Academic Computing and Communications Center (ACCC) at the University of Illinois at Chicago. ACCC's network is, in general, extremely fragile—not because the hardware is particularly bad, but because ACCC's Ed Zawacki has deployed a series of network drones whose entire function is to make the network fragile. These drones are remarkably trigger-happy, continuously operating even when Zawacki is asleep, and not subject to any human oversight; on several occasions critical portions of the cr.yp.to network infrastructure have been blasted away by those drones and rebuilt only with manual effort. Zawacki claims that this is somehow helping security.
Some cr.yp.to services are now transparently replicated on servers in Europe.
2000-11-16: The .to administrators destroy yp.to, for reasons that have never been explained. They try to fix the problem quickly, but it persists for many hours, thanks to BIND's obsolete zone-transfer protocol.
2002-01-20: It happens again. Aargh!
The cr.yp.to web pages were unreachable during most of these outages. If your browser's error messages misled you into believing that cr.yp.to didn't exist, complain to your browser vendor.
1998-06-03: Power outage at UIC.
1998-10-13: Power outage at UIC.
1998-11-23: Power outage at UIC.
1999-01-07: Power outage at UIC.
1999-07-06: Power outage at UIC. This one lasts for eight hours, which I believe is a UIC record. Rumor says that the technicians don't like hearing the generator alarms so they turned the alarms off. Rumor also says that UIC has owned a backup generator for a while but has never plugged it in.
1999-10-22 ~14:00 GMT: Power outage at UIC. A short one, only two hours.
2000-01-01 04:00 GMT: UIC's network goes down for four hours. This is a PHB problem, not a Y2K problem. As explained by ACCC's Ahmed Kassem two weeks earlier: "UIC's campus network has been tested ... we fully expect the network to remain completely operational ... we will disconnect UIC's network from the outside world ..." That's an interesting definition of "completely operational."
2000-01-10 ~02:00 GMT: Scheduled upgrade to OpenBSD 2.6. One monitor explodes in the middle of the operation, but fortunately I have a spare.
2000-01-18 10:16 GMT through 13:00 GMT: One of UIC's routers dies, cutting the department off from the Internet.
2000-01-29 16:10 GMT: Power outage at UIC.
2000-04-13 ~10:30 GMT: Power outage at UIC.
2000-06-30 00:00 GMT: A planned 10Mbps-to-100Mbps upgrade is stymied for a few minutes by a low-quality network cable.
2000-08-23 ~04:00 GMT: OpenBSD crash. Cause undetermined.
2000-10-08 17:45 GMT: Power outage in downtown Chicago. Power is restored five hours later. The computers are revived the next day.
2000-11-05: Network outage for more than 24 hours, thanks to Ameritech, the local phone company.
2000-11-22 11:00 GMT: Network outage for more than 4 hours, thanks to Ameritech.
2001-01-05 19:45 GMT through 21:10 GMT: OpenBSD crash. Cause undetermined.
2001-01-12 00:00 GMT through 00:25 GMT: Scheduled upgrade to OpenBSD 2.8.
2001-03-14 22:30 GMT through 22:40 GMT: Hardware upgrade.
2001-04-25 21:00 GMT through 21:10 GMT: Scheduled electrical upgrade.
2001-05-04 18:35 GMT through 18:40 GMT: Scheduled electrical upgrade.
2001-05-23 21:00 GMT: Scheduled electrical upgrade. Reboot is prevented by hardware failure: the mail disk super block has died. Web service is restored within an hour. Mail service is restored the next day.
2001-07-08 ~00:00 GMT through 17:35 GMT: Power outage at UIC. Power is restored at some point between four hours and seven hours later.
2001-07-30 ~19:30 GMT through ~23:30 GMT: Network outage at UIC. ACCC is more informative than usual: "There is a hardware problem with the ATM link leaving campus. We do not yet know the exact cause or when our external link will be fully operational."
2001-08-26 ~04:00 GMT through ~19:00 GMT: OpenBSD crash. Cause undetermined.
2001-09-06 04:00 GMT through 04:10 GMT: Hardware failure.
2001-11-03 15:00 GMT: Fiber cut at UIC. Fixed 50 hours later.
2002.02.26 ~17:30 GMT through ~19:30 GMT: OpenBSD network stack crash. The load was not heavy (about 20 web downloads per second from slashdot, plus a few mail deliveries per second) and presumably would have been handled without trouble by the FreeBSD network stack.
2002.03.09 09:39 GMT through 17:02 GMT: Power outage at UIC. Didn't hit the building with the cr.yp.to servers, amazingly enough, but did take down the campus Internet connection. Didn't quite match the previous eight-hour record.
2002.03.24 ~10:00 GMT: Network outage at UIC. For at least the first hour, ACCC claims that everything is just fine.
2002.08.27 through 01:59 GMT next day: Crash. Cause undetermined.
2002.09.11 through 17:57 GMT: Crash. Cause undetermined.
2002.09.18 20:40 GMT through 20:45 GMT: Scheduled outage. Web pages are moving to a FreeBSD machine.
2002.09.27 14:00 GMT through 16:00 GMT: Scheduled mailing-list outage. Mailing lists are moving to a FreeBSD machine.
2003.07.17 22:30 GMT through 2003.07.18 07:30 GMT: Power outage at UIC (under 2 hours), followed by network outage at UIC (over 7 hours).
2003.08.14 23:45 GMT through 2003.08.15 03:30 GMT, with occasional uptime: Supposedly a scheduled UIC network outage to install uninterruptible power supplies for the UIC routers. No explanation of why this takes four hours.
2003.10.11: Zawacki's trigger-happy drones obliterate the network connection to my DNS cache. They say that the machine is "scanning remote hosts on port 53."
2003.10.14: Network outage at UIC.
2004.09.14 09:30 GMT through 2004.09.15 17:30 GMT: Massive power outage at UIC. ("A water main break flooded electrical vaults on the east campus.") The power outage lasted roughly 12 hours; the rest of the delay was caused by a UNIX filesystem that had blocks written in an incorrect order. (This could have been a bug in the disk hardware, such as the idiotic idea of hardware disk-write caching, or it could have been a bug in the filesystem software.)
2004.12.07 19:05 GMT through 19:15 GMT: Power outage at UIC. One of my computers died, extending the outage a bit, despite the UPS; I wonder whether a power spike hit the computer through the network.
2004.12.09 12:05 GMT through 12:15 GMT: Power outage at UIC.
2004.12.26 several hours: Hardware failure. Equipment upgrades are being stymied by UIC's incompetent grant management.
2005.08.14 about an hour: Zawacki's trigger-happy drones obliterate the network connection for my web server. They say that the machine is running a "rogue FTP server."
2005.10.12 about two hours: Zawacki's trigger-happy drones obliterate the network connection for my web server. They say that the machine is "compromised" and is "scanning on port 25."
2006.03.16 about 16 hours: Zawacki's trigger-happy drones obliterate the network connection for my mail server. Could some masochist please hire Zawacki so that I can stop dealing with his incompetence?
2007.01.09 about 12 hours: For unexplained reasons, UIC disables all network traffic except to the computers at the UIC computer center.
2007.03.18 ~18:00 GMT: Scheduled outage to move servers.
2007.03.20 ~23:00 GMT?: Zawacki's trigger-happy drones obliterate the network connection for my mail server.
2007.09.29 ~04:00 GMT: Another multiple-hour power outage at UIC. The explanation, which you're not supposed to laugh at, was finally given on 3 October: "There are two ComEd grids that feed the UIC campus. One of the grids went down causing a power glitch for 2.5 minutes. As a result, many building level breakers tripped resulting in the loss of power on a building basis that lasted until the UIC electricians could reset the building equipment." You're also not supposed to laugh at the fact that, several hours after the outage, the computer center still wasn't aware that there had been an outage.
2008.05.05: Zawacki's trigger-happy drones obliterate the network connection for one of my servers.
2008.11.08 ~13:30 GMT to ~17:20 GMT: Intentional multiple-hour power outage at UIC, announced on a web page just 16 hours in advance.
2009.06.19 ~23:00 GMT: Power outage at UIC during severe thunderstorms. Mail server has trouble coming up after the outage and is not restored until a day later.
2009.08.24: Hardware failure is delaying email.
2009.11.08 ~06:00 GMT to ~14:25 GMT: Unexplained network outage at UIC.
2010.08.15: Hardware failure on main mail/web server.
2011.08.11: Long-lasting power outage affecting replacement mail/web server.
2011.08.12: Scheduled 3-hour network outage at UIC for a router upgrade, plus the expected fallout.
2011.08.20: Zawacki's trigger-happy drones obliterate the network connection for one server. They say that the server is "infected with mass mailing worm."
2011.08.21: Zawacki's trigger-happy drones again obliterate the network connection for that server.
2011.10.01 ~09:00 GMT: UIC CS router crash. Web service (except bench.cr.yp.to) restored an hour later on a backup server.
2011.10.05: Several machines taken offline by UIC router misconfiguration. Most service unaffected, but bench.cr.yp.to offline for a day.
2013.03.19 ~03:00? GMT to ~07:00 GMT: One cr.yp.to server unreachable for unknown reasons. Didn't crash.
2013.12.13 through 2013.12.17: RST packets being forged by misbehaving network switch at UIC. Symptoms: connections hang or are terminated. Network administrators blame "evil sprite" for cross-connection.
2013.12.18 until ~17:00 GMT: Unexplained network outage at UIC.
2014.02.04 for several hours: Unexplained network outage at UIC.
2016.12.08 for most of a day: "Typo" in firewall rules by Central ICT Services at Technische Universiteit Eindhoven, compounded by a hardware failure.
2017.07.24 roughly 21:50 to 22:10: Unexplained network outage at UIC.
2020.04.20 roughly noon GMT: Power outage at UIC.
2021.06.23 roughly noon GMT for several hours: 40Mbps denial-of-service attack, which shouldn't have been a big deal, except that the attack was amplified by Zawacki's trigger-happy drones turning off network service.
2022.12.13 roughly 06:00 to 11:00 GMT: Similar denial-of-service attack, again amplified by Zawacki.
2023.01.13 roughly 02:00 GMT: Similar denial-of-service attack.
2023.01.27 roughly 10:30 GMT: Similar denial-of-service attack.
2023.04.17: Similar denial-of-service attack.
2023.06.13 roughly 08:00 GMT for 18 hours: Similar denial-of-service attack.
2023.07.09 roughly 10:30 GMT for 16 hours: Similar denial-of-service attack.
2000-09 through 2000-11: UIC is paying its ISP for only 14Mbps, and is hitting that limit more and more frequently. The packet-loss rate hits 25% at busy times. The UIC computer center refuses to pay for adequate network service. Perhaps the router is being flooded by some easily fixed source of traffic; the UIC computer center won't even bother investigating.
2001-05-19 through 2001-05-21: Several reports of overloads. UIC is paying its ISP for 20Mbps.
2002.11.04 through 2002.11.07: More reports of overloads. The UIC computer center issues a statement: "Over the past two days the University has experienced degraded network connectivity ... Initially, this was the result of an unsanctioned and unannounced network experiment between researchers at UIC and a Greek research network which flooded our outside connection. Eventually, we managed to stop the traffic flow from bringing down our outside connections. In the meantime, our border router's connection to the rest of the campus had become affected in such a way that it was performing dramatically worse than before. After unsuccessfully attempting to fix the issue using a variety of means, including hardware replacement, the decision was made to completely redo the way this router connects to the University backbone. Only then was the problem completely resolved."
2002.11.13: Horrible network performance at the edge of the UIC network; no packets lost, but ping times up to 2 seconds. In the evening, periods of complete outages. The UIC computer center issues a statement: "Over the past 2 days, we have been having enormous difficulties in dealing with a Denial of Service attack that affected our ability to route traffic off of campus. After working with our two primary ISPs, we have finally been able to resolve the routing issue, and traffic is now leaving campus normally." Outages continue for a little while after that statement.
2005.11.22: Someone at IP address 196.34.112.44 (in Sentech's network) spends three and a half hours making 140000 connections to the cr.yp.to SMTP server with some impressively incompetent spam software. Slight degradation of service, nothing serious.