It started as two email messages in quick succession: both the City of Ann Arbor and the AATA reported that their phone service was out, due to a "regional outage". How bad is this outage, and what's the cause?
I did some searching and came across the very useful "outages" mailing list, where network operations staff goes to share network down problems. This pointed at Windstream as the source of the network problems, and expanded the scope of the problems well beyond Ann Arbor. One report of call failures listed the following cities:
Cedar Rapids Springfield, IL Madison, WI Green Bay, WI Minneapolis Sioux Falls Omaha Des Moines Dubuque Fargo St. Louis Davenport Kansas City Detroit
as affected.
From a comment left on this page (thanks to Roy Schmidt)
From: Zach Wilcox Sent: Wednesday, October 24, 2012 12:38 PM Subject: Windstream Outage Update Importance: High This just in. TNOC REPORTING DWDM SYSTEM B19201 POSSIBLE TRIPLE FAILURE. 1ST ISSUE, OC-192 CARD IN TERRE HAUTE, IN (WE ARE UNABLE TO SEE THAT NODE). TECH IS ENROUTE, ETA IS 45 MINUTES. ALSO, SHUTTLING A CARD FROM CHAMPAIGN, IL, ETA IS 1.5 HOURS. 2ND ISSUE, FIBER CUT IN GREENCASTLE, IN. ONE END IS SPLICED. THE SECOND END IS BEING PREPPED, SPLICING ANTICIPATED AT 0945. 3RD ISSUE, FIBER CUT IN MILWAUKEE, WI. STILL DEVELOPING DETAILS ON THAT. LOOKING INTO POSSIBLE REROUTING OF CIRCUITS TO RESTORE SS7 TRAFFIC. Zach Wilcox Ener Tel Communications
This page is getting a lot of traffic for the outage; here's a map showing where some of that traffic is coming from, which approximates the boundaries of the issue.
News coverage of this seems to depend on whether the newspaper itself was affected by the outage. Some of those stories follow.
Lansing (Michigan) State Journal:
The cause appears to be a cut fiber-optic cable in Wisconsin and an equipment malfunction in Indiana, said David Avery, a Windstream spokesman.
The outage, which started about 9 a.m., according to the City of Ann Arbor, was affecting customers of the vendor Windstream in several states, a woman who answered the phone at a company call center said. The company did not have an estimate about when service would be restored.
Authorities in Michigan are reporting scattered problems with phone service around the state.
Locally, a technical problem caused a phone outage to a number of Mid-Michigan customers of Windstream, a company based in Little Rock, Arkansas.
Phone outages are being reported across mid-Michigan today as officials from Detroit, Ann Arbor, and Grand Rapids say they're experiencing technical issues with local calls.
Tracker.outages.org: Unplanned: Windstream/PAETEC Voice Outage
Trouble Ticket: 122813
More as news rolls in.
From: Zach Wilcox
Sent: Wednesday, October 24, 2012 12:38 PM
Subject: Windstream Outage Update
Importance: High
This just in.
TNOC REPORTING DWDM SYSTEM B19201 POSSIBLE TRIPLE FAILURE. 1ST ISSUE, OC-192 CARD IN TERRE HAUTE, IN (WE ARE UNABLE TO SEE THAT NODE). TECH IS ENROUTE, ETA IS 45 MINUTES. ALSO, SHUTTLING A CARD FROM CHAMPAIGN, IL, ETA IS 1.5 HOURS. 2ND ISSUE, FIBER CUT IN GREENCASTLE, IN. ONE END IS SPLICED. THE SECOND END IS BEING PREPPED, SPLICING ANTICIPATED AT 0945. 3RD ISSUE, FIBER CUT IN MILWAUKEE, WI. STILL DEVELOPING DETAILS ON THAT. LOOKING INTO POSSIBLE REROUTING OF CIRCUITS TO RESTORE SS7 TRAFFIC.
Zach Wilcox
Ener Tel Communications
I just received this email from Windstream:
Dear VINCENT RICHARD:
We are sending this email in regard to Paetec Repair Ticket #4843190 on Circuit 04385538. Information Update: Hello All, We are currently experiencing a large scale outage that is affecting the entire Mid-West Region as well as a large part of the entire Windstream network. The result of this outage is creating loss of physical connectivity as well as call processing, this would include inbound, outbound local, long-distance and toll free calls. Due to this call forwarding will also not work. There is no estimated time of repair at this time.
Thank you
Paetec Repair
This email has been auto-generated, please do not respond.
Thank you for your excellent coverage of this! We recd an official msg this morning that corresponds basically with what you discovered. Sounds like one of those perfect storm situations. Better multiple unlikely problems occurring simultaneously than three separate outages. Our company represents carriers like Windstream so had at least one customer down yesterday but only at their Warren MI location, not at any of their others (Tampa, Indy, Tulsa, Charlotte NC).
That is really bad if there is not phone services.In Helsinki Finland they maintain all their phone services specially those call centers which providing customer services for everyone.Well glad you post it.
There’s another Windstream outage today, affecting the Southeastern US. One article here:
http://www.charlotteobserver.com/2013/04/29/4011437/windstream-customers-company-report.html
Windstream is pointing people at their Facebook page for updates
https://www.facebook.com/WindstreamCommunications
in part because their main web page
http://www.windstream.com
is down. (!)