Windstream outage takes out phone service to customers across the Midwest

It started as two email messages in quick succession: both the City of Ann Arbor and the AATA reported that their phone service was out, due to a "regional outage". How bad is this outage, and what's the cause?

I did some searching and came across the very useful "outages" mailing list, where network operations staff goes to share network down problems. This pointed at Windstream as the source of the network problems, and expanded the scope of the problems well beyond Ann Arbor. One report of call failures listed the following cities:

Cedar Rapids
Springfield, IL
Madison, WI
Green Bay, WI
Minneapolis
Sioux Falls
Omaha
Des Moines
Dubuque
Fargo
St. Louis
Davenport
Kansas City
Detroit

as affected.

From a comment left on this page (thanks to Roy Schmidt)

From: Zach Wilcox Sent: Wednesday, October 24, 2012 12:38 PM Subject: Windstream Outage Update Importance: High This just in. TNOC REPORTING DWDM SYSTEM B19201 POSSIBLE TRIPLE FAILURE. 1ST ISSUE, OC-192 CARD IN TERRE HAUTE, IN (WE ARE UNABLE TO SEE THAT NODE). TECH IS ENROUTE, ETA IS 45 MINUTES. ALSO, SHUTTLING A CARD FROM CHAMPAIGN, IL, ETA IS 1.5 HOURS. 2ND ISSUE, FIBER CUT IN GREENCASTLE, IN. ONE END IS SPLICED. THE SECOND END IS BEING PREPPED, SPLICING ANTICIPATED AT 0945. 3RD ISSUE, FIBER CUT IN MILWAUKEE, WI. STILL DEVELOPING DETAILS ON THAT. LOOKING INTO POSSIBLE REROUTING OF CIRCUITS TO RESTORE SS7 TRAFFIC. Zach Wilcox Ener Tel Communications

This page is getting a lot of traffic for the outage; here's a map showing where some of that traffic is coming from, which approximates the boundaries of the issue.

Picture 48

News coverage of this seems to depend on whether the newspaper itself was affected by the outage. Some of those stories follow.

Lansing (Michigan) State Journal

The cause appears to be a cut fiber-optic cable in Wisconsin and an equipment malfunction in Indiana, said David Avery, a Windstream spokesman.

AnnArbor.com

The outage, which started about 9 a.m., according to the City of Ann Arbor, was affecting customers of the vendor Windstream in several states, a woman who answered the phone at a company call center said. The company did not have an estimate about when service would be restored.

WNEM, Channel 5, Saginaw

Authorities in Michigan are reporting scattered problems with phone service around the state.
Locally, a technical problem caused a phone outage to a number of Mid-Michigan customers of Windstream, a company based in Little Rock, Arkansas.

Michigan Radio

Phone outages are being reported across mid-Michigan today as officials from Detroit, Ann Arbor, and Grand Rapids say they're experiencing technical issues with local calls. 

Tracker.outages.org: Unplanned: Windstream/PAETEC Voice Outage

Trouble Ticket: 122813

More as news rolls in.

Advertisements

5 thoughts on “Windstream outage takes out phone service to customers across the Midwest

  1. Roy Schmidt

    From: Zach Wilcox
    Sent: Wednesday, October 24, 2012 12:38 PM
    Subject: Windstream Outage Update
    Importance: High
    This just in.
    TNOC REPORTING DWDM SYSTEM B19201 POSSIBLE TRIPLE FAILURE. 1ST ISSUE, OC-192 CARD IN TERRE HAUTE, IN (WE ARE UNABLE TO SEE THAT NODE). TECH IS ENROUTE, ETA IS 45 MINUTES. ALSO, SHUTTLING A CARD FROM CHAMPAIGN, IL, ETA IS 1.5 HOURS. 2ND ISSUE, FIBER CUT IN GREENCASTLE, IN. ONE END IS SPLICED. THE SECOND END IS BEING PREPPED, SPLICING ANTICIPATED AT 0945. 3RD ISSUE, FIBER CUT IN MILWAUKEE, WI. STILL DEVELOPING DETAILS ON THAT. LOOKING INTO POSSIBLE REROUTING OF CIRCUITS TO RESTORE SS7 TRAFFIC.
    Zach Wilcox
    Ener Tel Communications

    Reply
  2. Richard Vincent

    I just received this email from Windstream:
    Dear VINCENT RICHARD:
    We are sending this email in regard to Paetec Repair Ticket #4843190 on Circuit 04385538. Information Update: Hello All, We are currently experiencing a large scale outage that is affecting the entire Mid-West Region as well as a large part of the entire Windstream network. The result of this outage is creating loss of physical connectivity as well as call processing, this would include inbound, outbound local, long-distance and toll free calls. Due to this call forwarding will also not work. There is no estimated time of repair at this time.
    Thank you
    Paetec Repair
    This email has been auto-generated, please do not respond.

    Reply
  3. Jim Greiner

    Thank you for your excellent coverage of this! We recd an official msg this morning that corresponds basically with what you discovered. Sounds like one of those perfect storm situations. Better multiple unlikely problems occurring simultaneously than three separate outages. Our company represents carriers like Windstream so had at least one customer down yesterday but only at their Warren MI location, not at any of their others (Tampa, Indy, Tulsa, Charlotte NC).

    Reply
  4. Ensio Mäentausta

    That is really bad if there is not phone services.In Helsinki Finland they maintain all their phone services specially those call centers which providing customer services for everyone.Well glad you post it.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s