Summary: Partial network outage due to upstream carrier configuration issue
Services Affected: VPS and dedicated servers in Winston-Salem, NC assigned to the 220.127.116.11/23 subnet
Timeframe: 2017-11-21 05:30 AM PST(UTC-8) - 2017-11-21 10:25 AM PST(UTC-8)
Due to a missing route authorization on our datacenter's secondary network provider, services assigned IP addresses in the 18.104.22.168/23 subnet were unreachable during the timeframe listed above. This issue was triggered by the primary network provider experiencing an outage relating to a fiber cut nearby. The majority of our network services in the NC location immediately failed over to the datacenter's secondary connection and experienced no outage window, however one subnet was apparently not properly authorized with the second carrier, leading to a situation where one network was unavailable.
We have taken the necessary steps to permanently resolve this issue and have confirmed that all networks are properly authorized with both network providers in this location. In addition to verification of both network providers authorizing the connection, we have also configured a virtual circuit to our Los Angeles datacenter that will allow us to globally route IP addresses allocated to our NC location while re-routing the traffic back to NC. This virtual circuit will be used for emergency situations only, in case any similar issue happens in the future.
Tuesday, November 21, 2017