On Wednesday, May 7 2019 at 8:14PM CDT, the VIRTBIZ DAL1 datacenter experienced a brief network impairment that prevented some customers from accessing the Internet.
Although core routing was unaffected, a recurring BGP flap at an off-site router caused external routes at the MDF level to be rapidly cycled. Routing software detected the failure and reset the BGP sessions. Unfortunately the software reset all sessions on the device simultaneously which caused routes to be re-learned individually. In this case routes began to repopulate instantly, but it took 8 minutes from start to finish to complete the process. Depending on the remote location, service may appear to have been down from 2 to 9 minutes. Service was verified to be fully restored by 8:24PM CDT.
Although we work very hard to ensure your uptime and satisfaction with our service, adversity can strike at the most inopportune time. For our customers impacted by the outage that resulted from today’s event, we offer our sincere apology.
