There are no Open Network Issues Currently

Network Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Asset HTTP FTP POP3 PHP Info Server Load Online
Ave01 PHP Info
Ave02 PHP Info
Ave03 PHP Info
Ave04 PHP Info
Ave05 PHP Info
Ave06 PHP Info
Ave07 PHP Info
Ave08 PHP Info
Ave09 PHP Info
Ave10 PHP Info
Matango PHP Info
opus25 PHP Info
Rodan PHP Info
Stats PHP Info
Web39 PHP Info
Web40 PHP Info
Web41 PHP Info
Webhosting2 PHP Info
Webhosting3 PHP Info
POWER OUTAGE AND RESTORATION EFFORTS (Resolved) High
  • 01/07/2024 13:21 - 01/08/2024 14:52
  • Last Updated 01/07/2024 13:23

A catastrophic event in our area caused a power failure that impacted our facility.  In addition, vandalism to our backup power infrastructure caused the generator to power on, but power was not transmitted to the facility.   Crews were dispatched immediately but UPS power expired roughly 4 minutes before generator power could be restored.  At this time we are aware some services are still down and must be brought online manually.   We're working on full restoration as quickly as we can.  Thank you for your patience.

Network Routing Issue (Resolved) Critical
  • 12/25/2023 05:18 - 01/01/2024 10:26
  • Last Updated 12/25/2023 07:58

We are aware of a routing issue that occurred Christmas Morning 12/25/2023, beginning 5:18AM CST.    The issue was traced to a license server fault with our VSR (Virtual Software Router) vendor which resulted in license files becoming invalid.   Technicians manually connected VSRs to the Internet via static routing in order to patch the systems for license file updates.  After that, normal routing was able to be restored.   Further updates will be available as developments occur.

Database unavailable (Resolved) High

Affecting Server - Rodan

  • 12/16/2023 14:24 - 12/17/2023 18:25
  • Last Updated 12/16/2023 14:26

Our technicians are aware of a database issue on the Rodan.virtbiz.com cPanel server.   This has been escalated to cPanel product support as it appears to be a malfunction within the software.  We are actively working to address the issue and restore full functionality ASAP.   We thank you for your patience.

Spectrum Enterprise network maintenance notification (Resolved) Critical

Affecting Other - Upstream Carrier Network

  • 11/17/2023 00:01 - 11/25/2023 07:00
  • Last Updated 11/07/2023 11:54

Spectrum Enterprise will be conducting network maintenance on 11/17/2023 12:00AM CST until 11/17/2023 06:00AM CST.   The purpose of this maintenance is to perform OS upgrade on connected circuit hardware.  

During this maintenance window, you will experience one or more brief interruptions in service while we complete the maintenance activities; the interruptions are expected to last less than 15 minutes (each circuit migrated separately) total. However, due to the complexity of the work, your downtime may be longer. Customers may also see some reconvergence during the maintenance window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you.

Cogent network maintenance notification (Resolved) Critical

Affecting Other - Upstream Carrier Network

  • 11/10/2023 00:01 - 11/25/2023 07:00
  • Last Updated 11/07/2023 11:51

Cogent will be performing network maintenance from 12:01 AM CT 11/10/2023 to 04:00 AM CT 11/10/2023.   The purpose of this work is circuit migration to a more robust device.

During this maintenance window, you will experience one or more brief interruptions in service while we complete the maintenance activities; the interruptions are expected to last less than 15 minutes (each circuit migrated separately) total. However, due to the complexity of the work, your downtime may be longer. Customers may also see some reconvergence during the maintenance window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you.

Cogent network maintenance notification (Resolved) Medium
  • 07/13/2023 00:00 - 07/13/2023 09:24
  • Last Updated 07/12/2023 16:20

VIRTBIZ Internet Services is passing along the following information regarding one of our service providers:

As a valued customer, Cogent is committed to keeping you informed about any changes in the status of your service with us. This email is to alert you regarding maintenance our local access provider will be performing on their network:

 

 

Start Date & Time: 07/13/2023 12:00AM CDT

End Date & Time: 07/13/2023 06:00AM CDT

Time Zone: Central

Outage Duration: 120 (minutes)

Reason: Circuit Migrations


This maintenance is non-service-impacting for VIRTBIZ customers, as traffic will be routed across other peers.  However, some remote networks with slower peering updates may see brief interruptions in service.  This is due to the nature of Internet routing and is not indicative of a deficiency with the VIRTBIZ network.

 

System Maintenance (Resolved) High

Affecting System - VPS and Hosted Firewall

  • 04/16/2022 11:38 - 04/16/2022 21:00
  • Last Updated 04/19/2022 18:14

In accordance with an announced maintenance window, certain VPS and hosted firewall services are temporarily offline due to network restructuring.   Engineers are actively working on this service and will restore connectivity as quickly as possible.   Thank you for your patience.

Aventurin{e} VPS migrations (Resolved) High

Affecting Server - Ave03

  • 03/19/2022 12:33 - 03/20/2022 00:00
  • Last Updated 03/21/2022 08:23

Customers hosted on Aventurin{e}-based VPS servers will experience periodic outages on Saturday, March 19 2022 - Sunday, March 20 2022 due to system migrations to enhanced platforms at our new datacenter.   We appreciate your patience as we complete this work.   Note that the migrations are an automated process and systems are queued in turn for conversion.   The process will automatically restore service as soon as it is completed, without the need for intervention.

VPS containers on AVE1 offline (Resolved) High

Affecting Server - Ave01

  • 12/16/2021 19:40 - 12/17/2021 00:00
  • Last Updated 12/17/2021 07:47

Technicians are addressing a problem with the AVE1 virtualization node.   The VLSA storage system has been impacted and is causing virtualized containers to remain offline.   We are sorry for the impact this is causing our customers and understand that restoration of service is a top priority.   We are actively working on this issue and appreciate your patience.   Please note that this only applies to the AVE1 system.

IPv4 network outage (Resolved) Critical
  • 10/07/2021 01:26
  • Last Updated 10/12/2021 17:10

 

INCIDENT RFO AND SUMMARY REPORT:

On October 7, 2021 at 01:26 AM (US/Central) the DAL1 network experienced an issue which impacted a good deal of IPv4 transit to the facility.  IPv6 traffic was not impacted.  

Our technicians quickly diagnosed that the issue did not originate within our facility and began the escalation process to management and our carriers.   The Zayo upstream fiber connection lost light path for a brief period and then came back, but would not pass IP traffic.    Meanwhile, although IPv6 traffic was routing normally across our Lumen connection, IPv4 traffic was not.   It was clear that there would be two external issues to resolve simultaneously.

Working with our colleagues at Lumen was hampered somewhat by less than optimal response times.  Our management and engineering team worked on a conference call for nearly 2 hours, pushing through 3 levels of escalation with Lumen until we were ultimately able to work with an IP engineer on duty.  This engineer confirmed our suspicion that Lumen was not accepting our BGP route advertisements.   The precise cause of this is unknown but suspected to be the byproduct of a recent configuration update to Lumen’s router network.   Our advertised prefixes are published in IRR registries, and those route objects are imported automatically into upstream carriers, usually on a nightly basis.   For reasons that Lumen has not been able to identify, their system ceased importing these objects and therefore would not pass our advertised routes through BGP as they were unable to be validated by their routing network.   We asked Lumen to manually push an update to their router.  However, as it was in the process of a nightly cycle this took an unusually long time in their queue, and was further hampered by delayed propagation into their production routing network.  Lumen service was restored at 03:23 AM (US/Central) and traffic began to pass normally at that time. 

Working with Zayo, we were able to confirm good fiber connectivity (light levels were adequate on both sides).   Their NCC advised of no known issues or planned/unplanned maintenance on the circuit.   BGP had a healthy configuration but there was no IP.    At our behest, the NCC engaged a local field engineer to check the physical port.  When we asked how long it would take to dispatch, the NCC advised the field engineer had just left that POP at 2323 Bryan Street (our Z-point) and was turning around to return.  

Our internal working theory was that someone at the 2323 Bryan end had made a physical port change of some sort, possibly connecting to the wrong port at their router and that assumption influenced our input and requests to the Zayo NCC.   A short time later, the connection went down again (loss of light) and came back up.   IP connectivity was immediately restored and BGP synchronized right away.  Zayo service was restored at 04:16 AM (US/Central), returning the network to full redundancy.

As is ever the case with situations like this, the cause cannot be attributed to a single issue.   Multiple issues are typically compounded into an aggregate end result.   This is especially true in this case where we do have redundancy and protections in place, but those were compromised through a bizarre combination of external forces.

Lumen’s dropping of legitimately advertised BGP routes is troubling and we are aggressively pursuing an acceptable understanding of why this occurred and what can be done to prevent the issue from happening again.   In the meantime, we have configured preliminary external BGP route monitoring to augment our own internal monitoring so that we can verify third-party carriers are processing routes correctly.  Equally concerning is the amount of time that was required to reach an IP engineer qualified to confirm our findings and take appropriate actions to restore service.   We have requested further dialog with Lumen management on this.

Zayo’s response stating  “Our Field Engineer replaced the fiber jumper between our equipment and the OSP panel in order to restore the service” leaves something to be desired.   We remain uncertain that the issue was with a “bad” jumper, since we confirmed adequate light levels at the optical transceivers on both ends of the circuit.  Given there were no other factors, we feel our working theory that the fiber may have been accidentally removed and then re-installed into the wrong port has merit. 

We appreciate your patience as our team worked through the night and early morning hours to correct issues, document each case and put preliminary plans into action.  In all cases, our goal is to focus on strengthening the network and the ability to effectively manage third-party partners whenever necessary rather than to place blame.   To that end we will continue to use the information gathered from this event to add to internal systems where needed as well as establish or improve communication with third party providers who are vital to ongoing stable operations.

Thank you,

Chris Gebhardt, President
VIRTBIZ Internet Services

 

 

INITIAL REPORT:

The DAL1 network experienced an issue which impacted IPv4 transit to the facility.  IPv6 traffic was not impacted.   Our service from Zayo was interrupted due to an as-yet undiagnosed issue at their POP at 2323 Bryan Street.  Our network automatically healed to Lumen, but there was a problem with that carrier's BGP filters that prevented hosted prefixes from being advertised.   Unfortunately, they were not able to quickly resolve the problem due to staffing issues and then their routers were in an update which could not be interrupted in order to push the fix through their system.   Our engineers were dispatched immediately and remained on conference call with relevant vendors throughout and following the event.    A formal RFO may be composed once full research has been performed.

Server Upgrade 09/17/2021 - 09/18/2021 (Resolved) Medium

Affecting Server - Rodan

  • 09/16/2021 21:30 - 09/17/2021 21:49
  • Last Updated 09/16/2021 21:34

The webhosting node "RODAN.virtbiz.com" is being upgraded to the latest platform.   This action is being taken in order to ensure ongoing platform stability as well as to offer the latest available features.   Some services (website, email) may be intermittently unavailable during this time.   Thank you for your understanding as we perform this important upgrade for you.

Service Interruption 05/10/2021 (Resolved) Critical

Affecting Other - DAL1 Datacenter

  • 05/10/2021 18:11 - 05/18/2021 22:02
  • Last Updated 05/15/2021 11:01

Update: 05/15/2021  10:49

Work has been completed on the below referenced Emergency Maintenance Window.    All systems are normal at this time and no further interruptions are expected.    We thank our customers for your patience.

 


Update: 05/14/2021 11:49

Preparations have been made to decomission the UPS responsible for Monday's power failure that impacted multiple customers at our DAL1 facility and transition affected systems to replacement power facilities.

EMERGENCY MAINTENANCE WINDOW
impacting customers with service from Panels L5, L8, L14 and RTR2
Saturday, May 15, 2021
7:30am - 12:00 Noon

If your service is expected to be impacted by this work, you have already received email notification.   If you did not receive an email notification, your service is not believed to be impacted by this maintenance window.

All preparations have already been made, construction is complete and systems have been pre-wired as far as practical.   The work performed on Saturday will be confined to physically changing the high-voltage connections to service distribution panels.  Through extensive planning, careful engineering and workshopping with our technical staff, we are confident that the actual work will go very quickly.   Therefore, we consider the maintenance window timeframe to be conservative, and we hope to complete our work much earlier.  In any event, we anticipate the actual outage to any customer equipment should be 15 minutes or less.  

Since we will be working with high-voltage electricity, we will be restricting entry to the facility throughout the duration of the maintenance.    Customers will not be permitted inside the datacenter while work is underway.   This is to prevent possible injury to our customers and also eliminate distractions to the technicians who will be performing the work.

Once service is restored to each panel, technicians will proceed with crash-carts and connect to each impacted system that has video output in order to observe a proper boot cycle.   After that is complete, we will begin working customer tickets. 

 


Update: 05/11/2021 13:41

Staff, management and contracting engineers have diagnosed the service-impacting issue and are currently working to fully rectify the trouble.   One of our main power backup systems has experienced an internal fault.   This UPS installation (the engineering, but not the actual hardware) is part of the original datacenter build from 2007 and has become functionally obsolete due to its inability to provide adequate load for our current (and future) requirements.   Therefore, we had planned to decomission the system in Q4 2021 or Q1 2022 after moving existing services to new power distribution.   Given the serious nature of the failure that has occurred, we will now be accelerating that replacement to an immediate timeline.  

Work is already underway and is progressing as quickly as materials can be delivered and installed.   We have already taken steps to isolate our core network infrastructure from the impacted system.    Customer racks and 3rd party network service providers will be moved as soon as practical.   Notifications will be sent out accordingly when power will be cut-over.  

We appreciate your patience as we carry out this work.   We recognize the critical nature of the service that we provide to our customers and everything we do is focused on that.

 


Initial report: 05/10/2021 18:11

We are aware of an event that caused an interruption in routing service to DAL1 datacenter from 5:45pm to 5:52pm Central Time.

The issue appears to be related to a catastrophic systems failure at one of the 480V 3-phase UPS systems.   Our UPS service company has already dispatched a technician to review this.   At first glance it appears there has been an internal component that has failed, causing the load to momentarily trip offline.   We are still investigating to determine what, precisely, has happened and what steps will be taken going forward.   At this time our technicians have mitigated the issue with alternate power routing.

We appreciate your patience and understanding as we take the necessary steps of properly diagnosing, then permanently correcting the issue and apologize for any inconvenience from the impact of this issue.

 

 

Web hosting server upgrades in progress (Resolved) Medium

Affecting Server - Rodan

  • 09/09/2020 21:00 - 09/24/2020 21:33
  • Last Updated 09/10/2020 07:55

In order to provide the best web hosting experience possible, VIRTBIZ is performing upgrades to certain infrastructure.   During this time, it may be periodically necessary to temporarily suspend some services in order to ensure a smooth transition.   cPanel-based web-hosting customers on the "Rodan" server may experience brief periods of unavailability for various services including website and email functions.    Please be assured that the automated process responsible will not interrupt services for longer than absolutely necessary.   We appreciate your patience during this time.

Network Maintenance 09-14-2019 (Resolved) Critical

Affecting System - DAL1 Datacenter

  • 09/14/2019 11:24 - 09/15/2019 01:30
  • Last Updated 09/14/2019 11:29

We have identified software errors that may contribute to unscheduled outages or service degradation.  In order to mitigate ill effects, engineers will engage in updating and replacing some critical network infrastructure.   We are taking steps to minimize impact but due to significant engineering changes, some interruptions are possible.

Start time: 12:00 PM (CDT) 09/14/2019
End time: 23:59 PM (CDT) 09/14/2019
Expected Outage/Downtime: 10 minutes (intermittent)

During this maintenance window, customers may experience brief interruptions in service while we complete the maintenance activities. We do not expect extended periods of connectivity loss. However, due to the complexity of the work, it is possible that unforeseen issues could result in temporary connectivity loss. Customers may also expect to see some re-convergence during the maintenance window. Customers may experience latency and packet loss intermittently throughout the window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please contact our Support Team for assistance.

We apologize in advance for any inconvenience.   We are working as diligently as possible to ensure reliability of service to our customers.

###

DAL1 Routing Maintenance Window (Resolved) High
  • 06/09/2019 16:00 - 09/14/2019 11:24
  • Last Updated 06/09/2019 21:44

UPDATE: 06/09/2019, 9:31PM

In the announced maintenance window for 06/19 we implemented new RaaS to improve routing performance.     Unfortunately the tested configuration did run but apparently had some issue with LACP / LAG functionality that caused the network to perform erratically.   Our engineers monitored this and engaged our vendor support and attempted to resolve but the LAG ports continued to reset. This eventually lead to memory problems in the MDF's handling customer routes.   At that point, hosted assets were effectively disconnected from the core.    We are working with the vendor to troubleshoot and perform regression testing against our configuration.  At this time we have rolled back to the previous infrastructure pending successful testing and sign-off with our vendor.


In order to promote ongoing network stability, VIRTBIZ will be performing critical maintenance to update and upgrade core routing infrastructure at the DAL1 datacenter location.  This work includes physical replacement of core routing assets that supply border and edge connectivity.  Equipment and connections are already in place and a procedure has been established in order to minimize impact to end-users.

Start time: 4:00 PM (CDT) 06/09/2019
End time: 6:00 PM (CDT) 06/09/2019
Expected Outage/Downtime: 10 minutes (intermittent)

During this maintenance window, customers may experience up to two brief interruptions in service while we complete the maintenance activities. We do not expect extended periods of connectivity loss. However, due to the complexity of the work, it is possible that unforeseen issues could result in temporary connectivity loss. Customers may also expect to see some re-convergence during the maintenance window. Customers may experience latency and packet loss intermittently throughout the window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please contact our Support Team for assistance.

###

Route flap impacting certain DAL1 connections (Resolved) Critical

Affecting Other - DAL1 Datacenter

  • 06/04/2019 10:22 - 09/14/2019 11:24
  • Last Updated 06/04/2019 10:29

Engineers have identified a route flap issue impacting some connected routes are currently working with our vendors for full resolution.  

MEM1 power event (Resolved) Critical

Affecting Other - MEM1 (Tennessee) datacenter

  • 08/05/2016 09:37 - 08/05/2016 11:30
  • Last Updated 08/05/2016 12:40

Technicians are currently investigating a power interruption and loss of routing at MEM1 (Dyersburg, TN) facility.   At this time, colocated customers at MEM1 are inaccessible from the Internet.   This is a facility issue and not a VIRTBIZ routing failure, and facility technicians are on-site working to resolve the problem.

UPDATE 11:33AM
Investigations have revealed a cascade of failures that contributed to nearly 90 minutes of service interruption.  When the power failed, load was handled by UPS (battery) systems while the generator was auto-started.   Although the generator started as expected, the automatic transfer switch (ATS) did not transfer the load to generator power.  Accordingly, UPS units shut down as batteries were exhausted, causing the load to drop.   As power to the load was restored manually, an intermediate distribution switch that connects our router to the facility network failed to properly start and load.   As the switch is owned and maintained by the facility (not VIRTBIZ) their staff was required to bypass/replace the failed unit.  Once they restored their network connectivity, all VIRTBIZ assets became visible on the network.  

Our partners at the facility are conducting a post-mortem of this event in conjunction with their electrician and generator maintenance provider.   VIRTBIZ is standing by to assist the facility with their investigation as needed.

Emergency electric maintenance - 08/04/2016 (Resolved) High

Affecting Other - DAL1 Datacenter

  • 08/04/2016 16:00 - 08/04/2016 21:00
  • Last Updated 08/04/2016 20:19

EMERGENCY MAINTENANCE NOTIFICATION
Start time: 4:00PM Central 08/04/2016
End time:  9:00PM Central 08/04/2016
Expected Outage/Downtime:  n/a

UPDATE, 8:00PM
Oncor crews worked quickly to assist in the replacement of the malfunctioning transformers.  Datacenter operations were unimpacted during the maintenance period.  The window was closed as of 6:15PM Central.

ORIGINAL POST:
During routine inspections this morning, technicians noted moisture and discoloration around one phase of our primary (A-side) electric service entrance.   This indicates a failure of the transformer and points to a pending imminent power failure.   Upon notification of this issue, we made contact with the electrical transmission provider, ONCOR, and have arranged for replacement of the entire transformer bank.

Scheduling the maintenance window involves balancing the risk of unplanned transformer failure or fire with the desire to utilize off-peak hours as much as possible.  Accordingly, replacement has been scheduled for today (August 4, 2016) at 4:00PM Central Time.  The replacement process is estimated to take about 4 hours.   We are setting a conservative maintenance window for this event from 4:00PM – 9:00PM Central Time.

During this maintenance window, we will transfer A-side datacenter load to generator.   We do not anticipate any interruption to datacenter operations.  

Although we are operating within the rated load capacity of the transformer bank installation, we have expressed concerns to ONCOR that this is the second time in as many years that the transformer bank will have been replaced, and it is our position that the transformer bank should be uprated with relation to our load.  ONCOR has agreed and will be upgrading the delivery with larger-capacity transformers

Senior datacenter techs and electrical engineers will be on-hand for the duration of the event to ensure the operation proceeds as smoothly as possible.  In addition, we have notified both our fuel delivery company as well as our generator maintenance contractor, and they are standing by in the event that we need immediate assistance.

Updates and notifications will be made via our website at this link:

In addition, we will post brief updates as necessary and available to our social media feeds:
Twitter: @virtbiz
Facebook:  www.facebook.com/virtbiz

IDF 2805.12 switching failure (Resolved) High

Affecting System - IDF Core 12

  • 05/07/2016 20:43 - 05/07/2016 21:44
  • Last Updated 05/07/2016 21:44

UPDATE: 05/07/2016, 9:32PM CDT
We have resolved the issue at this time and do not foresee further trouble, but a network engineer will be reviewing and will advise if further action needs to be taken. We appreciate our customers' patience as we worked to resolve this trouble.

ORIGINAL POST: 05/07/2016, 8:43PM CDT
We are currently investigating a possible problem with IDF (switch) services at DAL1, row 2805.12.   Colocation customers and dedicated server customers with assets in this area may be experiencing an outage at this time.   Technicians are working as quickly as possible to bring resolution to this issue.

FPL-Fibernet Maintenance notification (Resolved) Low

Affecting Other - Peering Carrier

  • 05/10/2016 23:01 - 08/04/2016 12:30
  • Last Updated 05/03/2016 15:38

VIRTBIZ Internet Services is passing along the following information regarding one of our service providers:
FPL-Fibernet will be upgrading the OS software in their Service Routers and Switches. As a result of this maintenance, service across the FPL-Fibernet network will be impacted 1 or more times during this maintenance window as well as latency on the circuit. Each interruption might last up to 15 minutes, since the Network Elements will have to reboot for the new OS to become active. Internet connectivity will remain active at other peering points during the upgrade on each router.

This maintenance is non-service-impacting for VIRTBIZ customers, as traffic will be routed across other peers.  Therefore, this notification is purely informational in nature.

VPS system unavailability (Resolved) High

Affecting System - VPS Hosting Network

  • 04/06/2016 04:16
  • Last Updated 04/06/2016 06:44

UPDATE 6:53AM:  Root cause has been identified as a failure in cloud power distribution hardware.  This issue has been corrected and engineers have taken steps to prevent further recurance. 

ORIGINAL POST:
We are currently investigating unavailabilty within our VPS / Cloud hosting network that is causing some virtual systems to be unreachable.  Technicians are actively working to restore service.

Level3 BGP routing issue (Resolved) Medium

Affecting Other - DAL1 Datacenter

  • 04/05/2016 11:06 - 04/05/2016 11:58
  • Last Updated 04/05/2016 11:58

UPDATE: 11:55AM CDT

We have confirmed with Level3 that the incident is resolved and are closing this issue.  Please see the below for information from the Level3 NOC:
Outage Start: April 05, 2016 16:15 GMT
Outage Stop: April 05, 2016 16:31 GMT

Root Cause: An unreachable device in Dallas, TX was impacting IP services.
Fix Action: Tier III Technical Support made the necessary adjustments to restore reachability to the device.
Summary: The Level 3 IP NOC responded to an unreachable device in Dallas, TX that was impacting IP services, and engaged Tier III Technical Support to assist with the investigations. Tier III Technical Support then made the necessary adjustments to restore reachability to the device, and restore services.


ORIGINAL POST:
We are currently seeing BGP flap between core routing and Level3's peer.   Our engineers have performed preliminary diagnotics to rule out internal errors and have escalated the issue to Level3 NOC.
Internal ticket: 20160405L31122
Carrier ticket: 10633442

Datacenter routing is not negatively impacted as other carriers are continuing to run unimpaired.  Level3-bound traffic will automatically be routed to the next available carrier.

Datacenter Floor Maintenance (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 03/12/2016 10:00 - 04/11/2016 11:26
  • Last Updated 03/11/2016 11:49

Crews will be performing regular maintenance on the datacenter floor surface on Saturday, March 12.   Work is expected to begin at about 10:00AM CST. 

Work will include a deep cleaning and removal of any surface imperfections, polishing, and finishing with an anti-static coating.  This work is necessary to both promote a clean environment as well as reduce static electricity by use of a static-dissapation coating.

At various times during the maintenance, the application of cleaning agents or coatings will prevent foot traffic.   This work is being broken up into sections in order to minimize any possible disruption or inconvenience.   However, during the maintenance, the rear entry of the DAL1 facility will remain locked.   We ask that colocation customers requiring access enter via the front entrance on Saturday, March 12.   In addition, we appreciate your patience as some routine requests such as IP-KVM or media changes, may incur a brief delay if the area of the request is currently being serviced.

Thank you for your understanding as we continue working to bring you and your services the best environment possible.

UPS battery replacement (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 03/07/2016 09:29 - 03/07/2016 13:30
  • Last Updated 03/07/2016 12:23

UPDATE: 03/07/2016, 12:22PM: Work is complete, datacenter is returned to utility power, all systems normal.

UPDATE: 03/07/2016, 10:45AM:
Datacenter is operating on emergency (generator) power while UPS maintenance is underway.

ORIGINAL POST: Technicians from our UPS maintenance contractor will be on-site to replace the battery string (24 jars) on UPS2.  This is regularly scheduled preventative maintenance to ensure continued power reliability.  A battery string has a typical lifespan of 3-5 years and in accordance with best practices, we replace the batteries pro-actively so that a potential failed battery does not lead to an unplanned outage.  During this maintenance, old batteries will be removed and new batteries installed and tested.  VIRTBIZ has contracted to have the discarded batteries recycled in an environmentally concious manner. 

Although backup battery power will be removed from the system during this maintenance, every effort will be made to prevent disruption in service to our customers.  Full generator backup service remains in place throughout the duration of the maintenance.

Time Warner Cable peer down (Resolved) Medium

Affecting System - DAL1 / DTX901

  • 02/05/2016 01:41 - 02/05/2016 04:00
  • Last Updated 02/05/2016 06:38

ISSUE RESOLVED
Update 02/05/2016 03:48
We have received the following notification from the carrier:

Upon further investigation for this issue we have discovered you were not on the notification listing.  You ARE however being affected by the work.  This was due to Change Ticket CHG1700319.

This change was due to a fiber construction/relocation in that area around the AT&T Main Hall.  Fiber construction crews were splicing fibers to move to a different location at the time services were lost.

Outage Start Time:  02/5/2016  12:43:49 AM

Outage Stop Time:  02/5/2016  2:38:40 AM

Update 02/05/2016 02:33
TWC NOC reports the issue is due to CHG1700319 / TSK21859362. The maintenance will be completed before 6am.


Update 02/05/2016 02:03
TWC NOC staff is aware of the issue and is investigating.


Initial report 02/05/2016 01:41
Monitoring has detected peering with Time Warner to be down.   No impairment in service to users is expected as traffic is routing normally via other carriers.    Technicians are investigating.

Zayo peer - carrier maintenance (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 02/04/2016 00:10 - 02/04/2016 16:24
  • Last Updated 02/04/2016 00:40

Please be advised that the below scheduled maintenance activity will be commencing momentarily.

Maintenance Ticket #: TTN-0000915149

Maintenance Window: 00:00 - 05:00 Central

Primary Date: 04-Feb 2016

Backup Date: 05-Feb 2016


Location of Maintenance: 5205 Bryant Irvin Rd Bryant Irvin/ Trailview Fort Worth, TX

Reason for Maintenance: Zayo will perform maintenance to upgrade software/firmware on router mpr1-dfw1 to prevent future outages

Circuit IdExpected ImpactA Location CLLIZ Location CLLI
IPYX/107784//ZYO Hard Down - Up to 3 hours    


Server Inavailability (Resolved) Critical

Affecting Server - Matango

  • 02/21/2014 04:58
  • Last Updated 02/21/2014 09:57

We are aware of an availability issue with cPanel server "Matango". Staff is currently reviewing for resolution.

UPDATE: 9:06AM
cPanel hosting services on "Matango" are impacted by a cloud storage availability problem. We are working to correct ASAP.

UPDATE: 9:57AM
We have corrected the cloud storage availability issue. cPanel "Matango" and related services should be available at this time.

Carrier Maintenance Notification for 02/06/20 (Resolved) Medium

Affecting Other - Network

  • 02/04/2014 20:58 - 02/04/2016 04:31
  • Last Updated 02/04/2014 21:02

We have received notification from one of our carriers that they will be performing network maintenance later this week. 

This Thursday, February 6, 2014 the Zayo network may be unavailable for up to 10 minutes during the window of Midnight - 1:00AM CST.   The VIRTBIZ network will automatically route around the Zayo network during this window and there will be no loss or impairment of service.  However, it is possible that some users may experience brief apparent outages if their remote or 3rd party ISP is slow to pick up the BGP advertisement change.

Zayo reports the following:
===========================
Maintenance Ticket #: TTN-389231
Urgency: Demand
Maintenance Window: 00:01 - 01:00 Central
Primary Date: 6-Feb 2014
Backup Date: 7-Feb 2014
Location of Maintenance: Dallas TX
Reason for Maintenance: Zayo will perform demand maintenance to switch to secondary routing engine on er1.dfw2.
Expected Impact: SA (Service Affecting) - Down up to 10 Minutes
Service ID: t787
===========================

If you have any questions or concerns, we invite you to open a support ticket through our customer support portal at https://www.virtbiz.com/support and we will be glad to assist you.
 

Carrier outage, Time Warner (Resolved) Low

Affecting Other - routing peer: Time Warner

  • 08/02/2013 10:11
  • Last Updated 08/02/2013 17:59

VIRTBIZ NOC has removed routing peer with Time Warner due to interconnect problems impacting customers of Comcast.  Comcast customers have been able to pass ICMP traffic across the Time Warner peer but no TCP or UDP traffic.   VIRTBIZ networking assets and routing policies have been eliminated as possible contributors to this issue.  We have passed the case to the carrier for their review.  External Ticket 3292206

UPDATE 8/2/13 5:56PM CDT   A new circuit between TW and VIRTBIZ has been activated.  Peering is restored and routes are activated.

Datacenter Environmental (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 07/10/2013 08:05
  • Last Updated 08/02/2013 10:20

The following is an informational graph displaying datacenter temperatures as measured over time.  Samples are taken in five-minute intervals from monitoring stations throughout the datacenter.

Graph Row 16 Temperature (F)
Row 14 Temperature (F)
Row 12 Temperature (F)
Row 17 Temperature (F)
Row 10 Temperature (F)
Bakers Racks Temperature (F)
 
 
 
 
 
 
 
 
 
   
   



Datacenter Temperatures
COLOR Temperature (F) Row 16
COLOR Temperature (F) Row 14
COLOR Temperature (F) Row 12
COLOR Temperature (F) Row 17
COLOR Temperature (F) Row 10
COLOR Temperature (F) Bakers Racks

For reference, here are the current weather conditions: