There are no Open Network Issues Currently

Network Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Asset HTTP FTP POP3 PHP Info Server Load Online
Ave1 PHP Info
Ave2 PHP Info
Ave3 PHP Info
Matango PHP Info
opus24 PHP Info
opus25 PHP Info
Rodan PHP Info
Stats PHP Info
Web38 PHP Info
Web39 PHP Info
Web40 PHP Info
Web41 PHP Info
Web42 PHP Info
Web43 PHP Info
Webhosting1 PHP Info
Webhosting2 PHP Info
Webhosting3 PHP Info
Service Interruption 05/10/2021 (Resolved) Critical

Affecting Other - DAL1 Datacenter

  • 05/10/2021 18:11 - 05/18/2021 22:02
  • Last Updated 05/15/2021 11:01

Update: 05/15/2021  10:49

Work has been completed on the below referenced Emergency Maintenance Window.    All systems are normal at this time and no further interruptions are expected.    We thank our customers for your patience.

 


Update: 05/14/2021 11:49

Preparations have been made to decomission the UPS responsible for Monday's power failure that impacted multiple customers at our DAL1 facility and transition affected systems to replacement power facilities.

EMERGENCY MAINTENANCE WINDOW
impacting customers with service from Panels L5, L8, L14 and RTR2
Saturday, May 15, 2021
7:30am - 12:00 Noon

If your service is expected to be impacted by this work, you have already received email notification.   If you did not receive an email notification, your service is not believed to be impacted by this maintenance window.

All preparations have already been made, construction is complete and systems have been pre-wired as far as practical.   The work performed on Saturday will be confined to physically changing the high-voltage connections to service distribution panels.  Through extensive planning, careful engineering and workshopping with our technical staff, we are confident that the actual work will go very quickly.   Therefore, we consider the maintenance window timeframe to be conservative, and we hope to complete our work much earlier.  In any event, we anticipate the actual outage to any customer equipment should be 15 minutes or less.  

Since we will be working with high-voltage electricity, we will be restricting entry to the facility throughout the duration of the maintenance.    Customers will not be permitted inside the datacenter while work is underway.   This is to prevent possible injury to our customers and also eliminate distractions to the technicians who will be performing the work.

Once service is restored to each panel, technicians will proceed with crash-carts and connect to each impacted system that has video output in order to observe a proper boot cycle.   After that is complete, we will begin working customer tickets. 

 


Update: 05/11/2021 13:41

Staff, management and contracting engineers have diagnosed the service-impacting issue and are currently working to fully rectify the trouble.   One of our main power backup systems has experienced an internal fault.   This UPS installation (the engineering, but not the actual hardware) is part of the original datacenter build from 2007 and has become functionally obsolete due to its inability to provide adequate load for our current (and future) requirements.   Therefore, we had planned to decomission the system in Q4 2021 or Q1 2022 after moving existing services to new power distribution.   Given the serious nature of the failure that has occurred, we will now be accelerating that replacement to an immediate timeline.  

Work is already underway and is progressing as quickly as materials can be delivered and installed.   We have already taken steps to isolate our core network infrastructure from the impacted system.    Customer racks and 3rd party network service providers will be moved as soon as practical.   Notifications will be sent out accordingly when power will be cut-over.  

We appreciate your patience as we carry out this work.   We recognize the critical nature of the service that we provide to our customers and everything we do is focused on that.

 


Initial report: 05/10/2021 18:11

We are aware of an event that caused an interruption in routing service to DAL1 datacenter from 5:45pm to 5:52pm Central Time.

The issue appears to be related to a catastrophic systems failure at one of the 480V 3-phase UPS systems.   Our UPS service company has already dispatched a technician to review this.   At first glance it appears there has been an internal component that has failed, causing the load to momentarily trip offline.   We are still investigating to determine what, precisely, has happened and what steps will be taken going forward.   At this time our technicians have mitigated the issue with alternate power routing.

We appreciate your patience and understanding as we take the necessary steps of properly diagnosing, then permanently correcting the issue and apologize for any inconvenience from the impact of this issue.

 

 

Web hosting server upgrades in progress (Resolved) Medium

Affecting Server - Rodan

  • 09/09/2020 21:00 - 09/24/2020 21:33
  • Last Updated 09/10/2020 07:55

In order to provide the best web hosting experience possible, VIRTBIZ is performing upgrades to certain infrastructure.   During this time, it may be periodically necessary to temporarily suspend some services in order to ensure a smooth transition.   cPanel-based web-hosting customers on the "Rodan" server may experience brief periods of unavailability for various services including website and email functions.    Please be assured that the automated process responsible will not interrupt services for longer than absolutely necessary.   We appreciate your patience during this time.

Network Maintenance 09-14-2019 (Resolved) Critical

Affecting System - DAL1 Datacenter

  • 09/14/2019 11:24 - 09/15/2019 01:30
  • Last Updated 09/14/2019 11:29

We have identified software errors that may contribute to unscheduled outages or service degradation.  In order to mitigate ill effects, engineers will engage in updating and replacing some critical network infrastructure.   We are taking steps to minimize impact but due to significant engineering changes, some interruptions are possible.

Start time: 12:00 PM (CDT) 09/14/2019
End time: 23:59 PM (CDT) 09/14/2019
Expected Outage/Downtime: 10 minutes (intermittent)

During this maintenance window, customers may experience brief interruptions in service while we complete the maintenance activities. We do not expect extended periods of connectivity loss. However, due to the complexity of the work, it is possible that unforeseen issues could result in temporary connectivity loss. Customers may also expect to see some re-convergence during the maintenance window. Customers may experience latency and packet loss intermittently throughout the window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please contact our Support Team for assistance.

We apologize in advance for any inconvenience.   We are working as diligently as possible to ensure reliability of service to our customers.

###

DAL1 Routing Maintenance Window (Resolved) High
  • 06/09/2019 16:00 - 09/14/2019 11:24
  • Last Updated 06/09/2019 21:44

UPDATE: 06/09/2019, 9:31PM

In the announced maintenance window for 06/19 we implemented new RaaS to improve routing performance.     Unfortunately the tested configuration did run but apparently had some issue with LACP / LAG functionality that caused the network to perform erratically.   Our engineers monitored this and engaged our vendor support and attempted to resolve but the LAG ports continued to reset. This eventually lead to memory problems in the MDF's handling customer routes.   At that point, hosted assets were effectively disconnected from the core.    We are working with the vendor to troubleshoot and perform regression testing against our configuration.  At this time we have rolled back to the previous infrastructure pending successful testing and sign-off with our vendor.


In order to promote ongoing network stability, VIRTBIZ will be performing critical maintenance to update and upgrade core routing infrastructure at the DAL1 datacenter location.  This work includes physical replacement of core routing assets that supply border and edge connectivity.  Equipment and connections are already in place and a procedure has been established in order to minimize impact to end-users.

Start time: 4:00 PM (CDT) 06/09/2019
End time: 6:00 PM (CDT) 06/09/2019
Expected Outage/Downtime: 10 minutes (intermittent)

During this maintenance window, customers may experience up to two brief interruptions in service while we complete the maintenance activities. We do not expect extended periods of connectivity loss. However, due to the complexity of the work, it is possible that unforeseen issues could result in temporary connectivity loss. Customers may also expect to see some re-convergence during the maintenance window. Customers may experience latency and packet loss intermittently throughout the window.

Our network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please contact our Support Team for assistance.

###

Route flap impacting certain DAL1 connections (Resolved) Critical

Affecting Other - DAL1 Datacenter

  • 06/04/2019 10:22 - 09/14/2019 11:24
  • Last Updated 06/04/2019 10:29

Engineers have identified a route flap issue impacting some connected routes are currently working with our vendors for full resolution.  

MEM1 power event (Resolved) Critical

Affecting Other - MEM1 (Tennessee) datacenter

  • 08/05/2016 09:37 - 08/05/2016 11:30
  • Last Updated 08/05/2016 12:40

Technicians are currently investigating a power interruption and loss of routing at MEM1 (Dyersburg, TN) facility.   At this time, colocated customers at MEM1 are inaccessible from the Internet.   This is a facility issue and not a VIRTBIZ routing failure, and facility technicians are on-site working to resolve the problem.

UPDATE 11:33AM
Investigations have revealed a cascade of failures that contributed to nearly 90 minutes of service interruption.  When the power failed, load was handled by UPS (battery) systems while the generator was auto-started.   Although the generator started as expected, the automatic transfer switch (ATS) did not transfer the load to generator power.  Accordingly, UPS units shut down as batteries were exhausted, causing the load to drop.   As power to the load was restored manually, an intermediate distribution switch that connects our router to the facility network failed to properly start and load.   As the switch is owned and maintained by the facility (not VIRTBIZ) their staff was required to bypass/replace the failed unit.  Once they restored their network connectivity, all VIRTBIZ assets became visible on the network.  

Our partners at the facility are conducting a post-mortem of this event in conjunction with their electrician and generator maintenance provider.   VIRTBIZ is standing by to assist the facility with their investigation as needed.

Emergency electric maintenance - 08/04/2016 (Resolved) High

Affecting Other - DAL1 Datacenter

  • 08/04/2016 16:00 - 08/04/2016 21:00
  • Last Updated 08/04/2016 20:19

EMERGENCY MAINTENANCE NOTIFICATION
Start time: 4:00PM Central 08/04/2016
End time:  9:00PM Central 08/04/2016
Expected Outage/Downtime:  n/a

UPDATE, 8:00PM
Oncor crews worked quickly to assist in the replacement of the malfunctioning transformers.  Datacenter operations were unimpacted during the maintenance period.  The window was closed as of 6:15PM Central.

ORIGINAL POST:
During routine inspections this morning, technicians noted moisture and discoloration around one phase of our primary (A-side) electric service entrance.   This indicates a failure of the transformer and points to a pending imminent power failure.   Upon notification of this issue, we made contact with the electrical transmission provider, ONCOR, and have arranged for replacement of the entire transformer bank.

Scheduling the maintenance window involves balancing the risk of unplanned transformer failure or fire with the desire to utilize off-peak hours as much as possible.  Accordingly, replacement has been scheduled for today (August 4, 2016) at 4:00PM Central Time.  The replacement process is estimated to take about 4 hours.   We are setting a conservative maintenance window for this event from 4:00PM – 9:00PM Central Time.

During this maintenance window, we will transfer A-side datacenter load to generator.   We do not anticipate any interruption to datacenter operations.  

Although we are operating within the rated load capacity of the transformer bank installation, we have expressed concerns to ONCOR that this is the second time in as many years that the transformer bank will have been replaced, and it is our position that the transformer bank should be uprated with relation to our load.  ONCOR has agreed and will be upgrading the delivery with larger-capacity transformers

Senior datacenter techs and electrical engineers will be on-hand for the duration of the event to ensure the operation proceeds as smoothly as possible.  In addition, we have notified both our fuel delivery company as well as our generator maintenance contractor, and they are standing by in the event that we need immediate assistance.

Updates and notifications will be made via our website at this link:

In addition, we will post brief updates as necessary and available to our social media feeds:
Twitter: @virtbiz
Facebook:  www.facebook.com/virtbiz

IDF 2805.12 switching failure (Resolved) High

Affecting System - IDF Core 12

  • 05/07/2016 20:43 - 05/07/2016 21:44
  • Last Updated 05/07/2016 21:44

UPDATE: 05/07/2016, 9:32PM CDT
We have resolved the issue at this time and do not foresee further trouble, but a network engineer will be reviewing and will advise if further action needs to be taken. We appreciate our customers' patience as we worked to resolve this trouble.

ORIGINAL POST: 05/07/2016, 8:43PM CDT
We are currently investigating a possible problem with IDF (switch) services at DAL1, row 2805.12.   Colocation customers and dedicated server customers with assets in this area may be experiencing an outage at this time.   Technicians are working as quickly as possible to bring resolution to this issue.

FPL-Fibernet Maintenance notification (Resolved) Low

Affecting Other - Peering Carrier

  • 05/10/2016 23:01 - 08/04/2016 12:30
  • Last Updated 05/03/2016 15:38

VIRTBIZ Internet Services is passing along the following information regarding one of our service providers:
FPL-Fibernet will be upgrading the OS software in their Service Routers and Switches. As a result of this maintenance, service across the FPL-Fibernet network will be impacted 1 or more times during this maintenance window as well as latency on the circuit. Each interruption might last up to 15 minutes, since the Network Elements will have to reboot for the new OS to become active. Internet connectivity will remain active at other peering points during the upgrade on each router.

This maintenance is non-service-impacting for VIRTBIZ customers, as traffic will be routed across other peers.  Therefore, this notification is purely informational in nature.

VPS system unavailability (Resolved) High

Affecting System - VPS Hosting Network

  • 04/06/2016 04:16
  • Last Updated 04/06/2016 06:44

UPDATE 6:53AM:  Root cause has been identified as a failure in cloud power distribution hardware.  This issue has been corrected and engineers have taken steps to prevent further recurance. 

ORIGINAL POST:
We are currently investigating unavailabilty within our VPS / Cloud hosting network that is causing some virtual systems to be unreachable.  Technicians are actively working to restore service.

Level3 BGP routing issue (Resolved) Medium

Affecting Other - DAL1 Datacenter

  • 04/05/2016 11:06 - 04/05/2016 11:58
  • Last Updated 04/05/2016 11:58

UPDATE: 11:55AM CDT

We have confirmed with Level3 that the incident is resolved and are closing this issue.  Please see the below for information from the Level3 NOC:
Outage Start: April 05, 2016 16:15 GMT
Outage Stop: April 05, 2016 16:31 GMT

Root Cause: An unreachable device in Dallas, TX was impacting IP services.
Fix Action: Tier III Technical Support made the necessary adjustments to restore reachability to the device.
Summary: The Level 3 IP NOC responded to an unreachable device in Dallas, TX that was impacting IP services, and engaged Tier III Technical Support to assist with the investigations. Tier III Technical Support then made the necessary adjustments to restore reachability to the device, and restore services.


ORIGINAL POST:
We are currently seeing BGP flap between core routing and Level3's peer.   Our engineers have performed preliminary diagnotics to rule out internal errors and have escalated the issue to Level3 NOC.
Internal ticket: 20160405L31122
Carrier ticket: 10633442

Datacenter routing is not negatively impacted as other carriers are continuing to run unimpaired.  Level3-bound traffic will automatically be routed to the next available carrier.

Datacenter Floor Maintenance (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 03/12/2016 10:00 - 04/11/2016 11:26
  • Last Updated 03/11/2016 11:49

Crews will be performing regular maintenance on the datacenter floor surface on Saturday, March 12.   Work is expected to begin at about 10:00AM CST. 

Work will include a deep cleaning and removal of any surface imperfections, polishing, and finishing with an anti-static coating.  This work is necessary to both promote a clean environment as well as reduce static electricity by use of a static-dissapation coating.

At various times during the maintenance, the application of cleaning agents or coatings will prevent foot traffic.   This work is being broken up into sections in order to minimize any possible disruption or inconvenience.   However, during the maintenance, the rear entry of the DAL1 facility will remain locked.   We ask that colocation customers requiring access enter via the front entrance on Saturday, March 12.   In addition, we appreciate your patience as some routine requests such as IP-KVM or media changes, may incur a brief delay if the area of the request is currently being serviced.

Thank you for your understanding as we continue working to bring you and your services the best environment possible.

UPS battery replacement (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 03/07/2016 09:29 - 03/07/2016 13:30
  • Last Updated 03/07/2016 12:23

UPDATE: 03/07/2016, 12:22PM: Work is complete, datacenter is returned to utility power, all systems normal.

UPDATE: 03/07/2016, 10:45AM:
Datacenter is operating on emergency (generator) power while UPS maintenance is underway.

ORIGINAL POST: Technicians from our UPS maintenance contractor will be on-site to replace the battery string (24 jars) on UPS2.  This is regularly scheduled preventative maintenance to ensure continued power reliability.  A battery string has a typical lifespan of 3-5 years and in accordance with best practices, we replace the batteries pro-actively so that a potential failed battery does not lead to an unplanned outage.  During this maintenance, old batteries will be removed and new batteries installed and tested.  VIRTBIZ has contracted to have the discarded batteries recycled in an environmentally concious manner. 

Although backup battery power will be removed from the system during this maintenance, every effort will be made to prevent disruption in service to our customers.  Full generator backup service remains in place throughout the duration of the maintenance.

Time Warner Cable peer down (Resolved) Medium

Affecting System - DAL1 / DTX901

  • 02/05/2016 01:41 - 02/05/2016 04:00
  • Last Updated 02/05/2016 06:38

ISSUE RESOLVED
Update 02/05/2016 03:48
We have received the following notification from the carrier:

Upon further investigation for this issue we have discovered you were not on the notification listing.  You ARE however being affected by the work.  This was due to Change Ticket CHG1700319.

This change was due to a fiber construction/relocation in that area around the AT&T Main Hall.  Fiber construction crews were splicing fibers to move to a different location at the time services were lost.

Outage Start Time:  02/5/2016  12:43:49 AM

Outage Stop Time:  02/5/2016  2:38:40 AM

Update 02/05/2016 02:33
TWC NOC reports the issue is due to CHG1700319 / TSK21859362. The maintenance will be completed before 6am.


Update 02/05/2016 02:03
TWC NOC staff is aware of the issue and is investigating.


Initial report 02/05/2016 01:41
Monitoring has detected peering with Time Warner to be down.   No impairment in service to users is expected as traffic is routing normally via other carriers.    Technicians are investigating.

Zayo peer - carrier maintenance (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 02/04/2016 00:10 - 02/04/2016 16:24
  • Last Updated 02/04/2016 00:40

Please be advised that the below scheduled maintenance activity will be commencing momentarily.

Maintenance Ticket #: TTN-0000915149

Maintenance Window: 00:00 - 05:00 Central

Primary Date: 04-Feb 2016

Backup Date: 05-Feb 2016


Location of Maintenance: 5205 Bryant Irvin Rd Bryant Irvin/ Trailview Fort Worth, TX

Reason for Maintenance: Zayo will perform maintenance to upgrade software/firmware on router mpr1-dfw1 to prevent future outages

Circuit IdExpected ImpactA Location CLLIZ Location CLLI
IPYX/107784//ZYO Hard Down - Up to 3 hours    


Server Inavailability (Resolved) Critical

Affecting Server - Matango

  • 02/21/2014 04:58
  • Last Updated 02/21/2014 09:57

We are aware of an availability issue with cPanel server "Matango". Staff is currently reviewing for resolution.

UPDATE: 9:06AM
cPanel hosting services on "Matango" are impacted by a cloud storage availability problem. We are working to correct ASAP.

UPDATE: 9:57AM
We have corrected the cloud storage availability issue. cPanel "Matango" and related services should be available at this time.

Carrier Maintenance Notification for 02/06/20 (Resolved) Medium

Affecting Other - Network

  • 02/04/2014 20:58 - 02/04/2016 04:31
  • Last Updated 02/04/2014 21:02

We have received notification from one of our carriers that they will be performing network maintenance later this week. 

This Thursday, February 6, 2014 the Zayo network may be unavailable for up to 10 minutes during the window of Midnight - 1:00AM CST.   The VIRTBIZ network will automatically route around the Zayo network during this window and there will be no loss or impairment of service.  However, it is possible that some users may experience brief apparent outages if their remote or 3rd party ISP is slow to pick up the BGP advertisement change.

Zayo reports the following:
===========================
Maintenance Ticket #: TTN-389231
Urgency: Demand
Maintenance Window: 00:01 - 01:00 Central
Primary Date: 6-Feb 2014
Backup Date: 7-Feb 2014
Location of Maintenance: Dallas TX
Reason for Maintenance: Zayo will perform demand maintenance to switch to secondary routing engine on er1.dfw2.
Expected Impact: SA (Service Affecting) - Down up to 10 Minutes
Service ID: t787
===========================

If you have any questions or concerns, we invite you to open a support ticket through our customer support portal at https://www.virtbiz.com/support and we will be glad to assist you.
 

Carrier outage, Time Warner (Resolved) Low

Affecting Other - routing peer: Time Warner

  • 08/02/2013 10:11
  • Last Updated 08/02/2013 17:59

VIRTBIZ NOC has removed routing peer with Time Warner due to interconnect problems impacting customers of Comcast.  Comcast customers have been able to pass ICMP traffic across the Time Warner peer but no TCP or UDP traffic.   VIRTBIZ networking assets and routing policies have been eliminated as possible contributors to this issue.  We have passed the case to the carrier for their review.  External Ticket 3292206

UPDATE 8/2/13 5:56PM CDT   A new circuit between TW and VIRTBIZ has been activated.  Peering is restored and routes are activated.

Datacenter Environmental (Resolved) Low

Affecting Other - DAL1 Datacenter

  • 07/10/2013 08:05
  • Last Updated 08/02/2013 10:20

The following is an informational graph displaying datacenter temperatures as measured over time.  Samples are taken in five-minute intervals from monitoring stations throughout the datacenter.

Graph Row 16 Temperature (F)
Row 14 Temperature (F)
Row 12 Temperature (F)
Row 17 Temperature (F)
Row 10 Temperature (F)
Bakers Racks Temperature (F)
 
 
 
 
 
 
 
 
 
   
   



Datacenter Temperatures
COLOR Temperature (F) Row 16
COLOR Temperature (F) Row 14
COLOR Temperature (F) Row 12
COLOR Temperature (F) Row 17
COLOR Temperature (F) Row 10
COLOR Temperature (F) Bakers Racks

For reference, here are the current weather conditions: