SEARCH FINANCIAL SERVICES INFRASTRUCTURE SECURITY SCIENCE INTERVIEWS

 

     

State of Internet Health During COVID-19

By Angelique Medina, ThousandEyes

March 24, 2020

Now, more than ever, the Internet has become a critical lifeline for billions of people around the world impacted by COVID-19. A vast portion of the workforce is working remotely. Students at every level are learning online. Fitness classes, religious services, and medical appointments have gone digital. Social interactions are now mostly virtual. Consumer networks have effectively become our all-in-one tether to the world, connecting every one of us to critical services and each other — and this macro traffic shift has not gone unnoticed. Verizon has reported a 20% increase in overall traffic on its network in just the last week. Vodafone has reported a 50% increase.

So, how is the Internet holding up under the weight of this unprecedented (and unforeseen) level of usage? So far, reasonably well.

Despite massive traffic increases — particularly across consumer last-mile networks — we have not seen a significant corresponding spike in Internet outages, which can occur when traffic levels strain network capacity. However, there has been an upward trend line in outages over the last three weeks compared with the previous three-week baseline. We have also observed some performance degradation across key UCaaS services.

Network outages ISPs cloud UCaaS edge
Figure 1. Network outages within ISPs, public cloud, UCaaS, and edge service providers

To illustrate the current state of the Internet, we’re going to explore its health from three angles: ISPs, public cloud, and UCaaS performance.

Internet Service Providers

Looking at ISP outages within the last six weeks, we can see that the number of outages increased globally. Sporadic increases are not atypical under normal Internet conditions, as the number of daily and weekly outages can change for a variety of (mostly unpredictable) reasons. However, we have seen a concerning upward trajectory since the beginning of this month coinciding with the broader spread of COVID-19 and subsequent changes in Internet usage.

Global ISP outages Feb Mar 2020
Figure 2. Global ISP outages, Feb 10 to Mar 19, 2020

ISP outages rose even more dramatically in the United States, with an almost 100% increase between early February and the beginning of March. That early March level has been mostly sustained over the last couple of weeks.

United States ISP outages Feb Mar 2020
Figure 3. United States ISP outages, Feb 10 to Mar 19, 2020

While we’ve seen an increase in outages overall, two transit providers, in particular, Cogent Communications (AS 174) and Hurricane Electric (AS 6939) had very large-scale outages over this period.

On March 11th, Cogent experienced service disruptions for the fairly lengthy period (by Internet standards) of 30 minutes.

Cogent outage March 11 2020
Figure 4. Cogent outage on March 11, 2020

One week later, on March 18th, Cogent suffered a nearly identical outage in scope and length that impacted its ISP peers, as well as the reachability of services such as Amazon and LinkedIn.

Cogent outage March 18 2020
Figure 5. Cogent outage on March 18, 2020

Yet another notable outage occurred just after midnight last Friday, March 20th. The outage within Hurricane Electric’s network wasn’t as far-reaching or lengthy as either Cogent outage, but it was accompanied by smaller, intermittent disruptions that collectively impacted hundreds of sites and services.

Hurricane Electric outage March 2020
Figure 6. Hurricane Electric outage on Friday, March 20, 2020

Public Cloud Provider Networks

Major public cloud providers, such as AWS, Microsoft Azure, and Google Cloud, have built massive global networks that are incredibly well-equipped to handle traffic surges. So, unsurprisingly, we have seen almost no COVID-19 related impacts within their networks. Looking at the number of detected outages over the last six weeks, the trend line remains relatively level globally, with only a slight increase within the United States.

Public cloud outages global United States Feb Mar 2020
Figure 7. Public cloud provider network outages, global vs United States, Feb 10 to Mar 19, 2020

As with ISPs, cloud network outages are generally unpredictable, and, when large-scale in nature, are often due to a routing or infrastructure state change, rather than traffic congestion. Luckily for its enterprise customers and their users, cloud performance has held steady in the last couple of weeks.

Reachability and Performance of Top UCaaS Providers

Collaboration applications have seen a dramatic rise in usage over the last several weeks. Looking at the global performance of these services between March 9–20 compared to the previous two-week time span, we can see that the top three experienced relative performance impacts. Interestingly, one of the providers actually showed improvements in availability, latency, reduction in packet loss and jitter, while the other two showed minimal (in the grand scheme of things) degradations on all fronts — not surprising given the unprecedented strain they’ve been under.

UCaaS performance Mar 2020
Figure 8. Performance (03/09/2020 11:19 – 03/20/2020 12:19 UTC) vs previous time span

There were sporadic increases in traffic loss for users connecting to these services over the last two weeks, as seen in Figure 9.

UCaaS network loss Mar 2020
Figure 9. Network Loss – 03/09/2020 11:19 – 03/20/2020 12:19 UTC

Traffic loss could be caused by services and factors external to the provider, for example, if ISP networks neighboring these providers experience congestion or infrastructure issues. However, in the case of one of the providers, outages within its own network spiked last week, meaning that the network issues impacting users were taking place on infrastructure managed by the provider versus an external ISP. The outage incidents that took place this past week contributed to the 4% decline in availability of its service.

UCaaS provider United States global Feb Mar 2020
Figure 10. Provider 2 Outages, United States vs global, Feb 10 to Mar 19, 2020

Outage incidents within large UCaaS provider networks are fairly infrequent; however, the recent massive surge in usage is clearly stressing current design limits. Capacity is reportedly being added across the board to meet new service demands.

Not all performance issues we observed have been related to usage volume. For example, this past Friday, March 20th, many users were unable to reach one of the provider’s services because some of its domains were not resolving to an IP address.

UCaaS DNS resolution failure March 2020
Figure 11. DNS resolution failure led to service disruptions for some users

The disruption lasted approximately 20 minutes, during which time, its service was still operational, but simply not findable due to a DNS resolution failure. The issue appears to be related to a DNS configuration change made around that time. Once the misconfiguration was corrected, the reachability of its service resumed.

UCaaS applications will likely continue to see users and usage levels increase in the coming weeks and months, so we will continue to track and provide regular updates on their performance.

Introducing the ThousandEyes Global Internet Outages Map

In an effort to provide ongoing insight into the state of the global Internet, today we announced a publicly-available Global Internet Outages Map showing ongoing and recent network outages.

ThousandEyes Global Internet Outages Map Internet Insights
Figure 12. ThousandEyes Global Internet Outages Map, powered by Internet Insights™

The map is presented in near real time (updated every few minutes), so be sure to bookmark the page to stay updated on outages taking place in ISP, public cloud, UCaaS and edge service provider networks.

Moving Into the Internet Unknown

While — so far — the Internet is not buckling under the weight of huge traffic increases, it’s early days and no one can predict what will happen next. We are actively monitoring global Internet performance and, in addition to our Internet Map, we will be providing regular updates and analysis to help you continue to understand the state of the Internet under COVID-19.

Terms of Use | Copyright © 2002 - 2020 CONSTITUENTWORKS SM  CORPORATION. All rights reserved. | Privacy Statement