Why volumetric DDoS cripples VoIP providers and what we see during pentesting

or How I learned to stop worrying and love the flood

Published on Oct 13, 2021 in denial of service, voip security

An epiphany

Until a few days ago, I was of the opinion that simulating volumetric DDoS attacks is not something we should be doing. If you had asked us for such a test, we would have given you a negative answer.

Ironically, we had been unwittingly simulating volumetric DDoS attacks while quietly ignoring our own results. But, it’s time to stop neglecting bandwidth saturation and start giving it the attention that it deserves.

The recent VoIP DDoS attacks

So what changed my mind? To explain myself, I need to describe a bit what has been happening to some major VoIP providers in the past 2 months.

In my previous post, I explained that providers were being blackmailed after their services fell prey to ongoing DDoS attacks. Primarily, the attacks appeared to consist of traffic commonly generated by booter services. This was not specific to SIP. Instead, they were saturating the network bandwidth and server resources.

Since I wrote that post, more attacks were launched. Most concerning was the case of Bandwidth.com. They were targeted right after VoIP.ms and suffered downtime from 25th until around 30th September. During that time, 911 emergency calls passing through the victim provider along with other critical services were failing. This also affected many other providers that route their calls through Bandwidth, including Twilio, Accent, DialPad, Phone.com, and RingCentral.

How did they resolve their issues? Similar to VoIP.ms, they routed their traffic through Cloudflare’s Magic Transit.

Cloudflare’s staff wrote two valuable blog posts where they shared details on the sort of traffic that they saw during these attacks. The first post explained how most of the malicious traffic was coming from DNS reflection and other common amplification and reflection vectors. They also gave out some numbers, most interestingly, the malicious traffic peaking at 130 Gbps and 17.4 million packets per second. And naturally, they highlighted their offers and technology - for example, that they have created filtering for invalid traffic on the SIP targets. In the second post, what I found interesting is that they started seeing other traffic, including SIP protocol-specific attacks.

The cyber-criminals behind this are now back to targeting VoIP Unlimited, a UK provider. This victim was previously targeted back in late August and early September. At the time of writing, not only do their SIP services appear to be unreachable, but also their websites. So, while the attacks have not stopped, they have shifted again to a different target.

VoIP Unlimited’s website was down at the time of writing

Our VoIP DDoS testing experience

So what does this have to do with our denial of service security tests? A lot, actually.

Our focus with DDoS simulation has been on non-volumetric attacks aimed at the applications on test. We consider these attacks as far more sophisticated than a simple UDP flood when attacking specific behaviors of the applications handling SIP, RTP and other protocols used for real-time communications. Proudly, we would start our attacks on say, a SIP target, only to find that the network is saturated with our traffic.

So we learned to ask our clients about their bandwidth limitations to avoid overwhelming their pipes. We would also slowly ramp up our tests so that if unknown limitations are suddenly discovered, they can be detected early. And then, in such cases, we would run our tests using the rate limiter built into our SIP flooding tool (and almost any tool part of SIPVicious PRO).

Then, we would restrict our traffic to stay under a given percentage of their total bandwidth. We have seen bandwidth limits ranging from 10Gbit/second to 1Gbit/second down to 100Mbit/second. The tests that we do are usually not for smaller providers but rather for some of the heavyweights who have sizable security budgets.

Naturally, mobsters pay no respect and will blast with full capacity during an actual attack.

A light bulb moment

By hitting such bandwidth limits during our penetration tests, we were discovering the same thing that the extortionists behind these attacks found for themselves: for VoIP providers, volumetric DDoS is a thing. This vulnerability appears to affect many VoIP providers - including some of the major players.

The cost of running such an attack using legitimate VPS servers is just a few cents. In the case of a botnet, booter services and other black-market offers, the price tags are considerably lower. In many cases, demonstrating this vulnerability is straightforward but unfortunately, bitter experience is what seems to be moving the needle.

That brings me to the conclusion that application-level DDoS attacks are the lesser evil. Until providers acknowledge that bandwidth saturation is indeed a critical issue, it will not be mitigated.

From our part, I know we will be making sure to explicitly check for and demonstrate these vulnerabilities instead of taking a move along, nothing to see approach.

Note

As part of our security audit and penetration tests, we simulate DDoS attacks on RTC applications and infrastructure. Talk to us to learn more.

Background information

Timelines

Sep 16, 2021 - problems with VoIP.ms started
Sep 24, 2021 - VoIP.ms started passing traffic through CF
Sep 25, 2021 - Bandwidth.com started reporting issues
Sep 29, 2021 - Bandwidth.com started passing traffic through CF
Sep 30, 2021 - Bandwidth indicated that they’re back to normal
Oct 08, 2021 - Attackers back to targeting VoIP Unlimited