Massive DDoS attacks on VoIP Providers and simulated DDoS testing

VoIP.ms and other VoIP providers under DDoS attack

At the time of writing, a major VoIP provider called VoIP.ms is under a distributed denial of service (DDoS) attack since over a week. As a result, they are unable to serve their customers with everyone and their dog complaining that they cannot connect to VoIP.ms’s SIP servers as well as other resources. At the same time, someone claiming to be part of the REvil ransomware group is blackmailing the provider.

This is not an isolated case, as earlier this month some UK providers were also under attack and the reports also mentioned REvil as the source of the attack. Various people from the security community have suggested that it is unlikely to be the real REvil but that is not what I’d like to write about here.

We did reach out to someone from one of the UK providers that were hit and he explained that in their sampling of the attack traffic, they did not see any SIP packets. Instead, what they did see were reflected DNS, SNMP and other traffic typically seen in amplification attacks and botnet DDoS attacks.

Not only about SIP denial of service

From reading public reports about these DDoS attacks, it is clear that the victim companies are not only suffering at their SIP endpoints. There are mentions of large amounts of traffic indicating that the attacks may be volumetric in nature rather than application-specific.

In the case of VoIP.ms, they initially had a DNS outage then moved to using Cloudflare for their DNS. They also put their website behind Cloudflare’s website DDoS protection and started asking for CAPTCHA etc.

They were also moving the IP addresses of their POP (point of presence) SIP servers and on the forums one can see people complaining about failing SIP connectivity. And lately, it has been noted that they are routing their SIP traffic through Cloudflare’s Magic Transit which provides UDP DDoS protection.

Yet they still seem to have intermittent connectivity, or appear to be offline at times.

Here’s a log from SIPVicious PRO’s SIP ping:

sipvicious sip utils ping udp://dallas2.voip.ms:5060
INFO[2021-09-24 12:41:47] pinging target (udp://dallas2.voip.ms:5060) with OPTIONS 
INFO[2021-09-24 12:41:48] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=1 time=268.91614ms 
INFO[2021-09-24 12:41:49] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=2 time=294.067084ms 
WARN[2021-09-24 12:41:51] timeout detected
INFO[2021-09-24 12:41:52] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=4 time=240.199604ms 
WARN[2021-09-24 12:41:54] timeout detected
INFO[2021-09-24 12:41:55] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=6 time=296.814325ms 
WARN[2021-09-24 12:41:57] timeout detected
WARN[2021-09-24 12:41:59] timeout detected
INFO[2021-09-24 12:42:00] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=9 time=251.79393ms 
INFO[2021-09-24 12:42:01] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=10 time=289.38613ms 
INFO[2021-09-24 12:42:02] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=11 time=307.496133ms 
INFO[2021-09-24 12:42:03] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=12 time=228.050179ms 
INFO[2021-09-24 12:42:04] 200 OK  from target: udp://dallas2.voip.ms:5060 cseq=13 time=207.497167ms 
^C
Interrupt signal received, shutting down.
Hit interrupt again for hard shutdown.
ERRO[2021-09-24 12:42:06] network error
INFO[2021-09-24 12:42:06] test complete
- target: udp://dallas2.voip.ms:5060
  - status: network error, network error
    results:
    - pingstats:
        14 SIP messages transmitted, 9 received, 4 error, 35.71% packet loss, time 18s
    issues:
      N/A

Not the first time that SIP servers have been attacked

Back in 2011, a botnet infected with malware called Sality started distributing sipvicious. Symantec wrote about this and we had blogged about this at the time on the SIPVicious blog. Essentially, the target SIP providers were unable to handle SIP cracking attacks that were being distributed. This had the same effect as a DDoS attack on the SIP servers, even if that might have been unintentional.

Attacking SIP is easy

Which brings us to the topic that is most interesting to me personally. What we discover when we do DoS and DDoS simulation exercises is that attacking SIP servers rarely requires volumetric attacks. Instead, SIP flood attacks will often overwhelm most SIP servers that have never been previously tested and therefore hardened against such attacks. With many of our clients, they already would have had protection against SIP DoS, yet we often find that their systems fail anyway as soon as we distribute our attacks. By distribute, I mean a handful of servers rather than the massive botnets that you see in actual attacks.

The main reason for such failures seems to be that protection mechanisms are often put in place without any realistic testing. Thus they tend to have gaps that we, during our DoS simulations end up abusing, causing the target services to stop responding to legitimate users.

That is why we are often successful in our tests. But there is another good question to ask: why do SIP servers fail so easily? The answer is: for various reasons - because SIP servers are complex.

Here are some of the things that often break:

• Authentication mechanisms, due to bottlenecks in the database storing the credentials
• Media servers running out of ports due to massive amount of SIP calls
• CPU and/or memory usage needed to handle SIP sessions and dialog exceeds the resources available
• Specific malformed SIP messages known to cause errors, filling up log files or logging servers
• On SIP over TCP or TLS, we encounter file descriptors resource exhaustion especially during a SIP INVITE flood attack

Most of these failure points are triggered without massive amounts of network traffic. And unfortunately, these attacks work rather well even when rate limiting is in place.

How do we test for SIP DDoS vulnerabilities?

As pentesters, we eat our own dog food, that is, make use of SIPVicious PRO’s SIP DoS Flood tool during our tests. During reconnaissance stage we do various tests to discover the most obvious weaknesses, by testing various different SIP message types for example. In the case of SIP INVITE floods, we also specify how the call is handled, whether to hangup the call at some point or not. The number of concurrent connections or which transport protocol to use for SIP is another important consideration. And finally, we choose a sending rate that runs below any protection mechanisms in place but higher than what the SIP server might withstand.

Then, we can distribute the attack.

For quick example of this, one can watch our unscripted Dangerous Demo for Kamailio World 2019 where we won the award of most entertaining demo.

Demo

Naturally, the person who setup the target machine did not have time to prepare for our denial of service tests. The system, running FreeSWITCH (I believe it was a FusionPBX installation) had no protection whatsoever. So this is not a fair example at all but it does show some of our tooling and, in part, how we do such tests.

Protection and mitigation

An excellent solution to all of this is to avoid exposing your critical infrastructure to Internet-based attackers in the first place. Simwood have a lot to say about that on various posts on their blog.

On the other hand, if your business model involves exposing your SIP servers to the Internet, the mitigation is less clear cut. So, we should first differentiate between volumetric attacks and application-level attacks.

The protection needed for volumetric attacks typically involves DDoS mitigation providers with large enough pipes to handle the massive bandwidth required to withstand such attacks. The attacks on VoIP.ms and the UK providers seem to fall under the category of volumetric attacks.

On the other hand, DDoS protection for attacks that are SIP (or application) specific, would involve tweaks to the infrastructure, architecture and underlying software configuration. Typically, one might place very well configured Kamailio / OpenSIPS (or similar) servers at the edge - perhaps configured to block excessive SIP traffic. For these changes to be effective, they need to be informed through the results achieved through DDoS simulation. This feedback loop is critical in hardening against such attacks.

If the attacks on VoIP.ms do in fact consist of SIP traffic in a volumetric attack, then it is likely that both protection/mitigation mechanisms are needed to actually address their problems. Which might explain why even behind Cloudflare’s Magic Transit, VoIP.ms seem to be still having major issues. But this is all just speculation at this point.

If you enjoyed this blog post, check out the next one called How I learned to stop worrying and love the flood.