Killing bugs … one vulnerability report at a time

Executive summary (TL;DR)

We tell the story behind the latest FreeSWITCH advisories and how it all came together one sleepless night in April 2021 so that we ended up with 4 vulnerabilities that needed reporting. And then, one more vulnerability found due to a bug in our own software, SIPVicious PRO. We explain how these flaws were discovered, reported, fixed and what we ultimately learned through this process.

What is this about?

This article shares the same title with the presentation that I gave at ClueCon 2021, organised by SignalWire - main FreeSWITCH sponsors. The presentation can be watched on Youtube.

On the 25th of October 2021 we released five advisories concerning vulnerabilities that were fixed in FreeSWITCH 1.10.7. The advisories that we published were the following:

• FreeSWITCH vulnerable to SIP digest leak for configured gateways
• FreeSWITCH susceptible to Denial of Service via SIP flooding
• FreeSWITCH does not authenticate SIP MESSAGE requests, leading to spam and message spoofing
• FreeSWITCH does not authenticate SIP SUBSCRIBE requests by default
• FreeSWITCH susceptible to Denial of Service via invalid SRTP packets

With two of them being quite similar, I originally thought that I would be telling 4 different stories, one for each security issue. I even came up with some fancy titles for each story:

• SIP digest leak but not as you (might) know it
• The mystery of the anonymous SIP messages
• Foom! Flooded out of memory
• Your encrypted call is important to us

But as I started working on this article, I realized something - the events that led to each vulnerability report were very much intertwined. What I noticed was that I could not talk about the process of finding one bug without mentioning the other.

So this is the story of how the security issues that we reported to the FreeSWITCH team were discovered, reported and fixed. At the end, you’ll find some of the lessons that we, as security researchers learned as well as some takeaways for open-source software developers.

The use of I and we

While reading, please keep in mind that I use I and we interchangeably because this is the work of the Enable Security team. Since this story spans a couple of years, apart from the usual suspects, we had assistance from Pinaki Mondal for some months in 2020 who was an intern with us.

How the process of vulnerability discovery is linked to developing SIPVicious PRO

Before delving into our story, I should give a bit of background about the security toolset that we’re developing called SIPVicious PRO. All the vulnerabilities that we found in FreeSWITCH were the result of working on our professional toolset. It is a completely new set of tools - entirely different from the open-source SIPVicious. One main difference is that while SIPVicious OSS focuses on SIP, with SIPVicious PRO we aim to cover Real-Time Communications security in general. With that out of the way, it is time for a story.

A single, longer story instead of four individual tales

Starting position: ignorance

Until recently, we had no practical experience with FreeSWITCH. It was always on our radar but we never actually sat down and configured one properly. One reason was that none of our main clients seemed to use it in the products or services that we were actually pentesting. We did, however, come across a few hints of potential vulnerabilities during the years.

For example, in 2018 we wrote this bash script:

When we gave it a run, we saw the following:

2018-03-25 20:56:55.095342 [NOTICE] switch_core_state_machine.c:690 Hangup sofia/internal/1000@172.17.0.2:5060 [CS_NEW][WRONG_CALL_STATE]
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 850 (sofia/internal/1000@172.17.0.2:5060) Ended
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 847 (sofia/internal/1000@172.17.0.2:5060) Ended
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1683 Session 849 (sofia/internal/1000@172.17.0.2:5060) Ended
2018-03-25 20:56:55.095342 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1683 Session 848 (sofia/internal/1000@172.17.0.2:5060) Ended
2018-03-25 20:56:55.115329 [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/1000@172.17.0.2:5060 [CS_DESTROY]
./__entrypoint.sh: line 15:    11 Killed                  /usr/local/freeswitch/bin/freeswitch

A year later, we were testing our DDoS simulator and decided to see how it works against FreeSWITCH. This time, we had the following result:

2019-03-27 15:42:20.812017 [CONSOLE] switch_core.c:2455 
[This app Best viewed at 160x60 or more..]
2019-03-27 15:42:20.814316 [INFO] switch_time.c:615 Clock synchronized to system time.
freeswitch@ddos-simulator> Killed
root@ddos-simulator:/usr/local/freeswitch/bin# 

Then in May 2019 we were at Kamailio World and demonstrated our DDoS simulator against a target that someone else had setup. Previously unknown to us, the target happened to be a FusionPBX which is based on FreeSWITCH. The target did not last beyond a few seconds. The quick video can be seen here:

After the event, we were naturally curious as to what had happened during our live session. So, a few months later, we decided to look into FusionPBX and installed it on a VPS and started playing with that. Then, while testing our SIP flood tool, this happened:

But there was something else this time. Once the system recovered, after a reboot, I ran our SIP method enumeration tool. That is when I saw the following:

➜  sipvicious sip enum methods udp://target:5060
...
- target: udp://target:5060
  - status: security issue, insecure method/s detected
    results:
    - responses:
        REGISTER - supported (401 Unauthorized)
        SUBSCRIBE - supported (202 Accepted)
        NOTIFY - supported (481 Subscription Does Not Exist)
        PUBLISH - supported (200 OK)
        MESSAGE - supported (202 Accepted)
        INVITE - supported (407 Proxy Authentication Required)
        OPTIONS - supported (200 OK)
        BYE - supported (481 Call Does Not Exist)
        CANCEL - supported (481 Call/Transaction Does Not Exist)
        ACK - not supported (no response)
        PRACK - supported (481 No such response)
        INFO - supported (200 OK)
        REFER - supported (420 Bad Extension)
        UPDATE - supported (481 Call Does Not Exist)
    issues:
    - insecure-methods:
        method: SUBSCRIBE, status: 202
        method: PUBLISH, status: 200
        method: MESSAGE, status: 202
        method: INFO, status: 200

This tool, part of SIPVicious PRO, sends different SIP messages to a target and tells you how the target responds to each SIP message. If any of the responses include a 200 or 202 answer, that means that that particular SIP message does not require authentication. This is often not ideal for anything other than as a response to OPTIONS; especially if the system is exposed on the Internet.

Time for research

So, while these were all red flags, we needed more understanding.

We had all sorts of questions. Were we doing anything wrong in our tests? What is the impact of these issues? Is this really how organisations are using FreeSWITCH? Is this a real-world problem or just something that you see in lab environment? Are these issues specific to FusionPBX? And finally, who can we speak to?

Back in 2017, we had contacted someone associated with the FreeSWITCH team and submitted a security report but nothing came out of it. Naturally, that was discouraging.

But then, we got very busy.

It was in 2020 when we started learning about the FreeSWITCH configuration options related to authentication. Around July we had a working installation of FreeSWITCH. With this simple setup, we could see for ourselves that sending a MESSAGE request did indeed allow spoofing of messages and SUBSCRIBE allowed monitoring of call events. And that worked without requiring any authentication by default.

It looked something like the following demo:

In the video, we have Zoiper registered with FreeSWITCH. And, the attacker can just send a MESSAGE to FreeSWITCH which routes it - without any authentication - to the softphone. The from address can be anything and so we set it to “Pancake Monster” and sent some pancake spam.

It was around this time that we noticed that a potential client of ours did not authenticate these same SIP methods. Although not obvious at the time, the reason was that they were using FreeSWITCH.

So the question was, is this really widespread? Anyone could scan the Internet or use Shodan to find the IPs of exposed FreeSWITCH systems across the world. Then, they could run some simple tests and realize that this indeed is a common problem.

But then, once again, we got very busy.

In late March 2021, FreeSWITCH 1.10.6 was released and I remembered that we really should learn FreeSWITCH. I was curious to see if this version had changed any of the behaviours that we had observed. So we started peeking into the code and checking out the changes.

The main deflection point was on the night of the 7th and 8th April. I had a tough time sleeping and so, I did what any other hacker would do - I tried to figure out if the authentication issues were fixed. What I noticed was that the new release came with a new flag called auth-subscriptions that was now included in the default configuration. This was definitely good but not good enough. Anyone upgrading would keep using their old configuration file and the problem would not be fixed on their system.

This must have frustrated me and I got bored but also curious. I thought, why not test our digest leak tool? This tool starts a call using SIP INVITE and when the other party tries to hangup with a BYE message, it asks the other party to authenticate by sending a 407 SIP response with a challenge.

In this case, somewhat unsurprisingly, nothing appeared to happen. But instead, the following line in the logs caught my eye:

2021-04-07 22:43:29.502943 [WARNING] sofia_reg.c:2642 Cannot locate any authentication credentials to complete an authentication request for realm '"192.168.188.128"'

This indicated that FreeSWITCH did actually care about my 407 response but did not find credentials that match. What would happen if it did find matching credentials? So we traced the code and realized that you need to have a gateway setup. We uncommented the default gateway, and could suddenly see the following:

sipvicious sip crack digest udp://192.168.188.128:5080 -e 1000
INFO[2021-04-08 01:03:04] started digest leak tool on udp://192.168.188.128:5080 
INFO[2021-04-08 01:03:04] call picked up by sip:1000@192.168.188.128
INFO[2021-04-08 01:03:05] received BYE, challenging that with a 407
WARN[2021-04-08 01:03:05] digest leaked: response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon 
INFO[2021-04-08 01:03:05] BYE received, terminating call
WARN[2021-04-08 01:03:08] security issue detected: digest leaked
INFO[2021-04-08 01:03:08] test complete
- target: udp://192.168.188.128:5080
  - status: security issue, digest leaked
    results:
      N/A
    issues:
    - digestleak:
        response: 04b472d51cf56b05f8c40d30670b7eac, realm=192.168.188.128, nonce=aC20, uri=sip:57878777@192.168.188.69:60196;transport=udp, method=BYE, username=cluecon

It was time to go to sleep.

The next day, I woke up to write two advisories, one for the digest leak vulnerability and another for the anonymous MESSAGE vulnerability.

But there was still some of that surplus curiosity left from the night before. So we tried our SIP flooding DoS tool on the FreeSWITCH TCP port. This resulted in an out of memory error, crashing FreeSWITCH.

Something like the following:

ubuntu-bionic kernel: [ 4205.446584] Out of memory: Kill process 22590 (freeswitch) score 939 or sacrifice child
ubuntu-bionic kernel: [ 4205.449845] Killed process 22590 (freeswitch) total-vm:10484768kB, anon-rss:7894136kB, file-rss:0kB, shmem-rss:0kB
ubuntu-bionic kernel: [ 4205.720680] oom_reaper: reaped process 22590 (freeswitch), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

This looked familiar, and reminded me that this was indeed something that we had seen before!

Making contact

At this point, we realized that we suddenly had 4 vulnerabilities for FreeSWITCH. We started working and polishing our reports with steps that consistently reproduce each issue. We needed to ensure that our instructions made it easy for the FreeSWITCH developers to replicate our findings.

But who do we contact? The OpenSIPIt event had just happened and we had briefly and virtually met some of the lead developers for FreeSWITCH. So we asked one of the main event’s organiser Maksym Sobolyev, who redirected me to David Duffett, who sent us to Chris Rienzo who indeed happened to be the right person within the FreeSWITCH team.

Then we initiated the conversation. Chris told us that they did not yet have any process for handling security reports. We had just dealt with a similar situation when reporting a vulnerability to another opensource project. The developers for coturn had found the Github security advisory feature really useful. We suggested that FreeSWITCH do the same.

Then, we sent them the first report. It wasn’t for either of the first two vulnerabilities that we discovered. Instead, we shared the report for the digest leak vulnerability because it was the easiest for us to polish and for the FreeSWITCH developers to reproduce. We even included instructions on using SIPp to reproduce the vulnerability, which is one of the useful tools in the VoIP developer’s toolset.

So why did we take so long to submit the next reports? For that, we should describe what each report consists of.

The first part of the report is the description, which should accurately describe what we’re doing and what we are seeing. The aim here is that by reading it, the developers can quickly answer the question: what’s the big deal?

In the case of the SIP flood denial of service issue, we gave as much information as we could by including a graph showing memory consumption during the attack. We wanted to make it clear that the memory was not being freed up once the attack was stopped and therefore the system could not recover automatically.

The next section in the report is the impact section. This should answer the questions:

• how bad could it be?
• how easy is it to pull off?

In the case of the SIP flood vulnerability, the impact is obvious. A crash in a VoIP product is a critical problem so we do not need to convince anyone that this is a problem.

Originally, it appeared that the unauthenticated SUBSCRIBE vulnerability was more serious than it actually was. Therefore, in the case of this issue, the impact section was the part that took the longest to write since we wanted to properly understand the worst case scenario.

Then comes what arguably is the most important section - how to reproduce the security issue. Here we need to provide clear steps and scripts to reproduce the problem. We’ll know when our instructions or scripts are not good enough. In such cases, the developers will tell us that they cannot reproduce the problem or that see a different behaviour than the one we are reporting. Thus they might assume that the impact of the vulnerability is less critical than what we reported in our advisory.

Finally, we include a section on solutions or mitigation steps. There are many solutions to the security issues that we find. Some fixes might not properly address the security issues that we report while other create new problems. Therefore, this section is useful because we want to guide or nudge the developers to what we think are good solutions.

We only confirmed that we could present at ClueCon once we had sent all four advisories to the FreeSWITCH team. We had two main conditions:

• the FreeSWITCH developers needed to be OK with us talking about the vulnerabilities
• they needed to release security fixes before we talk about the details

No pressure!

The process after the first reports

Some of the issues that we reported were easy to reproduce or fix but not necessarily easy to reproduce AND fix. For example, the digest leak vulnerability was easy to reproduce but the fix needed to be plugged in various places. In the case of the SIP flood DoS issue, we needed to have various discussions and a call to make sure that the FreeSWITCH developers see what we were seeing. We were noticing that the memory consumption was never freed after an attack, while the developers were initially seeing their system recover and memory consumption getting back to normal. One reason for this was that we did have better tools - i.e. SIPVicious PRO - rather than the simple scripts that we could provide in our report. In this case, better communication was needed to make sure that both parties were aligned.

The developers then started making their fixes available to us and we started testing them to make sure that the fixes were correct. This tends to be important because sometimes, the fixes do not address the vulnerabilities that we report. In the case of the SIP flood DoS, the first patch did not actually fix the issue. So we did an extra video showing that we could still crash the server. A second fix was provided the next day which did indeed protect against the attack.

That fifth report

But what about the fifth report?

Throughout this process of reporting and testing the fixes, we were developing the WebRTC component of SIPVicious PRO. As we were adding support for DTLS-SRTP in our toolset, we figured out that FreeSWITCH was the easiest system to test with for our needs. We ended up using it during our development to have something robust and reliable to test against.

Then, while working on the SRTP part of SIPVicious PRO, we were seeing the following error messages in FreeSWITCH:

2021-08-11 07:59:06.923256 [INFO] sofia.c:10362 sofia/internal/99988801120715425438@207.154.254.38 receiving invite from 185.44.76.73:56030 version: 1.10.6-release git 1ff9d0a 2021-03-25 13:16:09Z 64bit call-id: 1184444165-1963300639-337710667
2021-08-11 07:59:12.443278 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 10 errors
2021-08-11 07:59:12.643181 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 20 errors
2021-08-11 07:59:12.843235 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 30 errors
2021-08-11 07:59:13.043233 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 40 errors
2021-08-11 07:59:13.243215 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 50 errors
2021-08-11 07:59:13.443220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 60 errors
2021-08-11 07:59:13.643232 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 70 errors
2021-08-11 07:59:13.843220 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 80 errors
2021-08-11 07:59:14.043188 [WARNING] switch_rtp.c:6332 SRTP audio unprotect failed with code 21 () 13 bytes 90 errors
2021-08-11 07:59:14.243300 [WARNING] switch_rtp.c:6325 SRTP audio unprotect failed with code 21 () 13 bytes 100 errors
2021-08-11 07:59:14.243300 [WARNING] switch_rtp.c:6328 Ending call due to SRTP error

It turned out that this was due to a bug in our SRTP code which wasn’t sending correctly encrypted SRTP. Typical rookie mistake! We got busy fixing our mistakes. But then, our thoughts turned to something else: what if we forced this error on other calls?

We tried that out and that’s when, simulating a victim user, we saw the following SIP message:

BYE sip:1001@192.168.0.112:59438;transport=wss SIP/2.0
Via: SIP/2.0/WSS 192.168.0.106:7443;rport;branch=z9hG4bKgejmX1Ka5F2ZB
Max-Forwards: 70
From: <sip:1000@192.168.0.106>;tag=t9trvZ2D9cDtg
To: <sip:1001@192.168.0.106>;tag=jNWhecoTN9IR4Zv4
Call-ID: mkBDUpsWEr9CdCqD
CSeq: 39402862 BYE
User-Agent: FreeSWITCH-mod_sofia/1.10.7-dev+git~20210727T192513Z~432bfc0c45~64bit
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, path, replaces
Reason: SIP;cause=613;text=\"SRTP_READ_ERROR\"
Content-Length: 0

This meant that an attacker could simply spray the FreeSWITCH server with invalid SRTP packets and generate an error that would terminate the ongoing encrypted calls. Luckily this issue was rather easy to address, since to fix this one should simply ignore the SRTP errors.

So, it was time for an additional and final advisory which we submitted early in September.

Preparing for the release

Towards mid-October the FreeSWITCH developers had made their security fixes available to us and we had tested them and verified them. So, right before ClueCon was about to start, they released FreeSWITCH 1.10.7 with the security fixes. At this point, we published our advisories and they also published advisories on their Github advisories page. And then I gave my presentation with the same title as this article.

Lessons learned

Security researchers

So what have we learned through this whole story?

For us, the hard lesson is that we might not always immediately understand the vulnerabilities that we see. We might not even realize that we came across a vulnerability - such as the one with SRTP - because we’re too busy fixing bugs in our own software. And yes, writing our own buggy software helps us find security bugs in other people’s software!

Of course, without tools finding such issues would have been much harder. So having flexible but robust security tools is essential for doing RTC security research and testing.

But equally important for us is that we contribute towards more secure RTC solutions by sharing our findings with the developers. Therefore, writing clear reports is critical if we, as security researchers, really want to actually help developers and want to be taken seriously.

Open-source developers

All software is going to have security vulnerabilities, especially something as complex as a VoIP or WebRTC system. It seems having a process of handling security reports is one of the better solutions to this problem. So making it easier for security researchers to submit such reports seems vital. Luckily, with something like Github, it is rather easy to do and allows handling of reports separately from other methods such as the public issue tracker. A feature that developers find very useful in this case is that one can create a private fork with security fixes so that the security researcher can test without leaking any information about the bugs that are being reported.

FreeSWITCH users

Naturally, for FreeSWITCH users out there, I hope that this article helped you better understand if the issues that we reported affect you. We encourage you to upgrade so that your installation is not vulnerable to what we just described.

References

• Full original video for Kamailio World 2019 Dangerous Demos
• Slides for the presentation
• Our advisories: https://github.com/EnableSecurity/advisories
• FreeSWITCH official advisories page
• About Github security advisories