The great Kamailio security debate and some misconceptions debunked

Introduction

The Kamailio community has always been very welcoming to us since our first connection in 2015 where I gave a dangerous demo showing the open-source version of SIPVicious scanning the Internet and discovering all sorts of SIP devices. Since then, we’ve been contributing through presentations at Kamailio World each year, highlighting various security concerns for the RTC community and the occasional security report and advisory urging people to upgrade their Kamailio. One thing that I personally appreciate is the positive reception of security reports and the security fixes that are made very quickly available in the public git repository.

A bit of background before diving in

Recently we published an advisory with details of how the remove_hf function, often used to remove security-sensitive headers at the edge, could be bypassed before a particular fix. Our intention was to urge those affected by this security issue to get people to fix their Kamailio installation. This could be done by upgrading to the fixed version, porting the patch to their own code-base or mitigating using remove_hf_re. The advisory was first sent to the usual security mailing lists and I notified the users on the Kamailio matrix channel. We also published a post on this very blog with details explaining how this might also affect other SIP software (especially SBCs). At this point, @miconda (co-founder of Kamailio) asked for some changes and clarifications. The conversation can be seen here. Specifically, the following happened:

• it was not clear to me that the vulnerability affected only non-standard headers as I had only tested with non-standard headers and not standard headers such as the P-Asserted-Identity header
• I made an update in the blog post mentioning the usage of remove_hf_re, to reflect the recommendations in the advisory itself, and those made by @miconda
• removed the example of remove_hf("P-Asserted-Identity"); which was given as an example of how remove_hf might be used; although it was not an example of a vulnerable case, before this change, the blog post could have easily been misunderstood to mean that remove_hf("P-Asserted-Identity"); was affected
• I added a note in the advisory stating that this issue only affected header names that are not defined in src/core/parser/hf.h
• added a note in the blog similar to the one in the advisory stating which header names were affected; clarifying that one couldn’t use this vulnerability to bypass remove_hf("P-Asserted-Identity")

After those changes, I posted the update advisory to the sr-users mailing list, which is the official Kamailio users list. Following my post, @miconda responded to make sure that it is clear that this only affected non-standard headers and giving his explanation of the behaviour of the code before the patch was applied. He also mentioned that the bug had been around since 18 years! This post generated a large number of replies from various community members and I intend to go through some of the ideas (misconceptions in my opinion) and claims that emerged during this conversation. The full thread can be followed here.

One of the initial allegations was that the project leader was downplaying the issue. This was due to the wording in his reply and discussion on that focused around the use of the word only. This is a linguistics problem and it is certainly not my expertise to discuss. But with other topics, I do have something to say …

Claim: this issue does not affect many organisations

Without extensive data, this is hard to agree or disagree with. On the thread, @miconda said that his deployments were not affected due to various reasons (clarified more recently). On the other hand, Maxim Sobolev mentioned that 90% of specific engagements needed to use such configuration patterns (i.e. ones that include private headers). In our case, we can say that out of 4 of our clients that we informed of this vulnerability, 2 confirmed that they needed to fix, one seemed to be affected in specific cases, and the remaining client was not affected at all. Of course this is an extremely small sample-set to base any claim on. One thing that is quite obvious to us is that the use of sensitive, non-standard headers in SIP core networks, is higher than one would wish it to be.

Claim: custom headers are only known to internal users

Actually the full title for that should be Custom headers are only known to internal users who can do worse damage.

This one ties in with the previous claim in various ways. What we can say is that although this is a very common assumption, it is simply based on wishful thinking. See, custom headers should not be treated as a secret. Passwords, private keys, API keys and so on are secrets but custom headers, generally, are not. To prove this point, consider the following:

• search the archives of the sr-users mailing list, and perhaps other mailing lists for VoIP developers and implementers and you will find various SIP traces, configuration and log files being shared containing such SIP headers
• we have seen pcaps and log files in issues filed on publicly available issue tracking software, with custom headers
• in some cases, a disk image is provided by a vendor, such as an ISO or Virtual machine image; in such systems, one component might be Kamailio (or similar) together with its configuration files; adversaries may easily download such disk images and extract configuration files (no, the solution isn’t to encrypt your configuration files!)
• custom headers might be seen in web interfaces or online debug tools (or pcaps), especially in the case of CPaaS
• finally, there are cases where attackers may, intelligently or through brute-force, guess header names and possibly valid values

Having said that, of course, in many cases attackers do not have access to such information but we should not (and probably, cannot) threat special header names as secrets and therefore we need to assume that they might be disclosed at some point during their lifetime.

Claim: if it’s an 18 year old bug, it can’t have been high risk

There was the claim made, implied or not, that since it is a really old bug, it being discovered so late in its life, must clearly be an indication that it cannot be a high severity vulnerability!

Both opensource and proprietary software have had vulnerabilities that were that old, or even older in some cases. Take the case of the Shellshock vulnerability that affected bash. This was around for 25 years before it being made public, yet not many doubt its severity (being critical in many cases), since the vulnerability leads to remote code execution. There are various other similar cases of vulnerabilities that had been around for a long time before being squashed. Age of the bug does not correlate at all with the severity of the vulnerability.

That said, we’re not saying that the remove_hf bug was a critical vulnerability. Regardless of the age of the bug, the severity, especially in this case, is relative to the use of the vulnerable function. If your Kamailio setup removes SIP headers (using remove_hf naturally!) and if these headers have a critical security function, then it is likely that the severity rating of this vulnerability for your particular setup is at least high.

Of course, if your edge does not remove such SIP headers which are actually used in your SIP network, then you certainly need to take care of that at the edge. Or even better, avoid their use altogether when possible.

Claim: this should have been found if people were doing proper testing

The nature of security vulnerabilities is that often, they are only discovered when someone actually cares to look, takes a long stare at the code and perhaps be in the frame of mind to interpret something as a security vulnerability. Their discovery does not mean that no one is doing proper testing, and like many other things, security bugs often appear obvious after the facts are laid bare. Proper testing does not find all bugs, and the commonly quoted claim that given enough eyeballs, all bugs are shallow has been (various times) been proven false.

Having said that, it is also our opinion that not enough security testing is done in real-time communications infrastructure, which is one of the reasons why we find RTC security so interesting. This is the subject of a future blog post, but I’ll end this one noting that, in the VoIP/RTC world, even if these days security solutions are often implemented and in place, we have noticed that their effectiveness varies greatly. We suspect that this is due to the limited exposure that they get from fellow security researchers and penetration testers in the offensive security world.

Claim: infrequent advisories = project is not serious about security

Some open source projects publish advisories much more frequently than others. I do not think that the number of advisories per year that a project publishes is actually indicative of how much that project’s members value security. See, different code bases have different bug numbers. Another factor to keep in mind is that the security exposure for something like Kamailio is greatly dependent on configuration. This is unlike some projects like Asterisk where the standard configuration exposes a lot more complex code to potential attackers. On top of that, the default configuration for Kamailio is quite minimal yet functional. What this means is that security researchers who take a peek at the default configuration, are unlikely to find low hanging fruit (easily found vulnerabilities) and would need to dig deeper.

What we have seen is that in cases of security reports, the Kamailio project has taken them seriously and fixed them quickly; but advisories were only published in some cases where the vulnerability was critical and affected almost all installations.

Claim: limited number of advisories = project is more secure

Then there was the opposite claim that since very few advisories show up featuring Kamailio, that must surely mean that the project is more secure. In any comparison, we would need to have a second party or more for contextual reasons. One thing that we noticed, and probably was alluded to by @miconda, is that a large number of bugs are routinely fixed in Kamailio. It is a very active project and this is a natural part of its cycle. We think that at least a few of these bugs might have security implications. For example, consider these two separate fixes that were applied earlier this year:

• core: safer truncation of hostname using memcpy
• siptrace: fix memory leak in fake replies tracing

The first one affected the core while the second one affected a commonly used Kamailio module. Might they have security implications? Most certainly (although we’re not sure and have not verified). But, at least publicly, no one seemed to evaluate the security implications of these bugs. Without that, no advisory can be created explaining that one needs to upgrade due to a security fix. When fixing bugs, it is easier to just move on and not assess the full impact of that bug (which may affect security), rather than issuing an advisory with a release or doing similar extra actions.

I guess my point is that although the lack of advisories is not a clear indication that the project is more or less secure, it gives the end user little knowledge as to when bug fixes have to be necessarily (e.g. due to a company security policy) be applied or not.

Of course, at this point, one has to acknowledge that assessing the security of each single bug fix is not an easy task, is probably not what the project developers want to be liable to and simply costs time and effort. And, this being the open-source world, such time and effort is often not well-compensated.

Claim: if you’re serious about security, monitor the mailing lists

There is a lot of truth in this claim, but in my opinion, this is rather unfortunate. At the moment, if one wants to know if a security upgrade is necessary, one needs to watch the mailing lists, check out the issues posted on the bug tracker and watch for code changes indicating potential vulnerabilities. I think it is very easy to miss the needle in the haystack with a project like Kamailio, that has so many commits on a daily basis.

It would be great if one could reliably say that no security fixes were applied since the last security advisory, without constantly monitoring the project’s updates. But, that wouldn’t be the reality with many open source projects (especially the Linux kernel). I’m not going to try to cover closed source projects, which have different economics and, in my opinion (and experience), are typically worse off on this particular aspect of transparency and security.

With that one sorted, on to the topic of security experts!

Claim: security experts should decide what is a security vulnerability

In general, I would say that yes, security experts should give their evaluation based on their findings and explanations of how to reproduce them. That said, security experts often don’t know the whole picture. Case in point, I did not realize (until @miconda pointed it out) that the header smuggling issue only affected non-standard headers. Therefore it is important that the reports of security experts are questioned and evaluated too. So, although security experts need to be listened to, their word should not be taken as final and discussions are a healthy way to understand the implications of vulnerabilities that get reported.

Discussion: when should the project publish an advisory?

The simple answer is that security advisories should be issued by the project whenever there is a bug fix that has security implications. This does, however, require the project to assess if a bug fix has any security implications, which can require extra effort. End users can better decide if they need to apply a security fix when the project issues an advisory for a low severity issue, than when no advisory is issued at all. Security experts may disagree with the severity rating (and this happens often enough), but this is a better situation than having to dig through all the bug fixes that a project might apply from one release to another.

Discussion: educational security role

@Oej mentioned that the Kamailio project can have an educational role by explaining potential bugs that are the result of insecure configuration. He made the parallel example of Asterisk’s dialplan injection vulnerabilities, and similar common issues and how they created a special README file explaining such potential issues. Kamailio already has a tutorial called Overview of Security related config snippets which gives general recommendations on securing the Kamailio box itself and adding specific configuration in Kamailio to protect against well known attacks.

Although this is very valuable, it does not address what @oej had in mind, which is examples of configuration patterns in Kamailio that introduce vulnerabilities and then recommendations on avoiding such patterns. From our part, we hope to contribute in this space as it is something that we do as part of our work within our penetration testing framework.

Moving forward

If you made it so far, congratulations! There is much more that can be discussed on the topics covered in the mailing list thread. The posts on this thread showed that people do indeed care deeply about Kamailio as well as the security of their RTC systems. My wish list for the Kamailio project is the following:

• Keep treating security vulnerabilities seriously and making good fixes available as quickly
• Assess if a bug fix has security implications, and if that is the case, issue a security advisory
• Documentation with examples of insecure configuration patterns and parallel examples of how to do the same thing securely (when possible)

Of course, this can’t happen without volunteers and with open source, if one wants something done, there is often the opportunity to do it yourself. From our part, we hope to contribute especially with the insecure configuration documentation, thanks to previous penetration tests done for our clients. Anything else tends to be done during our spare time which is, unfortunately, limited and therefore hard to commit to. One idea that would be considered is sponsorship from companies that greatly benefit from Kamailio which would create incentive to work on improving the security of the project.

The Kamailio software is certainly not in a sorry state when it comes to security. But, security is an ongoing process that could always benefit from improvement and rarely ever sits still!