This is a tale of working closely with Microsoft on a security issue, the ethics, challenges and the collaborative relationship between engineers that we must have to resolve any serious security issue.
In October 2013 a client of Catalyst asked us to implement ‘bad password lockout’ in Samba’s Active Directory domain controller, following on from the similar support in Samba’s previous domain controller product. What followed however was a tale of how simple choices in how validation checks are performed can have profound security implications.
The full name of the network operation in question is SamrUnicodeChangePasswordUser2, and it is one of the many security-critical network APIs that Samba must implement, and must do so while matching the Microsoft Windows semantics.
The key phrase here is ‘matching Windows’, because both here at Catalyst and on the Samba Team, when building Samba we generally take the behaviour of Microsoft’s implementation to be the gold standard. Feature or mis-feature, we assume that the rest of the industry has tested against that, and that good mimicry of the external behaviour is the best way to blend in. (Internally the structures are quite different).
Taking on the client task first required setting a benchmark for what needed to be done, and so old tests were revived and re-targeted, new tests were written, and Samba’s older code was examined for clues. One area became quickly apparent: not only was the new AD DC code missing an implementation of bad password lockout, the older Samba DC code, implementing the NT4 DC, only handled half the job.
The problem was that while a normal logon would increment the bad password count, and lock out the user, no such check was performed during password changes. This quickly turned a normal feature implementation task into a security issue.
Given the need to correctly implement bad password lockouts for the SamrUnicodeChangePasswordUser2 call, we started writing tests, to confirm the behaviour of the ‘gold standard’, that of Microsoft’s Windows. It was in doing those tests, and watching the exact pattern of error codes that we got back that we got a nasty shock.
When implementing a logon system, the key is to maintain information asymmetry. That is, to ensure that the attacker always knows less than the target system. In a fanciful example, the logon system must not helpfully tell the attacker ‘almost right, try adding an extra digit’, or ‘that password is OK, but I’m locked out for the moment, try later’.
It was this second situation that happened against Microsoft Windows. When changing a password with SamrUnicodeChangePasswordUser2, both the incoming password and the lockout status were correctly confirmed, but in the wrong order, allowing the attacker unrestricted access to try new passwords, looking for failure codes of ‘wrong password’ or ‘locked out’, meaning ‘right password, but locked out’. The attacker would only need to wait another few minutes to log in with the cracked password.
Naturally we reported this to our friends at Microsoft. But this meant that suddenly we had not only two issues, but three, and the complexity of multi-organisation co-ordination. While this is nothing new in the security industry, it certainly amused the directors here at Catalyst to be helping Microsoft fix their own Security issues.
A number of months passed while Microsoft implemented the fix, and did ‘variant analysis’ looking for (and finding) where they had a second copy of the same code, but in March 2014, Microsoft released MS-14-016, credited Andrew Bartlett of Catalyst and the Samba Team. At the same time, Samba released Samba 4.1.6 with fixes for CVE-2013-4496, and our developers were finally able to release that addition of support on our AD DC (it released soon after with Samba 4.2).
While painful, the slow and sometimes painful process of working with other vendors is the correct thing to do. In particular we have a moral responsibility given our deep knowledge of the AD System to Microsoft’s customers. Few others outside Microsoft have reason to delve so deep, or probe so closely.
We are confident that will not be the last time that Catalyst and Samba is credited with finding security holes in Microsoft’s Windows.
