Techniques to play with custom and encrypted protocols

An interesting presentation from DEFCON20 provided by Elie Bursztein and Patrik Samy called “Fuzzing Online Games” touches areas of application security where traffic analysis is not enough to perform a penetration test.

As stated by the authors:
“In a nutshell the lack of direct access to the game server and having to deal with clients that are far too complex to be easily emulated force us to rely on injecting fuzzing data into a legitimate connections rather than use the standard replay execution approach. Top that with heavily encrypted and complex network protocols and you start to see why we had to become creative to succeed :)”

The problem of an application security analysts is that most of the communication is encrypted and is using custom protocols that can’t be intercepted using standard proxies like Burp or Charles. One’s thoughts on this could go into using Mallory as transport layer proxy and should be fine with some custom protocols but still doesn’t deal with the encryption problem. So reverse engineering and memory analysis and manipulation must be involved.

Techniques proposed by the presenters:
– Combining network traffic analysis with memory analysis (check what happens in the memory when certain packets are sent over the wire)

Challenges involved:
– Intercepting traffic
– Bypassing Encryption
– Reversing the protocol
– Monitoring the results of fuzzing

Traffic interception:
– DLL injection at the application level – direct access to game state
– Write a driver at the OS level
– Pass the traffic through an intercepting box – this is done at the network level; as a side note this can be done on the same box using WireShark; keep in mind that WireShark does not intercept packets sent on the loopback interface and you can use RawCap for this

DLL injection:
– Most application use Windows Winsock API and the interesting functions to watch for are connect, recv and send
– Ways to do it: Microsoft detour library and IAT (Import Address Table) hooking:
– The problem is that protection mechanisms like anti-cheating engines detect hooking

Writing a driver:
Windows Filter Platform – could be an excellent replacement for Mallory since we don’t need an external machine (even a virtual one) to capture the packets

The presenters then focus on the analysis of the custom LOL (League of Legends) protocol to give a practical example. Among others, they use packet, statistical, n-gram analysis and search for a feasible way to fuzz the protocol. They don’t go into details like tools, usage and how to perform the analysis but stay on a high perspective level.