[Techtalk] Update on that reengineering problem

[Meredydd]

OK, I think I know where I stand now. So here it is.

I am attempting to reverse engineer Microsoft's MSN Chat protocol. This is 
proving relatively easy in terms of the protocol itself (it's a bastardised 
version of IRC, using for authentication the standard Passport cookie 
scheme), but I'm having problems with the client-verification procedures used 
to check that the client is the genuine Microsoft OCX, not some *gasp* 
non-Windows user. This is a challenge-response sequence required before you 
can actually do any talking. Sequence of events goes something like this:

- Client connects to server, sends a version string
- Client sends a constant string to initiate client verification
- Server sends an eight-byte challenge wrapped in a constant string
- Client sends a challenge response
- Server sends "OK" notification, traffic continues

There are in fact two types of server connection. You first connect to a 
dispatch server, for which some kind soul has already reversed the response 
algorithm, which gives you the IP on which any particular chat room (channel) 
is hosted. You then make *another* connection, to the server you've been told 
about, and go through the verification routine again. Unfortunately, the 
algorithm, which was reverse engineered by the same person/group who did the 
first one, has been changed. And I haven't a clue what to.

So. The algorithm I need to replicate takes an eight-byte challenge and 
returns a sixteen-byte response. From past experience, it is likely to make 
heavy use of MD5 hashing. It may or may not be related to the known algorithm 
(as was, in fact, it was identical):
- Append challenge to constant string
- Take hash of the above, append that to another constant string
- Send hash of the above as response

I've tried a grep - not an occurrence of "md5" (in either case) in the OCX. 
I've also tried disassembly (an OCX is just an EXE with no main() as far as I 
understand), but can't seem to make head or tail of it. From looking at it in 
a hex editor, several string constants have had strange things done to them 
(spaced out every other char with null zeros etc), and the combination of the 
above leads me to suspect that it's been armoured against disassembly. 
Unfortunately, I don't know enough about this sort of thing to verify it, and 
I can't find any more useful information on the web :-(  Oh, and yes, there 
is no software apart from that OCX which talks to MSN Chat. No Mac version 
(there goes the nice UNIX debugging environment). And internet chat isn't 
covered by the DoJ settlement, because it's not a protocol used between 
standard MS clients and standard Win2k+ servers.

So - any bright ideas? What should I do next?

Meredydd