try to sniff the packets sent by your server to the master server and check their content for being sure of their format and moreover to know what is the hash derived by your cdkey.
I have a tool which catches these particular packets here:
http://aluigi.org/papers.htm#gshsniffanyway the "encoding" is a simple xoring with the word "gamespy"
the md5 function should be at offset 0061acb0 of halo.exe 1.0.8.616