Comments on: How to make MD5, Java and Unicode play nicely

Question: How to make MD5, Java and Unicode play nicely

idb — Mon, 10 Dec 2007 19:44:55 -0800

How do I make MD5, Java and Unicode play nicely together?

I don't really get unicode and character sets. Now I need to. I'm trying to reproduce the same effect in java that this javascript gets when the chrsz variable is set to 16. When the variable is set to 8, the java version gives the same result as the javascript. I need the java version to give the same result as the javascript version does when chrsz equals 16.

I'm not great at java. I know enough to be dangerous. I thought I was pretty hot stuff at javascript, but it might help if I knew what << or >>> meant in javascript. As you can imagine, that's really hard to google.

By: sanko

sanko — Mon, 10 Dec 2007 20:01:29 -0800

I can't help with the MD5 problem, but if you're looking for the meaning of << or >>> the search you want is "shift operator."

By: kdar

kdar — Mon, 10 Dec 2007 20:02:20 -0800

Those <<, >>, and >>> are bitwise shift operators. They appear to be the same in java. (Sorry for not giving you a more in-depth explanation, but maybe this will illuminate things a bit for you.)

By: burnmp3s

burnmp3s — Mon, 10 Dec 2007 20:22:34 -0800

Why are you trying to re-implement MD5 in Java? It's already built into the standard libraries.

By: sbutler

sbutler — Tue, 11 Dec 2007 00:12:47 -0800

I agree with burnmp3s... except I wouldn't use the String.getBytes() method. You don't want to trust that the platform encoding of a string is the same as the one your JS methods are using.

Now, I don't really understand the MD5 algo, but I do know Unicode. According to your JS source, it is encoding the unicode strings as UTF-16LE (16 bit code points, little endian byte order). If that's what the code actually does (and I'm too tired to verify it) then this Java bit should work:

MessageDigest m = MessageDigest.getInstance( "MD5" );
m.update( aString.getBytes( "UTF-16LE" ), 0, aString.length() );
System.out.println( "MD5: " + new BigInteger( 1, m.digest() ).toString( 16 ) );

If that DOESN'T work, then your JS version is broken. Find a better one.

By: idb

idb — Tue, 11 Dec 2007 04:06:08 -0800

Thank you all for the responses. The first thing I tried was to implement the standard libraries. They do fine with the 8-bit setting and give exactly the same result. I just couldn't get them to give the same result as when the variable was set to 16-bit.

Going to try again based on sbutler's code.

By: sbutler

sbutler — Tue, 11 Dec 2007 07:43:19 -0800

Ohhh... wait a minute. That code above is broken (I copied it from the website burnmp3 listed). aString.getBytes( "UTF-16LE" ).length() != aString.length(). Try this instead:

MessageDigest m = MessageDigest.getInstance( "MD5" );
byte aStringBytes[] = aString.getBytes( "UTF-16LE" );
m.update( aStringBytes, 0, aStringBytes.length() );
System.out.println( "MD5: " + new BigInteger( 1, m.digest() ).toString( 16 ) );

By: idb

idb — Tue, 11 Dec 2007 10:41:07 -0800

sbutler, you are awesome!

For reference sake, what worked was aStringBytes.length without parentheses. Also, there appears to be a leading zero that I have to worry about. That aside, it works.

Final proof of concept code - MD5.java:



    import java.security.*;

    import java.math.*;



    public class MD5 {

       public static void main(String args[]) throws Exception{

            String aString = "teststring";

            MessageDigest m = MessageDigest.getInstance("MD5");

            byte aStringBytes[] = aString.getBytes( "UTF-16LE" );

            m.update( aStringBytes, 0, aStringBytes.length );

            System.out.println( "MD5: " + new BigInteger( 1, m.digest() ).toString( 16 ) );

        }

    }

Thank you.

By: idb

idb — Tue, 11 Dec 2007 10:45:40 -0800

Thank you also to kdar and sanko for the explanations of bitwise operators.