I was recently looking at converting an old application from VB6 to Java that used MD5 in its output files as hashes for validation.

The first thing I did was to make a java class that read in the file and checked the hashes, I tried it on a few files and it worked fine, then I found a file that it failed on.

Now, this app wrote all the files using the exact same function, so it seemed odd that 1 of them wouldn’t parse and the rest would.

When I looked at the file closer, I found that this one contained some symbols in the output that the others didn’t - I eventually figured out that the symbol that was causing the problem was the pound sign (£).

Without going into too much detail, this presented a major problem, the string in question was used as part of the password validation for the app (the output files are encrypted using the password as a key), and the java code was getting different results than the old VB6 code, and was unable to decode the file as a result.

So, this sparked my curiosity a bit, the VB6 code I was using wasn’t a built in, it was code I’d gotten elsewhere and used, so I assumed it was faulty code (not that this helped me much, as I needed to get the exact same output, but ignoring that).

I edited the initial form of my application to return the MD5 String for £ on its own, and got: d527ca074d412d9d0ffc844872c4603c

I did the same for my java code and got: 6465dad1d31752be3f3283e8f70feef7

So now all I needed to do was to see which was right, so I made a quick PHP script, and did the same and got: d99731d14c7750048538404febb0e357 … Yet another different hash!?

Ok, I thought, md5sum will help me figure out which one is right. one echo '£' | md5sum - and I had 67160ce935d7cb5339047b12ad4611cb. Yes, that is correct, a 4th different hash.

So here I was with 4 different hashes and no idea which one was correct.

So after a bit of googling, I discovered that the MD5 RFC (1321) had the source code for a test application in it.

So I extracted the code from the Appendix of http://www.ietf.org/rfc/rfc1321.txt and tried to compile it with gcc md5.c mddriver.c -o mddriver only to discover that it failed to compile with lots of errors, fortunately this was an easy fix, near the top of mddriver.c, change #define MD MD5 to #define MD 5 and then it compiles without problem.

So, I ran ./mddriver -s£ and got the output MD5 ("£") = d99731d14c7750048538404febb0e357 which agreed with what the PHP md5() function gave.

(Its worth noting that echo '£' | ./mddriver agreed with md5sum, which made me remember that echo appends a \n, which was why I got a different output, running echo -n '£' | md5sum gives the correct result, and would have saved me googling and finding the test suite!)

I tested a few other things and got the following results:

        mddriver: d99731d14c7750048538404febb0e357
             PHP: d99731d14c7750048538404febb0e357
           mySQL: d99731d14c7750048538404febb0e357
          python: d99731d14c7750048538404febb0e357
      postgreSQL: d99731d14c7750048538404febb0e357
          md5sum: d99731d14c7750048538404febb0e357

      JavaScript: d527ca074d412d9d0ffc844872c4603c
     VisualBasic: d527ca074d412d9d0ffc844872c4603c
         Eggdrop: d527ca074d412d9d0ffc844872c4603c
   Java (custom): d527ca074d412d9d0ffc844872c4603c

 Java (built in): 6465dad1d31752be3f3283e8f70feef7

• JavaScript implementation from http://pajhome.org.uk/crypt/md5/
• VisualBasic implementation from http://www.frez.co.uk/freecode.htm#md5;
• Custom Java implementation from http://www.freevbcode.com/ShowCode.Asp?ID=741

There is also a list of MD5 implementations at http://userpages.umbc.edu/~mabzug1/cs/md5/md5.html

---

The differences are primarily due to character encoding in the different languages. (In the case of my app, there was also a flaw in the implementation for strings where (length % 64) is > than 55 as well)

Example:

[07:14:55] [shane@Xion:~]$ php -r 'echo md5(utf8_encode("£"))."\n";'
2ccf59396b3c0958eec4ba721e2d083f
[07:15:01] [shane@Xion:~]$ php -r 'echo md5("£")."\n";'
d99731d14c7750048538404febb0e357

Java: System.out.println((int)'£'); => "163"
PHP: echo ord('£'); => "194"
PHP: echo ord(utf8_encode('£')); => "195"

Edit: This is now pretty much unneeded, the new version of VirtualBox seems to handle this all nicely on its own.

As I mentioned in my last post, One of the useful advantages of the network boot setup is that I can use it to quickly install virtual machines.

Now a few things:

• My Desktop is a lot more powerful than my server, so I run the virtual machines on it.
• I use virtualbox rather than vmware.
• All the network boot stuff is on my server not my desktop (obviously)

So in order to allow this, virtualbox needed to be setup to bridge to my existing adapter, this was quite straight forward, pretty much exactly as the manual said.

sudo apt-get install bridge-utils

Edit /etc/network/interfaces, and add

auto br0
iface br0 inet dhcp
    bridge_ports eth0

Now the next suggestion was to setup a tap0 device and tell virtualbox to use that, or to use a dynamic configuration.

The dynamic configuration sounded better as it meant I didn’t need to remember to add a new tap device for each vm.

The suggested dynamic configuration suggests using kdesu/gksudo and a script in the home dir of the user that will setup and cleaup the tap device (this means inputting your password every tiem you start/stop the VM and requiring a separate script for each user that wants to have a vm with bridging) this seemed rather annoying so I came up with an alternative.

/usr/bin/setuptap

#!/bin/bash

# Make sure we are root
if [ $(whoami) != root ]; then
        exit 1;
fi;

# Create an new TAP interface for the user and remember its name.
interface=`VBoxTunctl -b -u ${SUDO_USER}`
# If for some reason the interface could not be created, return 1 to
# tell this to VirtualBox.
if [ -z "$interface" ]; then
        exit 1
fi
# Write the name of the interface to the standard output.
echo ${interface}

# Bring up the interface.
/sbin/ifconfig ${interface} up
# And add it to the bridge.
/usr/sbin/brctl addif br0 ${interface}

/usr/bin/cleanuptap

#!/bin/bash

# Make sure we are root
if [ $(whoami) != root ]; then
        exit 1;
fi;

# Remove the interface from the bridge.  The second script parameter is
# the interface name.
/usr/sbin/brctl delif br0 $2
# And use VBoxTunctl to remove the interface.
VBoxTunctl -d $2

Now these scripts run with sudo as any user will setup the tap device for that user (thats what ${SUDO_USER} is for)

This still requires a password for starting/stopping the VMs tho, so we use

sudo visudo

or if you prefer nano

sudo EDITOR=nano visudo

and add

# Allow virtualbox users to setup/cleanup tap devices
%vboxusers        ALL=NOPASSWD:/usr/bin/setuptap,/usr/bin/cleanuptap

now:

• configure virtualbox to attach the network device to a “host interface”
• leave the Interface name blank (setuptap creates the next available one)
• Setup Application: sudo /usr/bin/setuptap
• Terminate Application: sudo /usr/bin/cleanuptap

And virtualbox will be able to create/destroy the tap device as needed.

However. there is still one problem, DHCP will not work for these VMs without a little help, so we need to:

sudo apt-get install dhcp3-relay

and answer the questions asked. (DHCP Server IP, and Interface to listen on (br0))

Virtualbox unfortunatly seems to need a little push to actually network boot, so I also use an etherboot iso to actually boot from the network along with the “PCnet-FAST III” adapter type.

and thats all there is to it, you can now network boot and dhcp from virtual machines not hosted on the server.

On a daily basis, I get around 800 emails to my email accounts, of which most of it is spam.

Now as good as thunderbird is at detecting spam, even it fails at a lot of the spam I receive, leaving me with around 100-200 spam per day that gets into my inbox.

I’ve dealt with and accepted this for over a year now, before a discussion on IRC made me decide to do something about it. (When I say discussion, I mean Chris pasted one line showing how good the UTD-Hosting mail server was at preventing junk getting to him)

So, I recently (Today and yesterday) started prodding my postfix config to help with the problem.

Firstly I added some RBL checking, this was easy enough, 3 lines to my config in the smtpd_recipient_restrictions bit:

    reject_rbl_client list.dsbl.org
    reject_rbl_client zen.spamhaus.org
    reject_rbl_client dnsbl.sorbs.net

I also added:

    reject_non_fqdn_recipient
    reject_unknown_recipient_domain

I also added the following lines:

smtpd_helo_required = yes

smtpd_delay_reject = yes

smtpd_helo_restrictions =
    permit_mynetworks
    check_helo_access hash:/etc/postfix/helo_access
    reject_non_fqdn_hostname
    reject_invalid_hostname
    permit

smtpd_sender_restrictions =
    permit_mynetworks
    reject_non_fqdn_sender
    reject_unknown_sender_domain
    permit

/etc/postfix/helo_access looks like this (Its surprising how many mails this catches, ¹¹⁴⁄₇₅₀₀ - altho they would probably be caught later on):

soren.co.uk            REJECT You are not me.
207.150.170.50         REJECT You are not me.

Next step was SPF checking, this involved adding to smtpd_recipient_restrictions:

check_policy_service unix:private/policy

and to master.cf

policy  unix  -       n       n       -       -       spawn
        user=nobody argv=/usr/bin/perl /usr/lib/postfix/policyd-spf-perl

(One can apt-get install postfix-policyd-spf-perl or download it from http://www.openspf.org/Software)

Currently I use catch-all on all my domains (yes this is stupid I know) and as a result, I get a lot of spam to 1) Addresses that don’t exist and never have 2) Addresses that used to exist for others but now don’t.

To combat this, I added this line to smtpd_recipient_restrictions:

check_recipient_access hash:/etc/postfix/recipient_access

/etc/postfix/recipient_access looks something like this:

foo@example.com REJECT This account is no longer valid.
bar@example.com REJECT This account is no longer valid.
baz@example.net REJECT This account is no longer valid.

The result of all this can be seen by running the mailstats script Chris was kind enough to share with me:

root@soren:/etc/postfix# ./mailstats.php

Incoming --(7500)--> Valid HELO --(6707)--> Valid Sender --(6705)--> Passed by dsbl --(6136)--> Passed by spamhaus --(811)--> Passed by sorbs --(568)--> Passed by relay check --(565)--> Passed by SPF --(542)--> Forwarded to shinobu --(390)--> To a valid domain --(339)--> To a valid user --(306)--> Dropped Spam --(306)--> Delivered.
Total Rejections: 7194 (Unknown Reason: 0 | Pretended to be me: 114)

The “Forwarded to shinobu” entry is a server for which I am the backup MX for, this accounts for 152 mails (about 2%)

The delivered count of 306 mails is about 4%, meaning that 94% of all the junk mail I receive is now dropped by postfix and not delivered to my mailbox!

These simple additions have made a huge difference! I have a 10 day holiday coming up, and now rather than coming home to 8000 mails, I’ll only come home to 320!

As a further line of defence, prior to being sent to my mailbox, those 4% of mails get filtered through spamassassin (which I have configured to only run for certain domains, with different scores for different domains/users as needed) which does a good job of catching the spam that thunderbird misses, configuring a mail filter on thunderbird to filter these mails (Which get subject tagged with {Spam?}) into my junk folder (as well as configuring thunderbird to trust what the spamassassin headers say) means very little, if any, spam now reaches my inbox!

Brilliant!

With the release of windows vista, comes the start-of-the-end for Windows XP. with its EOL (End-Of-Life) date now set at January 30th 2008 (that’s less than a year away), people (by people I mean windows users) who are unable to upgrade to vista (due to Lack of computing power or so) or don’t want it (its crap, proprietary, riddled with DRM and probably bugs - Microsoft are already producing SP1!) need to start looking for alternatives, unless they want to stay using an unsupported (this means no more bug/security fixes) Operating System.

Imo, The best alternative is some derivative of Linux. (Although there is others such as MacOS x86 although its not supported on non-mac hardware, FreeBSD but I don’t think its desktop oriented, and others such as beOS or so)

As of Saturday 27/1/07 I have started using KUbuntu Linux as the main OS on my desktop, as a trial to see how well I can get by without my “trusty” windows installation.

I chose KUbuntu due to its use of KDE, and the fact it was Ubuntu, which is one of the more well-established desktop-friendly distros of Linux available at this time. It also has one of the best communities and followings, as well as a solid base (Debian).

(A distro (distribution) is the term used to refer to a specific version/flavor of Linux. Other distros include Fedora Core, Redhat, Slackware, Debian and Gentoo)

The installation went smoothly, it resized my NTFS partition to allow for an ext3 partition, and installed away happily, I was even able to irc/browse the web whilst it did it due to the live cd based installer (Which is just pure genius!)

After the install, I rebooted (twice into windows first to allow the NTFS Journal to be reset for the new size, and then to confirm it was “clean”, then into Linux) and was greeted with a nice graphical login screen, that had detected the correct resolution for my monitor and everything.

The next task was to update the system, and install the Nvidia drivers so that i could use dual monitors. This was a relatively painless process with kubuntu. I edited the “sources.list” file (alt+f2 then kdesu kate /etc/apt/sources.list) and uncommented the disabled repositories and opened a konsole window (alt+f2 then type konsole) and entered the following commands sudo apt-get update sudo apt-get upgrade sudo apt-get install nvidia-glx. Alas after rebooting, X didn’t want to work and I was greeted with a text-only console. Fortunatly the fix was easy, adding deb http://albertomilone.com/drivers/edgy/latest/32bit/ binary/ to the sources.list and running the above commands again and rebooting fixed the problem.

The next task was to enable ntfs write. Following http://ubuntuforums.org/showthread.php?t=217009 I was able to simply add some repositories to my sources.list, sudo apt-get install ntfs-config and then run ntfs-config and follow the prompts. Restarted and it was done.

I now had a fully usable desktop, running both monitors at their native resolution, and i was even able to easily setup the task bar to show only tasks on the monitor it is on.

Since installing I’ve installed quite a few packages (apt-get makes it really easy) and done alot of customisation. Suffice to say i’m pretty happy with my install, and don’t think I’ll be returning to windows any time soon.

One package I would recommending installing would be beryl (Google for “Beryl Ubuntu” for installation guide, and check youtube for videos of what it can do as well as the standard window decorating).

The great thing about Ubuntu is the community behind it at http://ubuntuforums.org/. pretty much any problem you have, has been reported there by someone else, and subsequently solved - this makes problem solving a snap! (If a problem isn’t there, searching Google for “Ubuntu <problem>” usually solves most things.) Another benefit is the fact it derives from debian which is a well established distribution in the Linux community, and subsequently has a lot of packages available for non-source applications should you not want to be compiling from source all the time to add things.

A tip to KUbuntu users wishing to follow some of the guides on the forums that were designed for the standard gnome-based version of Ubuntu, open a konsole and run the following 2 commands: sudo ln -s /usr/bin/kate /usr/bin/gedit and sudo apt-get install gksudo synaptic. These 2 commands install gksudo which some of the guides use to gain root, synaptic which is the most referenced package manager, and makes the editor kate work when the guides say gedit (as gedit just looks ugly on KDE. An alternative would be to run sudo apt-get install gksudo synaptic gedit which will give you the real gedit in all its ugliness.)

Overall, I’m now a happy linux-desktop user, and I’ve yet to reboot back to windows!

You can get a copy of KUbuntu here: http://www.kubuntu.org/download.php :)

Edit: http://www.ubuntuguide.org provides a page full of instructions on how to install commonly-wanted stuff.