If you're having GAL issues with 6.0.4 then definitely don't upgrade to 6.0.6. Major GAL/address-book auto-complete issues with 6.0.6. Hopefully will be fixed in 6.0.7. Doubt anything will be done with the issue concerning all contact groups showing up when you do GAL searching (your other post).
I recently upgraded from 5.0.13 to 6.0.4 and have also seen this. Right now, I am getting a lot of flack from the higher-ups on how the address book and the GAL both behave. All prior upgrades have gone smoothly and uses have been happy. This time not so good, they may never let me run another upgrade.