Wouldn't say it SPOILS the game to not do all the quests. I only did about half of them the first time around, so now I'm having fun trying to find and do all the quests I missed.
What *is* a problem is defeating the final baddies and finishing the final chapters if you're low on XP and levels. A party with an average of level 13 will run into some difficulties in the final chapter. In my party's case, it was a LOT of difficulties, which led to a lot of me swearing at my computer and reloading the game.
------------------

Playing the cutest li'l Chaotic characters since 1996.