I do like a 4-character party myself. That's what I tended to play in Wizards & Warriors. What you lack in manpower, you make up for in higher levels earlier on.
I'm enjoying my six-character party right now (eight if you count the NPC's) but I'm playing at "Expert" and get no end of XP.

I may try a 4-crack party later on.
Cheers,
-Saz