The exp difference from 6 to 5 doesnt really warrant the extra person you're missing imho (see table below) and the extra fighter/spellcaster/healer/amusement is definetly worth it, in games without interaction (icewind dale) I limit my party, but in BG I always go for the full six. (also because some quests give exp to each person, so the more people you have the more exp gets given)
Table of increase in exp with lower party size:
| Party size | Exp gain (each) |
6 | 0 % |
5 | 20 % |
4 | 50 % |
3 | 100 % |
2 | 200 % |
1 | 500 % |
(formula is 6/partySize)
Anyways, even with six, by the end of SoA most party members are around level 25 (might be a lvl or two lower, it has been a while)