I'm just wondering who would fit better with my party of:

inquisitor (me) with carsomyr
mazzy
viconia
edwin
jan (imoen once I get her back)

Minsc could probably soak up more damage, but his somewhat poor AC doesn't seem to get the job done. Opinions?

Minsc won't get along with Edwin, I don't think. Minsc can wear better armor, but Valygar doesn't complain so vociferously if you lean toward evil. Your party seems to be neutral, leaning to evil, I'd pick Valygar.

<span style="color: lightblue">Like ElfBane said, Minsc doesn't get along with Edwin. Minsc can wear more armour, but not necesarily better armour. Quite a bit of the good armour is leather, and hence wearable by Valygar. Minsc, though, has much more interesting things to say, especially with his conflict with Edwin.
One question, though: why are you planning on dumping Jan for Imoen? Jan has the major advantage of being able to level up as both a thief and a mage, while Imoen can only level up as a mage - she is perpetually a level 7 thief... This makes her thief side near useless unless you give her items to boost it.

Well, not exactly true. The best leather armors are still at 1. The best Plate armors are at -2. That's still a difference of 3!

I managed to get through to the end of SoA and into the start of ToB with both Minsc and Edwin in my party without them strangely enough, ever coming to blows at all. A verbal argument here and there, but that's all part of the fun, trying different party combinations. I can't say I've used Valygar extensively yet, but his general blandness (for lack of a better word) as a character usually puts me off him and I have dropped him after doing his quests in the two times I've played the game.

Seeming as how you've gone for a nice evil touch to your party, (very similar to my current party in fact) why not even consider getting Korgan instead? Its almost a cliche around here, but he really must be seen to be believed. :D

Alternatively, the only other thing I'd maybe suggest is that you could always get away with making Minsc your bow specialist and back up tank to your already, potent party.

Minsc is definately the better tank of the two. I always wind up using Anomen as tank and Valygar as hitter. Valygar can do considerable damage with backstabs.
You'll have problems keeping Edwin and Viconia in the party because you are a Paladin, and thus have no way to lower you reputation without falling. Since you allready have Carsomyr you don't have any way to restore yourself if you fall.

Unless of course he wants to fall...

plus people always seem to forget the AC modifiers against different weapon types - leathers are weaker in that respect

my adivce,

you may consider adding Jaheira. with your inquisitor and Mazzy you only need a back up tank, and a TN Jaheira (a more than sufficient back-up tank) will fit into your party better than either Minsc or Valygar party chemistry wise. Plus, Jaheira can add some great Druid spells.
or if you want to keep the five people party and so everyone levels up faster, that'd be fine as well.

Personally, choosing between Minsc and Valygar (without concerning the party chemistry issue), I'll go for Valygar.

Hrm. Stat for stat, equipment for equipment, Valgar will make a better Tank than Minsc.