CON: Worse fighting ability, for one (obviously).
CON: You will have more trouble in the beginning of SoA as a low level fighter and then a very low level mage on top of that.
CON: You won't gain many HP later on.
PRO: You'll get the opportunity to advance farther in mage to gain more spells and get more of a certain type of spells in ToB as well.
PRO: Time stop, MMM