Well the way I look at it is thus (bear in mind, I do powergame, and the purpose of powergaming is to produce the most powerful character possible in the long run):
17 CON to start... +3HP, not exceptional, but bearable.
+1 CON from everyones favourite tome give 18 CON +4HP, matching most human fighters.
Now this is where you need to stay with me... there are two places where we can increase this statistic further in the game (Watchers Keep and Tears). Therefore we can get a total of +3 to CON on top of the 18, giving us a total of 22 CON.
21 to 23 CON inclusive only grants +6HP, so we may as well be an elf warrior, because we end up with the same HP anyway...
