My first game, I finished it with Minsc, Anomen, Imoen, Aerie, and Nalia. Talk about irritation.
Imoen can be irritating, especially in Chateau Irenicus. Almost worth getting Ease of Use mod just to install Dungeon Be Gone. However, later in the game, she cools down with the talking and becomes a lot easier to deal with. That's when I like having her in the party. In Chateau Irenicus, I just want to throw her off the edge of the air platform.
Anomen is incredibly annoying. He is downright boorish in his arrogance. The worst part is that he ends up in my party almost as many times as with Imoen, because he is a useful character in the party. I hate hearing him talk about the "ill-conceived group"; you're a part of it, aren't you? If anything, I find him to be worse after passing his test if your reputation is high, because he says that stupid line over. And over. And over.
Aerie, until you get to a certain point in the romance, is so incredibly annoying. She is beyond clingy; she might as well be Superglued to you. It drives me nuts. Once you get past that point in the romance, however, she is much easier to deal with. But she still remains irritating. And, considering I play good most of the time, she's pretty much my main squeeze...
Minsc is not too bad, mainly because he is good comic relief. However, he does get irritating at times. Still, I can't help but like the guy.
Nalia is truly obnoxious. They should do a mod where Nalia and Anomen fall in love with each other and run off from the party, just to have a boulder fall on them and kill them both. Nalia is mildly less annoying than her aunt, and combined with the stupid Roenall involvement in your life, just makes you not want to have her in your party.
Other really irritating NPCs: Jan (some people love his stories; I find them incredibly irritating), Cernd (waste of oxygen), and Haer'dalis (the next time he refers to himself as a sparrow I will get a rifle and go sparrow hunting).