he is the guy in the upstairs room and the copper coronet, next to the guy with the teddy bear. I just back-stabbed him for the hell of it, but I got 64 exp and no reputation loss (still on 16). I closed the door to his room before killing him, so no-one saw it, but that doesn't usually matter does it ? Shouldn't I lose reputation for killing someone unless they're meant to be killed ?

I think he is part of some quest, so you might want to save him.

He isn't.
However, he has some info on Shadow thieves, and if you have Nalia, Yoshimo or Haer-Dalis with you, they'l chat a bit.

Even more confusing, to me, is the guy upstairs in the Five Flagons. Does anyone know what he's there for, if indeed anything at all?

That guy that looks like a thief? No clue.

When I attack Yarin the entire area turns hostile, including Madam Nin (and Rumar if he's still there) even with the door shut. If you speak to him before taking the GoP he tells you he's the greatest ever pickpocket and has just retired. Afterwards, apart from the NPC interaction, it's just a warning to watch out for the Shadow Theives. (Maybe he should warn them about ME!)