Does killing the captain in the counting house end the side quest? Just checking If that's the reason Mendas never came back.

Nope. Done it both ways. Killed him sometimes and pickpocked him sometimes. Mendas is always there on return. Have you been into his house after coming back with the charts? You may have to go in to speak with him before he goes out onto the docks. He goes onto the dock where the fisherman comes up to talk to you.