Hmm, that's a bit of a bug I think, though it does make sense. The scripting worked out too well for that one.
The key then is to free him first, then find the stone, then he won't go missing even if you don't take him into the party.
------------------
