Have your most powerful party member talk to him. If you have a fighter/ranger/paladin, etc. I had this happen a couple of times to me. It seems he isn't scared of the little people.
If that doesn't work. Charm him and then continuously pick his pocket until you get it.