Welcome to the board, Snowblind!!
Are you sure you're talking about Ardibren? He doesn't follow you around, well at least he never followed me around. You meet him at the base of some curving stairs in the castle. After helping you defeat some monsters, he'll talk to you and tells you to meet him outside, south of the castle. You don't need to give him anything so I'm a bit confused about his taking something. He certainly would not take the relic as that is a quest item for town.
If you can, I'd recommend you reload this area and try it again. Something doesn't seem quite right here.
Also, check Xignal's maps :
http://www.volberding.com/WHints/index.html
to see exactly where you do meet Ardibren.
Wyv