Are you talking about a voodoo doll, or Ardibren's Relic? You need the Relic for a quest for the Wizard's guild I think. You're supposed to meet Ardibren outside the Castle right before he.... well, I won't spoil that part, but check your journal notes for Ardibren to see what he said to you. You may have missed something.
If you have any previous saves, you could redo that part of the game, or...
If you gave him his Relic back, it may be gone for good, unless you can cast a Charm spell on someone here to hack the Relic back into your inventory.
+Dr.Pick