Hmm... It seems you haven't the right key. The key you need cannot be removed from your inventory. When you find such a key, it'll be the thing needed for activating portals
.
You can find it by yourself (searching carefully) or dryads can tell you where it is - if you help them
Hope this helps
And welcome to the forum !
------------------
Tanda,
with best regards from Guild of Assassins