Ok, people, thanks for help.
I've replayed the last few hours of my game, and then tried to kill that warder one more time. And this time just after I defeated him, my party was automatically transported to Irenicus (of course via meeting with prisoners). No "key" was needed. So this was a clear bug (the warder's death should "trigger" the battle with Irenicus, while that did not happen).
Now everything is ok.