You possibly just opened the door that the key opens with a lockpick or spell. If you 'went everywhere' and didn't have to leave with doors left locked because you couldn't open them, then chances are the key won't do anything for you, so I wouldn't bother going back. IIRC there's nothing particularly interesting in the places it opens anyway.
__________________
|