MM, you did kill the Big trolls downstairs AND open the front gate?
If so, and if you don't have the annoying Nalia with you, you should try to speak to her or the man in the big like dinner room. She should not be far and talking to you to give you the keep.
But I don't know for sur, just presume 'cause I had her in my party just for that quest ...
------------------
Maybe it's because I hate drows more than I hate Humans - Jarlaxle to Entreri