I thought if you were invisible you could only be attacked if someone cast a spell such as invisibility purge, true sight, dispel illusion, that kind of thing? Please tell me if I'm wrong, I wouldn't want to make some deadly error as a result!!
I have used both invisibility and sanctuary to get those items off the golems (with invisibility it's a bit more difficult as you have to grab all the gear and then cast invisibility - I needed some summoned creatures to distract the golems while I cast - potions are better, but I didn't have any on me). I can put up with missing the exp. because I usually have a low level, wounded exhausted party at that point of the game - I tend to do the keep early, since with the exception of the golems it is not too difficult and reaps some rich treasures.
------------------
Time is an illusion. Lunchtime doubly so.