Well, clay golems aren't that easy, but make sure your fighter has a mace, hammer, club...whatever because blunt weapons are all that's going to kill that golem.
Since you haven't actually got the key, you can go back outside and get korgon for that golem if you wish...
He's a great golem killer! You might want to think about using him for that particular quest anyways. Goodluck.
------------------
To quote the pestering peasant, Neeber, of tradesmeet: "
I killed a rabid rabbit! I'm experienced, now!"