You're not running around with the "Light" spell on, are you? Read the description of light. You'll note it's a hovering ball of incandesence (flame).
Monsters ignore the Chameleon spell because they're attracted to the flame.
Also, it helps if you can do the Hypnotic Lure spell. Cast it AWAY from you (as far as possible, but still in their range).
But Chameleon is not perfect. It's just many people make it far less useful than it is by having the darn light spell on...