Gee, I don't know -
I only did in the Leprechaun once when I thought you had to do that to fulfill the quest (my first game)and I did that out in the wide open area where you first are annoyed by him. I just kept following him around and blasting him, healing and blasting him again until he was gone. When I learned that I didn't need to off him, I reloaded my game.
I certainly never tried to do him in near his treasure chest. He suddenly appears when you get there. He might be capable of suddenly disappearing too. I just don't know.
Wyv