I'm a pacifist, but I can't blame ya. Sounds like the guy deserved a spanking.
You could have punched the guy, but instead you left your palm flat. So instead of a slap on the wrist, he got a slap in the face. No real physical harm done.
I would be wary of retaliation though, you have humiliated this guy. He might get some buddies and try to jump you in an alley or something, or even try to sue you. I'd keep an eye looking back for a while if I were you.