Hmmm...the best way to climb down a ladder? That would be....to close your eyes and JUMP!
I have been successful a time or two climbing down ladders, but usually, I get so impatient and frustrated that I end up jumping. If you pick up a trait called Deadfall, you'll take less falling damage. (Apparently the creators of the game realized how insanely difficult descending a ladder is in W&W and decided to take mercy upon us, the poor, frustrated, bruised and battered gamers.)
If I remember correctly, when I actually did succeed in climbing down a ladder, I turned my back to the ladder and slowly...and boy, do I mean slowly, used the down arrow key to back up. I made sure the ladder was centered perfectly in the screen, and I was facing the far wall squarely, not angled at all (if that makes sense). Eventually, by pushing the down arrow, I climbed down the ladder. That only happened once or twice, and then I just decided my sanity was worth more to me than saving the few hit points that falling damage would inflict!