Another way to go down ladders when you're not facing it (like toad village) is to:
1- approach the ladder backward (like Saz pointed out) but to look slightly up. When your guys start falling, you start "walking" forward, catch the ladder, look down and go down the ladder.
or
2- approach the ladder from the side, jump off, turn to face the ladder facing a bit downwards, start "walking", catch the ladder and go down.
They both take a bit of practice but you'll get use to it.
Saz: What about the ladder in the Dragon Spire or the one in Skull Castle?
