[Dinonykos]

If the fin was vertical, a human upper body would not make sense. With a vertical fin, you would also need a horizontally slender body with a great vertical dimension. Without such a body, flapping with the fin would not result in effective propulsion. As an alternative, mermaids with a vertical caudal fin would need a very high dorsal fin and additional anal/pelvic fins. A horizontal caudal fin would work much better with a human upper body...

Here some ideas concerning evolution of mermaids:
Possible are a mammal-back-into-water scenario or an intelligent-fish/shark-develops-hands-and-breasts scenario. The first one could happen due to sea-level rise (as in the Jurassic/Cretaceous), when almost a dozend of terrestrial vertebrates developed marine forms), while the second could happen due to low sea-levels or e.g. periodic lake drying...