This is happening because all of your party members have stealth. Stealth makes it so that an enemy cannot see/target a character. So when monsters cannot see your party they will either defend or run away. The higher Stealth you have, the more likely this will happen.
I had a party were a class changed everyone to a Rogue for a level and raised their Stealth to 100, then class changed back on the next level. The same thing happened that you are describing.
|