Bayjin is a place where monsters spawn quickly and often - but you already knew that
One time, I tried to clean out the island. Got rid of every dot on the map (except us and the prisoners). Just after breathing a sigh of contentment, I checked the map again - yep, red dots.
So, after killing as much as you can stand, grab Glumph and make tracks for Gigas (or the Sea Caves; I prefer dropping Glumph at the camp for the nice reward and then going on).
There is no time limit that I know of in the game.
Nightowl2