You can return but you have to make sure you make a portal before you jump the Spire ledge (you'll understand when you get there
). You'll have to walk all the way back to Ishad N'Ha. But when you want to come back, you won't have to make your way through the Spire.