Possessing Temporary Spirit Worker Breaks Game

I've found on a few occasions that I would throw down some temporary spirit workers for a task, then attempt to possess one of them to help things along only to find that my game interface bugs out and it becomes unplayable. Basically the game stops loading anything not directly around the point where the spell was cast and it becomes impossible to interact with anything. Both times it happened to me I was trying to possess them to go all "Saving Private Ryan" on a summoned Democorn.