Second playthrough of the same map during MP without reboot causes numerous problems/errors

Previous Header: Workers stuck in "fortify" state for rest of the game after empire claims player tiles

I was playing this map with a friend yesterday: https://steamcommunity.com/sharedfiles/filedetails/?id=758896333

Everything was working fine, workers were doing all stuff they are supposed to (including claiming empire tiles) until my minions were pushed back and empire started clamining my gateway and rooms. After that all my workers got stuck, when I mouseover them description says their activity is stuck at "Fortify" and they wont move for rest of the game. Dropping them somewhere else does nothing. All new workers also get stuck in "fortify" state. The only thing that worked was slapping them in direction of enemy tile. After few slaps to get them above enemy tile they will claim only this 1 tile and again get stuck.

I was unable to continue without workers doing anything so we tried replaying this map 4 times and it happened every time after empire claimed one of my rooms. Not sure if that matters but It was always pretty late into the game (almost all upgrades researched)

I tried destroing all walls or casting fortfy spell to make sure there is nothing for them to fortify, yet they were still stuck trying to "fortify" something.

Not sure if there is something wrong with the map itself or not becuse It never happened anywhere else. That bug happened client side, friend of mine who was hosting didnt complain so I guess he wasnt affected by this bug or didnt notice because it was mostly me being pushed back all the time.

I will post output log when I get home later today.