Extreme lag with Idle Workers

I have been experiencing extreme lagg when playing and i recently noticed that it is related to workers on Idle. I tested with 16 workers on 4 goldshrines and about 40 units on the map and the game runs smothly, but as soon as i add 1 worker the game starts lagging like crazy and my fps goes from 60 to around 5 or 10.

I also loaded up a few saves were i was having bad lag and removed the idle workers and the lagg immediately stopped. I made a quick search on the forum but i did not find anything about this problem, please fix :)

Otherwise i love the game!