Large collection of issues, suspect memory leaks/GC

Avatar
  • updated
  • Completed - Resolved

I've been reluctant to create an issue on this since there's not that much to go on with most of them, what's mentioned at the end regarding a response I got from a map creator after reporting a nasty bug points to an issue in Unity itself, and your previous comment that a majority of your coders are working on new projects.

A few examples sorted by perceived time required to happen.

The texture and icon for gold piles changing to a forge box. Rarely see this happening to micropiglets as well, in this case its the icon when picked up.

Purely visual. 

Related: https://brightrockgames.userecho.com/communities/1/topics/4761-my-pet-dungeon-level-2-gold-pile-loaded-defense-part-texture

Corpses in canyons are not properly despawned after "reaching" the bottom. Instead they become invisible until player leaves and returns to view the area, making the corpse visible again and the falling-animation is replayed. This continues until corpse decays completely.

The previous progress is kept for captured rooms. Minions will attempt to research even if all sins are unlocked, or summon a non-existent ritual. Not sure if ritual progression is applied to original room owner.

In relation to possession mode:

SFX sound(steps, minions interacting with things, doors, combat etc) is amplified each time you re-enter possession.

Crosshair getting stuck on the screen. Reloading does not help, starting a completely different level or campaign does not help. Not visible in main menu, but is visible on pause screen as well as in normal UI.

Both should be easy to reproduce in Taktikus's campaign "Squad Leader", especially the second level as a Beastmaster. Reason being both levels take a while to finish, and you are automatically forced back into possession mode so mode-cycling is a must if you have to pause from time to time or change anything in options.

WFTO not closing down properly on exit. Either have to let Windows force close it(this program is not responding) or Task Manager.

Log gives no information, looks like a proper shutdown. Can attach debugger to catch it properly, if you give me pointers on how-to.

Possibly related: https://brightrockgames.userecho.com/communities/1/topics/2728-crashes-when-exiting-from-menu-on-steam

Ritual UI menu becoming stuck, not showing that a ritual is selected for summoning or completely charged. Ritual-display on the top screen shows correct value and can be trusted, never seen it bug out.

If a ritual is charged, its fixed by clicking that ritual to summon it. If a ritual is currently charging, you have to wait until its completely charged and summon it to fix UI.

Manufacturing Shrines loosing the limit or multiplying its spawn points. What looks like one box turns out to be 5-10 in the same spot.

Addition: Believe the cause is that current boxes are ignored on loading the level, so workers place crafted boxes(from regular Foundry). This happens on every load after a long play time, so can stack to very high numbers.

Workers(and other minions) forgetting static work-points unless you use Worker Rally to remind them (Static meaning Tavern+Foundry and if the distance is large enough even Crypt, Prison and Treasury. Moon Gate or Vortex does not always help). Without a Worker Rally for Taverns they end up running out of food and gold is not collected even with workers being in the near-by area. If you drop a Worker on the food spot(usually just north of the food cannon) they suddenly remember it again(all workers) for a short while, then forget.

Minions pathing into obstacles(Forge workspot, Crypt grave, Dungeon Core , Crucible in HoG) or unreachable areas and because of that not able to satisfy needs. Seen this frequently on for instance Gilton, third mission in Heart of Gold, with the large crucible.

Most common minion this occurs to is the Juggernaut(guessing he has the largest hitbox?) with Huntress second. In the example picture(from Big Bad Boogie 2.0) I assume the Huntress was chasing a flying beast and that there's some clip-check that ends up failing after a while on U-shaped corners. Seems to take a while to start, but after that it will happen at least semi-frequently and some levels are more prone to it.

Related:

https://brightrockgames.userecho.com/communities/1/topics/4938-worker-pathing-causing-performance-issues

Possibly related:

https://brightrockgames.userecho.com/communities/1/topics/1363-pathing-issues-minions-cant-find-locations-or-just-stay-standing

https://brightrockgames.userecho.com/communities/1/topics/4856-imps-foundry-pathing-problem

https://brightrockgames.userecho.com/communities/1/topics/300-path-finding-issue-when-fighting-far-away-from-core-base

Ghostunits, coming from units linked to Arcane Inhibitor and are supposed to be killed off after its destruction. They are not interactable in any way, arrows go right through them.

Related: https://brightrockgames.userecho.com/communities/1/topics/4963-some-of-the-opposition-walk-around-and-you-cannot-interact-with-them

Minion tags vanishing(both friend and foe), only visible by hovering over a specific minion. Fixed with saving and reloading.

Fragmented AI. Its hard to pinpoint when this happens since its quite laggy playing with a revealed map, but it seems at some point the AI forgets its previous layout and attempts to build a new dungeon on top of the old one. This can be seen by partly-built or fragmented rooms scattered around with odd placement of walls. Looking at the pattern it, at least to me, looks like someone took pieces from 2-3 different layouts and smothered them together.

It might be related to player(or other AI) pressure forcing the AI to change priorities and that if GC is working overtime it sometimes completely forgets what it was doing or what is already there mid-change, thereby ending up with a mix of various room-tiles(most non-functional) and weird wall patterns. The above images is from an autosave, an action the AI was in the middle of performing. I've increased autosave frequency in the hopes of catching the bug evolve, especially the more fragmented ones since I've seen much worse examples(entire layout being non-working for 3x3+ rooms). I can't see how those could be explained by external pressure forcing a change in priority, might be several causes.

CTA(Call to arms) not clearing for captured minions (in prison). You can see this by all of them pathing to the same spot, usually a corner nearest to exit. If it happens to one group(lets say Empire), it seems to always happen to others as well.

CTA not clearing or color tags not updating for converted minions.

The first one is fixed by giving a new CTA and clearing it. The color tag not updating properly seems to be purely visual and never seen it happen with Empire-controlled units, only those belonging to other Underlords.

Converted minions turning into drones that do not satisfy their own needs and end up rebelling as a result if one notification is missed. They seem to train or do other work but not eat or sleep until angry and won't collect pay unless player interacts.

Both CTA bugs and the converted minions issue seen on level 1 in Heart of Gold after extended playtime(around 60-90 minutes).

Ritual summon progress counter getting stuck for the Sanctuary. You can still cast a finished ritual, but units will assume there is a ritual to summon and head there.

Fixed by actually casting a ritual.

Imp flags and CTA becomes invisible. Goes invisible again after recasting them if it ever goes outside render view and returns, if the bug is present on level load.

Occasionally, the worktiles of Tavern is bugged after loading a save from a level with previous high playtime. They are empty and the food cannon does not refill them. Fixed by rebuilding the Tavern.

Other room-types are affected in various ways as well:

Micro-piglets sometimes get teleported to the same one, but are sustained by the original Hatchery they came from.

Room toogle switch(Spirit chamber, Crypt, Arena) gets reset.

Possibly related:

https://brightrockgames.userecho.com/communities/1/topics/83-tavern-broken-after-loading-a-saved-game

After loading up a 5-hour save of a custom level, it had randomly scattered Impenetrable Rock on controlled territory, even one replacing a corner piece of the players core.

I've now seen this on another level, also by Green Sliche. It was not as extreme, and some were placed on Empire-controlled territory previously untouched so its not based on Player interaction. If it replaces a Shrine or dungeon core, replacing it with tile_dirt returns it to correct state.

Seems to prefer replacing first tile of a room next to wood door, for some reason.

The creators response(the 5-hour save) was a recommendation to save the game every hour or so, return to main menu and reload, as this helped negate a Unity GC issue that could cause a lot of odd behavior, and that this is a Unity-engine issue and not something fixable on your end(at least without more hacks upon hacks and probably countless redundancy checks). This comment is what caused me to group the above issues together, since they all share the common trait of not happening early on in a level and increase in frequency over time.

Following this advice has definitely reduced the frequency of serious issues, but at a certain point it seems just loading the level is enough to "bypass" the time limit for issues to start occurring. As in, you'll start seeing various minor bugs almost instantly, and it quickly gets worse.

Related: https://brightrockgames.userecho.com/communities/1/topics/4666-impenetrable-rock-appearing-on-map

After spending some time in various playthroughs trying to pay attention to finer details in order to catch things missed in the past, one thing keeps popping up.

Whenever issues start cropping up they are always accompanied by slow tasking response(Imps take a few seconds to respond, this increases over time to reach near infinity at worst), workers forgetting path to rooms on different landmasses or over large distances, minions get stuck more often, and a microlag that is hard to notice at first unless you really pay attention but approach unplayable at worst. That microlag also spikes whenever workers are unable to find a valid path for current work, and continues until work fixes itself(e.g. corpse decay) or player builds a new dropoff-point close by(again, Moon Gate does not always rectify the issue).

Are a lot of various game mechanics - unit pathing/tasking, Mighty Script, GC and most graphical work except GPU handoff and Coherent UI on the same thread? Total CPU usage stays at 25-28% and while load is divided among available cores, one in particular is pegged fully and if I force any other instance onto this core the same sort of microlag starts appearing(6850K-HT off).

Is it possible that routines ran when saving a game times out because the thread is swamped and as a result certain information ends up missing or corrupt(eg. random Imp. Rock, Siege Shrine, Tavern), and likewise regarding GC that it ends up timing out on certain operations so various buffers(terminology..?) end up full/overflowing and causing some or most of the listed game-bugs?

PS: Sorry QA, I've tried to be as detailed as possible on the possible triggers, but its such a complex game.. :)

Game Version:
Steam Public
Platform:
Windows
Avatar
Rudolph Reindeer

"and your previous comment" referring to previous conversation with Noontide.

All I currently have to go on regarding WFTO crash on exit is lackluster Windows event logs:

https://pastebin.com/6qYuyhSh

If you PM me details I can attach a debugger to see if that yields any more information, since I can easily reproduce it consistently.

Avatar
Lee "Noontide" Moon
  • Marked for Review

Thanks for the incredibly detailed report of all these issues Rudolph Reindeer. It's much appreciated.

I'll assign it to the standard review queue as I think Cian will want to pour over this before we decide exactly when and how to address it.

Thanks again we'll let you know when we've had a look over it,

Lee

Avatar
Rudolph Reindeer

Thank you Lee. I am still impressed by the fact you guys are to this day offering active support for WFTO.

One issue I forgot to extend on:

The WFTO crashes on exit probably has two different sources. First as described above(Unity GC), but I think there's a memory leak in WFTO in relation to units being killed as well. Cannot find the issue on tracker right now, remember stumbling on it before(it was from 1-2 years ago I think, about crashes when exiting Crucible level), but its especially obvious on Crucible levels or others(HoG level 1) with a large amount of units being killed. Memory usage of WFTO will keep climbing, and if high enough result in crash just from exiting the level(and not counting Crucible score).

Highest value I've seen is 20-25GB used for WFTOGame.exe on those sort of levels.

Avatar
anonymous
  • Pending Customer

Can you get us some logs please 

Posting Output Logs

Avatar
Rudolph Reindeer
Quote from anonymous

Can you get us some logs please 

Posting Output Logs

In context of everything(after lets say 1 hour of large level/old save), or specific cases like the crash-on-exit?

Avatar
Rudolph Reindeer

Ignore last. In process of reproducing as many of the bugs as possible and provide save(before and after if applicable)+logs+steps to reproduce(if applicable). Will probably take another few days.

Avatar
anonymous

Cheers Rudolph

Avatar
Rudolph Reindeer

OK this has taken a lot longer than I initially expected. Still need to revisit certain custom scenarios that I know will trigger the crashes mentioned above, though I suspect its the same cause(s) as most others just that those scenarios pull the engine to its breaking point and GC is not able to handle the workload when exiting. Might also be a link to importing old levels, its very common on larger levels made pre-2018.

Also lacking regarding the AI issue mentioned above, on to-do list.

For clarity - "Inconsistent reproducibility" is meant as not able to find an easy way to reproduce outside actually playing the game for an extended period of time, and even then not 100% guaranteed to see most issues occur.

Minor:


The texture and icon for gold piles changing to a forge box + Piglet with forge box icon + Forge box with gold pile icon:
No log info. Inconsistent reproducibility, but seen on every playthrough of a custom scenario.
It goes both ways, but its rarer to see forge boxes with the gold pile texture or icon. Could be because of more frequent interaction with gold piles.
Piglets also take on the forge-box icon on rare occasions.

Corpses in canyons are not properly despawned after "reaching" the bottom.
No log info.
Inconsistent reproducibility outside of playing a large scenario for 1-2 hours, then spawn in enemy flying units to kill over a canyon.

Manufacturing shrines loosing the limit or multiplying its spawn points on load.
No log info.
On load of pretty much any level with 30min+ playtime. Workers will drag boxes from Forge prop-storage to the shrine(s), each load. Not sure exactly how many they deliver to each storage spot, but seems to be more than one.

3hLoadWorkerFillShrine.jpg


Ghostunits

Probably linked to MightyScript(see serious issues) failing to properly execute scripts on reload.

I assume this is directly linked to "Converted minions turning into drones", that these enemy minions that turn into "ghostunits" are either captured or converted before the event(destruction of Dungeon Core or Inhibitor) occurs that turns them into disabled units.

Shows as "Unit is disabled!" in debug menu.

UnitIsDisabled.jpg

Minion tags vanishing.
No log info.
Inconsistent reproducibility.
Its made more rare(and easier to spot) by setting shield visibility to "Always on" in options but still happens often for imprisoned minions if the count is high(10+).

Ritual UI menu becoming stuck.

Inconsistent reproducibility.

Seemingly always linked to output log with:

"An unpooled VFX Object was destroyed, elements remaining in stack 1" on ritual execution. Pretty sure I've not seen the UI bug out without that being printed in log.

This output has also become an indicator for me to expect things to get wonky in general, IE pathing becoming unreliable and that save/reload will probably results in Impenetrable Rock and/or Tavern and Crypt props being reset.

output_log-14.7GBpvb-9.8GBws.txt

14.7GBpvb-9.8GBws.jpg 
WFTORitualBug.jpeg

Medium:


Workers(and other minions) forgetting static work-points.

Inconsistent reproducibility.

See "output_log-14.7GBpvb-9.8GBws.txt" above, as well as output_log_Ren_of_Life_modified.txt below.

Seen on all large levels after 60+ minutes of playtime and high minion and/or kill count.

In regards to Tavern and refilling the cannons, its potentially aggravated by having more than one Tavern. IE the Tavern last built is given priority, and results in all others being neglected as playtime increases. Have not seen the issue with cannon-refill as often after I started having one very large tavern instead of several spread out, in conjunction with Worker Rally on cannons.

Example below had a previous instance of that Tavern emptying completely, but didn't screenshot it. After taking this screenshot and watching the 3 top-left cannons empty completely I dropped one worker near closest food cannon and all workers woke up to start filling the empty cannons as well.

11.7GBpvb-6.8GBws.jpg
output_log.txt
IdlingAlmostEmpty-EntireTavern.jpg

PS: Two Taverns, the one in screenshot and one on far left side that had no issues being refilled. Beasts "lived" in bottom left corner with their own large hatchery.

All cannons had a Worker Rally flag on screenshot, and is an example of how flags occasionally go invisible on load after leaving render view, even after replacing(Imp flags and CTA becomes invisible).

Occasionally the last-placed flag remains, as in this case.


CTA(Call to arms) not clearing for captured minions.

Inconsistent reproducibility.

I've made a habit of converting these minions to see what happens, and it usually results in them attempting to follow previous CTA after conversion. Fixed by giving a new CTA and clearing it.

Based on notes taken for #1, 20181128165019 should be before or just as the bug occurs and was able to reproduce it consistently on load, while 20181128171410 is after it has occurred. Problem is I am unable to verify this, since copying the save games back into folder does not work, I can't see them in load menu.

#1

output_log.txt 
CTABugPrison0.jpg
CTABugPrison1.jpg

the transylwania castle_20181128165019
the transylwania castle_20181128171410.meta
the transylwania castle_20181128171410
the transylwania castle_20181128165019.meta
#2

output_log.txt
CTAbug.jpg



Serious: (Suspect they are all linked to GC)

Crash on exit.

Will come back to it.

Currently only have one log, from Lockshackle.

output_log.txt

Fragmented AI.

Will come back to it. Haven't had time to properly play through a large official or custom skirmish. Potentially linked to MightyScript though.

Unknown exception linked to AI Colossus spawning.

Cornucopia(Undergames), Kasita(AI) summoning Colossus. Extreme microlag as a result.

Good luck with this one.. Had it happen twice in 5 attempts with recent patch, never seen it in previous playthroughs on old patch.

First Instance - Paused as I saw the first exceptions, took screenshot and copied output_log to help pinpoint cause.

Flood continued until Colossus was slain. Full output_log ended at 273,000 lines. No crash!

 FirstInstance_output_log.txt
FirstInstance.jpg

79,42.jpg

4 bonus round cornucopia_20181129172600.meta
4 bonus round cornucopia_20181129172600

output_log12.5GBws.txt
EndLevel.jpg

Saving does not properly save tile states or MightyScript script-states and any previously applied modifiers.

I can't see how these are not linked.

On a level with high playtime or a large number of states(gold piles, rooms, props, minions etc), the level is not saved properly and results in a large amount of issues ranging from seemingly minor to gamebreaking.

Minor:


-Tavern table prop has only 1 food slot(always west). The other 3 are visually empty, and not refilled by cannons. 

Fixed by selling prop-tile and rebuilding.

-Barracks prop vanishing completely. Fixed by selling and rebuilding.

-Forge stored boxes becoming invisible, even after picking up and placing back down. Fixed by dropping on ground and letting workers move it back.

-(Guesswork)Manufacturing Shrine box-count forgotten or prop wrongly set so workers think its a Forge storage-prop thereby on level load moving extra boxes from Forge to Shrine. Seemingly no cap when saving and loading, seen at least 15 boxes in one pile.

-Room toggle-switch state visually resetting, but not functionally resetting. Mendechaus gives wrong audio prompts.

-Room toggle-switch state functionally resetting, but not visually. Mendechaus gives correct audio prompts.

-All micro-piglets moved to a single Hatchery. Seemingly no adverse affects, they are sustained and don't die.

Medium:


-Crypt prop looses stored corpses.

-Crypt prop vanishing completely. Fixed by selling and rebuilding.

-Forge storage prop looses stored boxes.

-Torture Chamber prop interaction coordinates being set outside of level bounds, always south-west corner on the edge of the Impenetrable Rock.

Serious:


-Treasury tile loosing its tile information(has no tile name), with solid collision. Minions struggle to path around it, and is something I suspect as potential cause for observed late-game pathing issues even on reloading. Finding those tiles is next to impossible for me if its not a room tile with visual states.

Also suspect this is the beginning of Impenetrable Rock popping up, after saving and reloading the level.

BuggedTreasury.jpg

-Randomly(Seemingly) scattered Impenetrable Rock on level load. Only pattern observed is that I've never seen them pop up on neutral territory. If you can offer instructions on how to load backed-up saves(they don't show up in-game when copying into save folder), I can give save games both before and after.

output_log-Saved-NoImpenetrableRock-GoodShu....txt
output_log-Loaded-ImpenetrableRock.txt

-Parts of Mightyscript breaks on loading a save.

(Ren of Life https://steamcommunity.com/sharedfiles/filedetails/?id=1588211607 - Imported, verified ID's and actions, added two additional On Click scripts for "Gateway Spawning" and "Faction Modifier")

Parts of Mightyscript breaks in the course of playing through a level, having memory-usage/states climb, then saving and quitting.
On load, parts will not execute as they should(even with On click trigger or similar) or previous applied modifiers and never function again for that save game.

Reproduced:
"On Start" scripts and additional cloned "On Click" scripts set to a corner-block of quartz. "Actions" set to "Faction Modifier" and "Gateway spawning" units for player faction, modifying:
Unit XP, Gold Income, Mana Regeneration, Beast+Intelligent Population Cap, Beast+Intelligent spawn time, Gold Value, Starting Workers, Max Mana, Necromancer Spawn - Additional gateway and beast units.

First and second save/quit/reload worked as expected with all modifiers applying correctly.

Third load forgot previous "On Start" modifiers, and applied Unit XP, Beast Population Cap+spawn time, Max Mana and Gateway Spawning from "On Click" script.

Other modifiers were not applied. No log output at On Click execution.
It also forgot all other modifiers applied On Start(for Empire), and did not properly execute scripts when triggered - IE halting time-triggered patrol spawns on Inhibitor destruction, or applying modifiers to these time-triggered spawns of patrols(It was applied to just 1 unit in a group of 4).

Level immediately faded to black on objective completion, even though that is set to On Click with player Dungeon Core as condition.

Second load saw occasional pathing issues near end. Third load also had a few Impenetrable Rocks scattered over empire territory.
No food bug, a lot of pathing issues from about line 1725 and onwards with occasionally major microlag, mostly resolved by building prison, crypt and vault near expansion area - Moongate ineffective.
Line 2070->3742, captured a Tavern and log was spammed with these exceptions - stopped immediately after selling it. No script triggers attached to that tavern.

output_log_Ren_of_Life_modified.txt

Avatar
Lee "Noontide" Moon
  • Marked for Review
Avatar
anonymous
  • Under Review