Stuttering every 10-15s

Just to give an update: It might happen that Unity 5.5 final gets out of beta during the next couple of weeks.
(https://unity3d.com/unity/roadmap)
This would fix the stutter problem for you guys.
With some luck it will make it into version 1.5 which is supposed to be released mid of December but it is yet unsure if the new Unity version will be ready at this time.
If not, it will be likely be in 1.5.1.
Stay tuned! :)

Same issue, win10, i5 4960k, 8GB ram and R9 290X, same as OP, no matter the settings/quality it just starts stuttering after some point (size/number of minons/dungeon explored seems to make it more severe)

And that is playing on the very latest release of 1.5, I guess the Unity update didn't get into this patch?

This should improve once we got integrated to the new Unity version.

wow Unity 5.6 "Vulkan Support"

that's great news :O

Yea and hopefully with generational garbage collection, which we are eagerly waiting for,

a question

the game (now) supports the Dx12?

I have windows 10 x64 and R9 390 (dxdiag.exe says the dx 12 are installed and Total War: warhammer with dx 12 works perfectly) ... I tried to force dx using "-force-d3d12" but I view a black screen at startup

This entirely depends on Unity3d, it should work but obviously doesn't for you.
But since it is a new feature it is likely to be fixed for everyone soon, but this is not in our hands.

Any progress on this issue? As much as I enjoy this game, this "1 frame/sec" every 3-5 seconds gets infuriating.

Mobo: Gigabyte GA-990FXA-UD3 rev.4

CPU: AMD FX-8350 x8 4.0GHz

RAM: HyperX 16GB DDR3 1600MHz Dual-Channel

VGA: Gigabyte Radeon R7 260X 2GB GDDR5

OS: Windows 7 Ultimate 64-bit

All software is up-to-date

We are currently migrating the project to Unity 5.6, this is a fairly long process as many many things have broken since we have made the move.

We don't yet have a timeline for release of the Unity 5.6 version as the last thing we want to do is release a more broken version of the game, but it is coming.

Hey Antti,

Could I get you to read our How to: Submit a Bug Ticket guide and attach the required info?

12 subjugation_20170426002621.meta
12 subjugation_20170426002621
output_log.txt

I completely forgot this. Here are the requested files.
-A

Hey Antti,

I played through the save game you sent me, while playing I didn't personally experience the drastic drop in performance you're experiencing. My guess is that the lag is caused by Unity's Garbage Collection, this is when the game engine comes through and clears out the memory for things that are no longer in use; when it does this there is a performance hit due to it having to scan all the in use memory and deallocate it.

This should get better in the coming months as we upgrade to the newest version of Unity which contains minor improvements to this formula. Unfortunately we are forced to wait until a stable version of the engine is released, but this shouldn't be much longer.

I want your setup to finally finish the game :D

Haha unfortunately I can't ship it over. I have heard some users having a better time by completely disabling their paging file in windows

Give that a go and let us know if it helps

I tried moving the game to my ssd drive if it would have made any difference and also took a video this time.

You can really see the chopping in this doomed gamesave after the first minute when the screen isnt moving but the lag is real :O

Are you running at the lowest settings?

I don't doubt that the issue is happening theres just not much to be done

I never actually tried lower settings :O

I would strongly recommend not playing on Ultra, put it to low and the lag should be much better. A lower amount of Pixel light counts would also help

But as mentioned it only started happening on this level so something is propably up anyhow.
...some time later...


Well here is video with the lowest settings and the same chopping still occurs :(

Have you disabled your paging file?

Makes no change to game performance in eather situation sadly. :/

Well then we're just going to have to wait until the next major patch

i hope migration to the newer unity engine release and the problems of garbage collection is of the utmost importance...

i love this game but the hitching that starts to occur every 3 seconds after you're 45 minutes into a dungeon makes the game unplayable.

to this day i still haven't completed the heart of gold campaign because of this problem...

Hey drivetheory,

Unfortunately we are still waiting for Unity to upgrade their garbage collector. This is something that is completely out of our hands, we will keep everyone up to date with any information we receive from Unity themselves.

I would recommend in the meantime for you to save the game and reload.

i know it's out of your hands, i've been following this issue for a while now, thank for the reply though!

We have released a patch today that gives a small performance bump, it likely won't solve the issue but may make it more bearable

We have released a patch today that gives a small performance bump, it likely won't solve the issue but may make it more bearable, please let us know if it helps

Hi, Unity developer here. This is an issue because the Garbage Collector keeps firing far too often. There is no engine that has a garbage collector that can handle the amount of memory garbage that the game is producing, and there is no garbage collector in the universe that can handle a persistent memory leak. Garbage collector is there for the occasional garbage generated by single-run routines, it is not designed, and cannot handle per-frame produced garbage. The stutters happen because the garbage collector reaches a critical mass threshold and it dumps its heap.

In old terms before proper GC handling this would be referred to as a severe memory leak. The engine's robust nature and the massive amounts of memory and powerful CPUs we have nowadays compensates the issue, but it is still a severe issue. Your programmers, not those of Unity have to find out what is producing the garbage, and make it produce less garbage. Unfortunately there is absolutely no magic bullet for this. General rule of thumb is to move as much as possible away from Update loop, though the easiest thing to do is to ensure there are no foreach statements that fire very often, simply replace them with for statements which do not produce garbage. Another quick fix is to ensure there are no "new" keywords within update loop(s).

It's a great game otherwise, but the very janky state of the game's programming, especially this late to the release is severely hampering enjoyability. It stutters on a new generation i7 with 32gb RAM and 1080GTX so much that i have to restart the game after every mission because your programmers have not done proper cleanup routines. Unity comes with a profiler to figure out these sorts of issues easily.

If i had a week with the code i could fix the memory leaks, but i find it unlikely you'd give me access to the source. You should've done something about this way earlier instead of trying to shift the blame on Unity when you're, in an analogue, using a bed frame as a ladder.

Here's some reading about the subject: https://unity3d.com/learn/tutorials/topics/performance-optimization/optimizing-garbage-collection-unity-games

Hi Kanakotka.

The subject have been discussed countless time, and as such I will be brief here.

1) The game is coded in C#, so there is no "before proper GC"

2) Big project end up using multiple framework, which are black boxes that might generate GC without us easily knowing (Unity being one)

3) You mention foreach for example. It's such a simple language feature yet actually improperly handled by mono and produces GC. It shows that "the bed on which we lie" is not very comfy right. And the list on those incorrect handling goes forever. A simple event subscription ? Mono considers it needs to regenerate the full list of subscriber, producing GC. A simple access to direct property of Unity, like transform ? A GC is produced for marshaling the ref from their native code.

4) You mention leak. People have played for more than 4h straight without OOM. I'm not saying it is not possible to trigger one, I'm sure some unfortunate scenario might, but we can park this one at least.

5) Have you ever tried to profile a game of this scale with Unity tools ? Probably not by the sound of it. The profiling just become unresponsive after a certain amount of nested calls. And due to the code architecture we have, there is quickly too many nesting for it. Not to mention the constant crashing it produces. Or the fact that Memory snap never worked for us.

So we tried cleanup routine. There is no "new" in update, there is no foreach used anymore. We're using pooling as much as we can. We had optimization sprints. I'm not saying we're perfect. There is no doubt room for improvement. But due to the nature and the scale of the project, those point you mentioned are very far from being as simple as you make it sounds like.

Hi. I mostly program in C#. While there are no tools like heap allocation and manual garbage removal (apart from forcing GC to do its thing) it's entirely down to the programmer to manage their garbage regardless of what they program in. To someone unfamiliar with garbage and its allocation, the hoops you have to jump through may seem very strange and even counter-intuitive in places, but the goal is to ensure minimal impact on GC Alloc in the profiler.

Essentially, there should never be per-frame allocation of garbage outside of obvious cases (streaming content, loading, certain usages of coroutines).

2)

All projects end up using multiple frameworks. Even though Unity tries to fill the gap, it is not "one program to rule them all" style of affair. Although what it has largely achieved is removal of dependency from using multiple languages to work on the project.

3)

Foreach is actually "improperly" handled by the .NET framework Unity uses. To be specific, it is improperly handled from the viewpoint of being used in a game engine; it is excellently handled from the viewpoint of a generic programming language. While patch 5.5(IIRC) made a workaround to prevent boxing usage in foreach clause, the end effect is a less efficient foreach. Keep in mind game programming is extremely abnormal when compared to normal, sequential programming.

You argue that it is improper for Unity's GC to throw away memory that's not being used, something that it's designed to do, such as a temporary variable access of parent or child of an object, yet any programmer should know to cache commonly used variables such as that. Imagine that you made a flexible regex algorithm and instead of simply writing "include System.Text.RegularExpression;" you argue that the compiler should instead arbitrarily understand to, without direction, cache the variable when you call for it each time with "new System.Text.RegularExpression.Regex".

It may seem counter-intuitive at first, but keep in mind Unity allows usage of these "improper methods" to make concepting faster. It's easy to forget to cache your variables in memory if they're used often. A very simple change in code will provide a massive performance improvement. This is not "because Unity", it's because how object-oriented languages and nesting works. It'll produce massive amount of garbage in a standard .NET program as well, but since an office program for instance doesn't need to update its view tens of times per second, it's not as visible. :)

4)

The Garbage Collector is handling the memory leak. Any GC alloc in the profiler is leaked memory. You have a severe memory leak if the GC is forced to run its routine more often than say, once every 5-10 minutes. In the later missions it's noticably being run every 20 seconds or less. The worst case so far was the mission before Mira's mission. (avoiding spoilers)

5)

Yes. While i'm not at the liberty to say which game, i can blurt out some ballpark numbers about 300,000+ lines of code and 20-25 developers not counting the QA.

The profiler doesn't need to run perpetually and you can create a test environment for its specific needs. From just a "fly in the ceiling" perspective, the major garbage producers seem to be related to creature behavior in some manner as the problem seems to compound with larger amount of creatures. Though a garbage issue should show itself in a small scale scenario as well in the profiler's GC Alloc listing. Ideally this should be <1kb / frame with 100 entities.

I'm afraid you misunderstood me. There are some very easy measures you can do right now listed in the link i posted, but those very easy measures are just the very tip of the iceberg. On the long run, there is no magic bullet for this unfortunately. The absolute best you can do is to follow the guide and see what comes up.

I hope for the best, because i really, really want to recommend the game onward. Despite the apparent negative tone of these posts, i absolutely adore what you're doing with it. I just really wish it didn't stutter, because that just takes me out of it so much. I guess this is more of an issue from a game dev standpoint, because it just makes me want to get the wrench out and come and tweak some bolts.

I appreciate the intent. But there are still a few point of miscommunication. 

2) We're using uLink framework for networking, it uses obfuscation and produces GCs on update. We're using CoherentUI for our UI, and isn't innocent either. And we have more frameworks. Those are the black boxes I meant, that we have no control over, like Unity.

3) Let's forget foreach, it's minor compared to the rest of the hurdles we encounter working on the old Mono version. My point was you make it sound as easy as hunting a few foreach, and caching variable (which we did). But when you realize a simple MyEvent += MyMethod (or -=) procudes GC, it does change your whole architecture, and since it is optimization, you generally realize that too late late in the process.  

4) You have a different conception of what memory leak is in a managed environment than me. GC can only collect what have been released. In my definition a leak is when the memory grows until OOM (so constant allocation never released unknowingly by the coder). I'm not contesting we have GC being produced.

5) Well to shed some light we have well over 300k+ loc, and ... 2 coders ;) I do like your suggestion of running in an isolated environment, but realistically it is not currently doable with the resources at hand.

And I also appreciate the wish to help. If you're really confident in your ability to fix the issue(s), with the constraint we have (we are stuck at 5.4 for the time being), you can always PM me for further discussion :)

2) That networking isn't on in singleplayer, is it? My own issues are in singleplayer, multiplayer matches generally don't last long enough for the issue to surface for me. UI i can understand, as a somewhat flimsily coded UI can produce mass garbage. If your bought frameworks are causing notable performance dips though, you should contact their developers.

3)

Unfortunately += and -= are notoriously garbage-filled because what they actually do behind the scenes, and i can totally understand the sweeping changes that would need to be done for. What i mean by "easy" is that when you know what to avoid, it's reasonably easy to just scroll through and replace certain things and that alone could be beneficial for performance. More invasive, "proper" optimization is always extremely hard and work heavy.

4)

It's essentially a memory leak being handled by GC automatically because the GC inclusively tracks what the code is doing at any given point in time. It would be the classical case of memory leak if it managed to get past Unity's GC, but i just prefer to use the term because as far as i know memory leaks as we knew them are things of the past outside of making your own engine or something.

5)

Having just 2 coders on a project of this scale is scary... and impressive. Hats off to you guys.

Unfortunately right now i'm full-time employed with a project for at least a month and i couldn't manage to concentrate well enough on an optimization pass. After that though? I can get back to you :)

Unfortunately the project i'm working at is 5.3x version and i don't have any experience of porting larger scale projects to newer versions (which unfortunately often isn't worth the trouble)

Hi Darkassassin98,

Thanks for the praise for the game and for your support. I can say that we truly appreciate the frustration that comes from the GC Stuttering, for us as developers it's extremely irksome to watch the game struggle in later stages and we keep trying to look for ways to reduce it. We've made some changes internally which should improve the problem very slighty but as you may know the true solution lies with the planned new low latency & generational garbage collection algorithm that is planned by Unity.

Unfortunately we currently have no timeline for this feature and we've been waiting for some time on it, from our side there's sadly not a great deal we can do at the time being but that doesn't mean we'll stop looking for further optimisations. All I can say is that we do still plan for some sort of long term support of WFTO but I can't speak to the scope of support this might entail, it's certainly not something we want to leave unaddressed however and it's frustrating that a solution has thus far eluded us.

It is our hope that should the new GC be added in Unity that we'll do our best to bring the game up to that version simply to make use of it. However given this unknown time scale for the feature we can't say how likely it is to happen. Upgrading the engine is a time consuming task and can even be roadblocked by support issues with other middle-ware that exists in WFTO. We're looking at options to keep the game as up to date with the engine as possible and at the very least we'll be keeping our fans informed.

I'm sorry that I can't offer anything more concrete at this time but I hope that as we get closer to 2.0 we can at least have a more clear answer in sight.

I'll look to merge this with the lead topic on this matter.

Cheers,

Lee

Hi ShadowMajestic,

Unfortunately the issue you are experiencing is a known issue and is due to Unity's own Garbage Collection algorithm (which is outdated and in all honesty kind of rubbish) rather than WFTO itself. Unity are "looking at it" and working on a new (and improved) algorithm which should fix/solve the issue but there is no known timelime for it to be finished/done.

In the mean time Brightrock themselves continually optimise the game as best as they can to squeeze every last drop of performance possible from the engine but ultimately this is an issue they can only put a bandage over. Unity themselves are the only ones who solve this issue.

The best thing I can suggest is playing the game on low settings, and when you start seeing the issue, save and reload the game (which clears all the garbage and will temporarily stop it happening for a while).

Sorry, because i'm sure that's not the answer you were looking for but I hope it helps a little and you can still enjoy the game whilst we wait for Unity to get their act together.

Hey Andrew,

Thank you for your reply.

I do not experience similar issues in other Unity games though, like 7 Days to die(I had similar issues in the past though, very tiny stutters every 5-10 seconds or so, but they were hardly noticeable with the other stutters at the time), which at times, is a bit more demanding that WftO. But they are entirely different games, so it is hard to compare these of course. I do recall an issue with minor freezes a couple of patches ago(i think it was alpha 14), their PlayStation porting partners found a solution for it. Might be related, most likely is not. (I think Joel talked about it on his dev blog at one point). If it might be helpful, I can try and find this.

For now I'll just try and stick to the smaller maps with the lowest settings where the freezes are sort of tolerable(your own maps seem to do better than the community maps). And on other fronts, fantastic work on the optimizations. From 30FPS which dropped to 10FPS or less after <30minutes of gameplay, to a steady 60 FPS all around (If I ignore these garbage collecting freezes). Really enjoying the fact that my crappy GPU can easily draw all the pixels you can throw at it.

Of course I was looking for an solution, but the main reason to post it was to make sure you guys were aware of it. As I couldn't really find others with similar issues. But you guys are, which is awesome. Thanks for the explanation.

I'm glad to hear that your overall FPS is higher. We are always trying to make the game more performant, hopefully you will see that as we march forward with WFTO development

As for the stutter we cannot do too much at this time to correct it as we wait for Unity to improve their Garbage Collection solution, as Nutter said

I remember hearing that garbage collection was improved in newer versions of Unity; what version is WFTO built in? The game still ends up freezing for a couple of seconds, every ten seconds or so, after an hour or so into a game.

Hey Rohaq,

Unfortunately no improvements have yet been made to the Unity GC. We are still monitoring the situation.

WFTO is built in 5.4.6f1. You can always check the output_log after playing to double check that, it should be on the first line.

There should be an improvement next patch, please let us know

I have since swapped out my old GTX 770 for a GTX 1070 Ti so my WTFO experience has changed since my post a year ago and I no longer experience the stuttering because I threw money at the problem.....

However I can say that at regular intervals I experience a significant frame rate drop (90 FPS to 20 FPS), which i suspect is perhaps being caused by the garbage collection routine? (polling done at 100ms in the graph below).

How regularly does this occur? How long have you been playing before this occurs

The intervals I can't say precisely, not every 10-15 seconds though, longer than that.

Shortly after that screenshot i had to save the game and exit WFTO.

I started to notice the drop in frame rate probably after 1.x hours of of playing through a single level/map.

Unfortunately this is the best it's going to get, there's not much more we can do

So this periodic chug to low framerate is some 'garbage collection' going on in the background and will never be fixed?

I've just returned to the game again and have been suffering an awful chugging at regular intervals, even on tiny maps.

It interrupts play by dropping inputs while it's going on; or at least some freeze does.  I think there's both a 'low framerate' chug and a 'freeze' chug.

This feels worse than it did in the past.

Essentially we've made a huge improvement to it. It should now be something that very occasionally happens but should not interrupt you playing.

@Ace if you have a log and your system specs I can look into it. It should not be occuring at regular intervals.

Posting Output Logs

Output log: https://hastebin.com/asicatesob.cs

I did a test stream which probably picked up the issue, but I can't quite remember when or where and don't have the time to sit watching it right now.  Here's a link if you do, linking time code to where I started the crucible where I really noticed the issues: https://www.twitch.tv/videos/308530119?t=47m41s

Was that log from when you were playing Crucible?

No, I think that log was just from 2 campaign missions I did trying to get the speedrun achievement (mission 2).

Did another round of testing, this time with a whole bunch of stats on screen.

Output log: https://hastebin.com/hahonidalu.cs

Twitch archive: https://www.twitch.tv/videos/309048949

Seems there were both chugs to a huge amount of GPU usage and a dropping of GPU usage, both combined with frametime spikes.  You can see big hitch here: https://www.twitch.tv/videos/309048949?t=11m47s and then another at 12:15 combined with some FPS wildness.

Framepacing itself was all over the place, especially with the Sins of Evil(?) open and a bunch of units on screen.

Overall, it was a small map so the issues were much less severe.  I'll have to test with a big 4 player map or late campaign mission or something. (I'm still re-learning the game and half tempted to reset my campaign to play through again, but then I lose all previous progress, so, meh)

I'm really not seeing any solid freezes, just drops in framerate. for the most part you are pinned at 75FPS so I wouldn't expect the usage on anything to be 100%.

High GPU usage is not a bad thing, unless you are at your max fps(defined by vsync etc) you will see 100% usage in any game.

You will get drops just due to the fact that GC has to happen at some point.

The VOE is renderer in our UI middleware, if that causes it to lag then theres not much we can do