Stuttering every 10-15s

Avatar
Rubyeyed
  • zaktualizowano
  • Completed - Resolved

Hi, i have these like frame skips/stutters every 10-15s and in mission 3 (HoG) they get more severe. What i mean is they are longer more like a second long which makes it very annoying and distracting. It happens regardless of the settings and what happens in the game. I can start a mission and move camera in circles around my core and it will still stutter every 10-15s. Although, in skirmish when the map is small and nothing much happening on it the stutters are so brief it is hard to notice them. On big maps with units etc. already on them (like HoG lvl 3) the stutters are longer and are a real pain.


Windows 7 x64

12gb RAM

i5 2500k @ 4.2

gtx 970

game version 1.4.0f4

output_log.txt

Game Version:
Steam Public
Platform:
Windows
Duplikaty 4
second to last mission unplayable due extreme lag

Every second or so the game just freezes/lags a bit. Its unpossible to finish the campaing. Im running I7-4770K @ 3.5GHz and GTX970 and 16GB ram.

Stuttering during play
I haven't played this in a while so with the release of the new DLC I decided to give it another bash, I've never completed it since low fps on later levels caused the game to become unplayable for me, less than 1fps.


I've built a completely new system since last time, wasn't expecting any issues but I've just completed the level to Defile Uthors Tomb and the last half of the level I kept stuttering every few seconds, I wouldn't think this should be an issue on my system at all but perhaps there is a fix?

My System:
i7-5820K
24GB DDR4 2666MHz
AMD R9 Fury-X

Windows 10

output_log.txt

error.log

Please fix the constant stuttering

I have a really powerful computer, i5, Gtx 970, 32 GB RAM and the game constantly stutters for a fraction of a second. This is really annoying and also often causes miss-clicks, or clicks not working, and is really immesion breaking.

I read that this is due to the engine you use and the garbage collection of that engine. I also read that you are wrapping up extra content for this game in the next patch. I bought this game twice. Once on steam and once from GOG because I really loved DK2 and I really love this game. You guys have done an amazing job and I wanted to support you. I also like my games DRM free. However please don't leave this game with such an annoying bug on an otherwise amazing game. This bug literally affects all campaign levels, skirmish, multiplayer etc.

Please fix this issue soon.

Freezes after 5-10min each 5-10 seconds getting worse overtime
  • Specify your platform
    • OS Windows
    • Version 10
    • 64bit
      • Ram 16GB
      • CPU FX8350
      • GPU GTX 760
  • Specify the build you are using (there will be a timestamp visible on the main menu)

        1.6.4

  • Include as much information as possible (what map you were on, screenshots, videos, etc)

        After about 5 to 10 minutes, a small freeze starts appearing every 5 or so seconds. This freeze gets progressively worse the longer you play. I have no issues in between those time frames, the game runs smooth.
There seems to be little to no effect if I change the graphical options from the highest to the lowest possible settings (while keeping it 1080p though). The map size does seem to matter on how fast the freezes start appearing, under 100x100 it can take up to 10minutes. On the larger ones, it'll start after ~5 minutes.


I feel like the main thread or game ticks are not getting enough resources. I see 2 cores basically at a 100% while the other 6 are mainly idling.


I've been struggling with the performance since for ever now. In the past I was able to play the smaller maps decently, now I can't comfortably play this game at all.



  • Search the Bug Tracker for any similar or identical open reports, if one exists comment there instead of opening a new ticket

       I did a search, didn't seem to find anything related to this.

Odpowiedzi 63
Avatar
0
Kanakotka

Hi, Unity developer here. This is an issue because the Garbage Collector keeps firing far too often. There is no engine that has a garbage collector that can handle the amount of memory garbage that the game is producing, and there is no garbage collector in the universe that can handle a persistent memory leak. Garbage collector is there for the occasional garbage generated by single-run routines, it is not designed, and cannot handle per-frame produced garbage. The stutters happen because the garbage collector reaches a critical mass threshold and it dumps its heap.


In old terms before proper GC handling this would be referred to as a severe memory leak. The engine's robust nature and the massive amounts of memory and powerful CPUs we have nowadays compensates the issue, but it is still a severe issue. Your programmers, not those of Unity have to find out what is producing the garbage, and make it produce less garbage. Unfortunately there is absolutely no magic bullet for this. General rule of thumb is to move as much as possible away from Update loop, though the easiest thing to do is to ensure there are no foreach statements that fire very often, simply replace them with for statements which do not produce garbage. Another quick fix is to ensure there are no "new" keywords within update loop(s).


It's a great game otherwise, but the very janky state of the game's programming, especially this late to the release is severely hampering enjoyability. It stutters on a new generation i7 with 32gb RAM and 1080GTX so much that i have to restart the game after every mission because your programmers have not done proper cleanup routines. Unity comes with a profiler to figure out these sorts of issues easily.


If i had a week with the code i could fix the memory leaks, but i find it unlikely you'd give me access to the source. You should've done something about this way earlier instead of trying to shift the blame on Unity when you're, in an analogue, using a bed frame as a ladder.


Here's some reading about the subject: https://unity3d.com/learn/tutorials/topics/performance-optimization/optimizing-garbage-collection-unity-games

Avatar
0
[Dev] Nanorock Programmer

Hi Kanakotka.

The subject have been discussed countless time, and as such I will be brief here.

1) The game is coded in C#, so there is no "before proper GC"

2) Big project end up using multiple framework, which are black boxes that might generate GC without us easily knowing (Unity being one)

3) You mention foreach for example. It's such a simple language feature yet actually improperly handled by mono and produces GC. It shows that "the bed on which we lie" is not very comfy right. And the list on those incorrect handling goes forever. A simple event subscription ? Mono considers it needs to regenerate the full list of subscriber, producing GC. A simple access to direct property of Unity, like transform ? A GC is produced for marshaling the ref from their native code.

4) You mention leak. People have played for more than 4h straight without OOM. I'm not saying it is not possible to trigger one, I'm sure some unfortunate scenario might, but we can park this one at least.

5) Have you ever tried to profile a game of this scale with Unity tools ? Probably not by the sound of it. The profiling just become unresponsive after a certain amount of nested calls. And due to the code architecture we have, there is quickly too many nesting for it. Not to mention the constant crashing it produces. Or the fact that Memory snap never worked for us.


So we tried cleanup routine. There is no "new" in update, there is no foreach used anymore. We're using pooling as much as we can. We had optimization sprints. I'm not saying we're perfect. There is no doubt room for improvement. But due to the nature and the scale of the project, those point you mentioned are very far from being as simple as you make it sounds like.

Avatar
0
Kanakotka

Hi. I mostly program in C#. While there are no tools like heap allocation and manual garbage removal (apart from forcing GC to do its thing) it's entirely down to the programmer to manage their garbage regardless of what they program in. To someone unfamiliar with garbage and its allocation, the hoops you have to jump through may seem very strange and even counter-intuitive in places, but the goal is to ensure minimal impact on GC Alloc in the profiler.


Essentially, there should never be per-frame allocation of garbage outside of obvious cases (streaming content, loading, certain usages of coroutines).


2)

All projects end up using multiple frameworks. Even though Unity tries to fill the gap, it is not "one program to rule them all" style of affair. Although what it has largely achieved is removal of dependency from using multiple languages to work on the project.


3)

Foreach is actually "improperly" handled by the .NET framework Unity uses. To be specific, it is improperly handled from the viewpoint of being used in a game engine; it is excellently handled from the viewpoint of a generic programming language. While patch 5.5(IIRC) made a workaround to prevent boxing usage in foreach clause, the end effect is a less efficient foreach. Keep in mind game programming is extremely abnormal when compared to normal, sequential programming.


You argue that it is improper for Unity's GC to throw away memory that's not being used, something that it's designed to do, such as a temporary variable access of parent or child of an object, yet any programmer should know to cache commonly used variables such as that. Imagine that you made a flexible regex algorithm and instead of simply writing "include System.Text.RegularExpression;" you argue that the compiler should instead arbitrarily understand to, without direction, cache the variable when you call for it each time with "new System.Text.RegularExpression.Regex".


It may seem counter-intuitive at first, but keep in mind Unity allows usage of these "improper methods" to make concepting faster. It's easy to forget to cache your variables in memory if they're used often. A very simple change in code will provide a massive performance improvement. This is not "because Unity", it's because how object-oriented languages and nesting works. It'll produce massive amount of garbage in a standard .NET program as well, but since an office program for instance doesn't need to update its view tens of times per second, it's not as visible. :)


4)

The Garbage Collector is handling the memory leak. Any GC alloc in the profiler is leaked memory. You have a severe memory leak if the GC is forced to run its routine more often than say, once every 5-10 minutes. In the later missions it's noticably being run every 20 seconds or less. The worst case so far was the mission before Mira's mission. (avoiding spoilers)


5)

Yes. While i'm not at the liberty to say which game, i can blurt out some ballpark numbers about 300,000+ lines of code and 20-25 developers not counting the QA.

The profiler doesn't need to run perpetually and you can create a test environment for its specific needs. From just a "fly in the ceiling" perspective, the major garbage producers seem to be related to creature behavior in some manner as the problem seems to compound with larger amount of creatures. Though a garbage issue should show itself in a small scale scenario as well in the profiler's GC Alloc listing. Ideally this should be <1kb / frame with 100 entities.


I'm afraid you misunderstood me. There are some very easy measures you can do right now listed in the link i posted, but those very easy measures are just the very tip of the iceberg. On the long run, there is no magic bullet for this unfortunately. The absolute best you can do is to follow the guide and see what comes up.


I hope for the best, because i really, really want to recommend the game onward. Despite the apparent negative tone of these posts, i absolutely adore what you're doing with it. I just really wish it didn't stutter, because that just takes me out of it so much. I guess this is more of an issue from a game dev standpoint, because it just makes me want to get the wrench out and come and tweak some bolts.

Avatar
0
[Dev] Nanorock Programmer

I appreciate the intent. But there are still a few point of miscommunication. 

2) We're using uLink framework for networking, it uses obfuscation and produces GCs on update. We're using CoherentUI for our UI, and isn't innocent either. And we have more frameworks. Those are the black boxes I meant, that we have no control over, like Unity.

3) Let's forget foreach, it's minor compared to the rest of the hurdles we encounter working on the old Mono version. My point was you make it sound as easy as hunting a few foreach, and caching variable (which we did). But when you realize a simple MyEvent += MyMethod (or -=) procudes GC, it does change your whole architecture, and since it is optimization, you generally realize that too late late in the process.  

4) You have a different conception of what memory leak is in a managed environment than me. GC can only collect what have been released. In my definition a leak is when the memory grows until OOM (so constant allocation never released unknowingly by the coder). I'm not contesting we have GC being produced.

5) Well to shed some light we have well over 300k+ loc, and ... 2 coders ;) I do like your suggestion of running in an isolated environment, but realistically it is not currently doable with the resources at hand.


And I also appreciate the wish to help. If you're really confident in your ability to fix the issue(s), with the constraint we have (we are stuck at 5.4 for the time being), you can always PM me for further discussion :)


Avatar
0
Kanakotka

2) That networking isn't on in singleplayer, is it? My own issues are in singleplayer, multiplayer matches generally don't last long enough for the issue to surface for me. UI i can understand, as a somewhat flimsily coded UI can produce mass garbage. If your bought frameworks are causing notable performance dips though, you should contact their developers.


3)

Unfortunately += and -= are notoriously garbage-filled because what they actually do behind the scenes, and i can totally understand the sweeping changes that would need to be done for. What i mean by "easy" is that when you know what to avoid, it's reasonably easy to just scroll through and replace certain things and that alone could be beneficial for performance. More invasive, "proper" optimization is always extremely hard and work heavy.


4)

It's essentially a memory leak being handled by GC automatically because the GC inclusively tracks what the code is doing at any given point in time. It would be the classical case of memory leak if it managed to get past Unity's GC, but i just prefer to use the term because as far as i know memory leaks as we knew them are things of the past outside of making your own engine or something.


5)

Having just 2 coders on a project of this scale is scary... and impressive. Hats off to you guys.


Unfortunately right now i'm full-time employed with a project for at least a month and i couldn't manage to concentrate well enough on an optimization pass. After that though? I can get back to you :)

Unfortunately the project i'm working at is 5.3x version and i don't have any experience of porting larger scale projects to newer versions (which unfortunately often isn't worth the trouble)

Avatar
0
Lee "Noontide" Moon
  • Under Review

Hi Darkassassin98,


Thanks for the praise for the game and for your support. I can say that we truly appreciate the frustration that comes from the GC Stuttering, for us as developers it's extremely irksome to watch the game struggle in later stages and we keep trying to look for ways to reduce it. We've made some changes internally which should improve the problem very slighty but as you may know the true solution lies with the planned new low latency & generational garbage collection algorithm that is planned by Unity.


Unfortunately we currently have no timeline for this feature and we've been waiting for some time on it, from our side there's sadly not a great deal we can do at the time being but that doesn't mean we'll stop looking for further optimisations. All I can say is that we do still plan for some sort of long term support of WFTO but I can't speak to the scope of support this might entail, it's certainly not something we want to leave unaddressed however and it's frustrating that a solution has thus far eluded us.

It is our hope that should the new GC be added in Unity that we'll do our best to bring the game up to that version simply to make use of it. However given this unknown time scale for the feature we can't say how likely it is to happen. Upgrading the engine is a time consuming task and can even be roadblocked by support issues with other middle-ware that exists in WFTO. We're looking at options to keep the game as up to date with the engine as possible and at the very least we'll be keeping our fans informed.


I'm sorry that I can't offer anything more concrete at this time but I hope that as we get closer to 2.0 we can at least have a more clear answer in sight.


I'll look to merge this with the lead topic on this matter.


Cheers,


Lee

Avatar
0
Andrew "Nutter" Jaggar
  • Pending Third Party

Hi ShadowMajestic,


Unfortunately the issue you are experiencing is a known issue and is due to Unity's own Garbage Collection algorithm (which is outdated and in all honesty kind of rubbish) rather than WFTO itself. Unity are "looking at it" and working on a new (and improved) algorithm which should fix/solve the issue but there is no known timelime for it to be finished/done.


In the mean time Brightrock themselves continually optimise the game as best as they can to squeeze every last drop of performance possible from the engine but ultimately this is an issue they can only put a bandage over. Unity themselves are the only ones who solve this issue.


The best thing I can suggest is playing the game on low settings, and when you start seeing the issue, save and reload the game (which clears all the garbage and will temporarily stop it happening for a while).


Sorry, because i'm sure that's not the answer you were looking for but I hope it helps a little and you can still enjoy the game whilst we wait for Unity to get their act together.

Avatar
1
shadowmajestic

Hey Andrew,


Thank you for your reply.

I do not experience similar issues in other Unity games though, like 7 Days to die(I had similar issues in the past though, very tiny stutters every 5-10 seconds or so, but they were hardly noticeable with the other stutters at the time), which at times, is a bit more demanding that WftO. But they are entirely different games, so it is hard to compare these of course. I do recall an issue with minor freezes a couple of patches ago(i think it was alpha 14), their PlayStation porting partners found a solution for it. Might be related, most likely is not. (I think Joel talked about it on his dev blog at one point). If it might be helpful, I can try and find this.


For now I'll just try and stick to the smaller maps with the lowest settings where the freezes are sort of tolerable(your own maps seem to do better than the community maps). And on other fronts, fantastic work on the optimizations. From 30FPS which dropped to 10FPS or less after <30minutes of gameplay, to a steady 60 FPS all around (If I ignore these garbage collecting freezes). Really enjoying the fact that my crappy GPU can easily draw all the pixels you can throw at it.


Of course I was looking for an solution, but the main reason to post it was to make sure you guys were aware of it. As I couldn't really find others with similar issues. But you guys are, which is awesome. Thanks for the explanation.

Avatar
0
anonymous
Quote from shadowmajestic

Hey Andrew,


Thank you for your reply.

I do not experience similar issues in other Unity games though, like 7 Days to die(I had similar issues in the past though, very tiny stutters every 5-10 seconds or so, but they were hardly noticeable with the other stutters at the time), which at times, is a bit more demanding that WftO. But they are entirely different games, so it is hard to compare these of course. I do recall an issue with minor freezes a couple of patches ago(i think it was alpha 14), their PlayStation porting partners found a solution for it. Might be related, most likely is not. (I think Joel talked about it on his dev blog at one point). If it might be helpful, I can try and find this.


For now I'll just try and stick to the smaller maps with the lowest settings where the freezes are sort of tolerable(your own maps seem to do better than the community maps). And on other fronts, fantastic work on the optimizations. From 30FPS which dropped to 10FPS or less after <30minutes of gameplay, to a steady 60 FPS all around (If I ignore these garbage collecting freezes). Really enjoying the fact that my crappy GPU can easily draw all the pixels you can throw at it.


Of course I was looking for an solution, but the main reason to post it was to make sure you guys were aware of it. As I couldn't really find others with similar issues. But you guys are, which is awesome. Thanks for the explanation.

I'm glad to hear that your overall FPS is higher. We are always trying to make the game more performant, hopefully you will see that as we march forward with WFTO development


As for the stutter we cannot do too much at this time to correct it as we wait for Unity to improve their Garbage Collection solution, as Nutter said

Avatar
0
Rohaq

I remember hearing that garbage collection was improved in newer versions of Unity; what version is WFTO built in? The game still ends up freezing for a couple of seconds, every ten seconds or so, after an hour or so into a game.