[1.0.3.2] Very slow campaign save loading

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#1
I'm talking around 40 seconds (I timed it) for a fresh load from main menu (output log attached) or re-loading from within campaign (normally closer to 10s).

Nothing special about the save file - previously free of glitches, just my YouTube series save with quadcopter loaded in. Save file attached, too. Although, this happens in other campaign saves but not creative saves.

CPU's not getting battered or anything obvious. Loading animation is very faltering, and frame rate is very low the whole time.

This was not happening in 1.0.3 (thankfully).



Also, not sure if this is worth a second thread: since at least 1.0.3 (could be since way before this) I've been noticing the auto-save micro-hangs seem worst, too. I've been editing some of them out of my video footage and I see from my time-line that they are pretty much a whole second long (with no frame update), like every minute or thereabouts. Pretty intrusive. Was something changed here a little while back? Any plans to optimise it better?
 

Attachments

Last edited:
Likes: Aardvark123

A Friendly Tech

Well-Known Member
Joined
Sep 21, 2018
Messages
106
Likes
312
Points
170
Age
51
#2
Have you tried "set affinity" on the process and un-checking CPU 0? I get lots of stuttering if I don't do that (disclaimer: I always have Waterfox running with thousands of tabs open, so CPU 0 gets hammered on my system). When I first started playing Terra Tech, I had to do that all of the time. After one of the patches I didn't have to do it anymore, but recently I have to again.
 

Matt

PAYLOAD STUDIOS
Joined
Jul 23, 2018
Messages
442
Likes
1,324
Points
305
#3
This is something we were anticipating so we have an official response prepared. I'll post it here as it seems to be the most fitting:

We've recently gone to a newer version of the Unity Engine which is great for things like improved physics performance and engine bug fixes.

Unfortunately the new version also created an incompatibility with the way snapshot data was saved. This was written way back in the early days of TT, and unfortunately the language tools have changed beyond being compatible.

This meant we couldn't load any snapshots created in previous versions of TT!

We've created a tool to convert from the old format to a new, more future proof format, and loading old snapshots will automatically be converted by this tool. The downside is that it's a bit slower to load, and being a separate executable it might be blocked by your Antivirus or OS. So if you have lots of snapshots saved on your PC you will notice longer load times on startup.

As for the slow load times for Campaign specifically, this is because the game 'loads' in around 20 invaders. Each one takes 1 second to convert so it essentially adds another 20 seconds to the load time. @Django is looking into a permanent fix for this ready for the Stable Release.


I hope that helps clear some things up.
 
Likes: Aardvark123

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#4
So if you have lots of snapshots saved on your PC you will notice longer load times on startup.
Yeah, I read that in the patch notes, but I didn't see it happen. Maybe I was tabbed out during the loading process, because clearing out the game's "Cache" folder caused me to see the exact same type of slowdown on game load as I described above.

As for the slow load times for Campaign specifically, this is because the game 'loads' in around 20 invaders.
Ahhh, yes, it's the snapshots from Twitter, specifically. I get the same horrible 40 seconds of 2fps if I open the Twitter tab of the tech loader in R&D (or Creative). It makes the already existent frame dropping from the Steam tab look mild (relevant that I poked @Django on that just yesterday then, or, particular irritating, assuming the poor chap's already working on all this, what fun!).

Anyway, so the workaround I found for the half minute campaign save loading times is to simply disconnect Twitter.

I did this by removing the current "UserData.sav" file from "C:\Users\[Windows_User]\Documents\My Games\TerraTech\[Big_Number]\Saves". It's not ideal, because that looses a couple of settings, but I don't know of a better way. Fine for me. A big improvement in game useability (~5 second campaign loading times now, heh). You could suggest this to anyone that's particularly miffed with this problem.
 
Last edited:

Django

PAYLOAD STUDIOS
Joined
Apr 21, 2016
Messages
269
Likes
495
Points
475
Location
London, UK
#5
Anyway, so the workaround I found for the half minute campaign save loading times is to simply disconnect Twitter.
Good point! I believe you should also be able to achieve this by going into your Twitter account settings and revoking access, though I'm not 100% sure if that also applies to grabbing tweets as opposed to posting.

Maybe I was tabbed out during the loading process, because clearing out the game's "Cache" folder caused me to see the exact same type of slowdown on game load as I described above.
I'll spare you full details, but the files in the cache folder are in a compatible format. So actually as all your (old) snapshots are pre-cached you won't notice any difference in speed. Once you remove the cache it will need to load from the old snapshots, which hits the converter.

The problem with Twitter (and Steam) is we have no way forcing an update of the online files. Every time the files are downloaded anew (and we probably do this slightly more often than strictly necessary to ensure we have latest up to date snapshots) they need converting.. which is now a lot more costly.
 

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#6
The problem with Twitter (and Steam)
Just to be (overly) clear. The steam snapshots don't seem to be hitting this new converter, in that the slowdown I get from opening that tab is not at all as bad as from the Twitter tab, exactly as bad as it already was before your new converter. (I don't really understand how this is so, given what you've said, unless you're using the convert in advance of actually needing it, I guess..?)
 

Django

PAYLOAD STUDIOS
Joined
Apr 21, 2016
Messages
269
Likes
495
Points
475
Location
London, UK
#7
The steam snapshots don't seem to be hitting this new converter
I stand corrected - I thought we uploaded the encoded png's to Steam but turns out we upload the equivalent of the cache folder data. That's actually great as that's the higher traffic area.

Twitter is still slow, but I'm looking at pushing most of that slowdown off of the main thread. It will still take longer to load the snapshots, but it won't cause the game to slow down as it does currently.
 
Likes: ZeroGravitas

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#8
I thought we uploaded the encoded png's to Steam but turns out we upload the equivalent of the cache folder data.
Interesting. It has been kind of a shame that non-steam users can't manually take the png image from the items's workshop page. But it looks like that decision worked out for the best then.

I'm looking at pushing most of that slowdown off of the main thread.
Can you do something similar with the Steam tab thumbnail loading process too..? (Aside from other optimisations to avoid unnecessary refreshes.)
 

Django

PAYLOAD STUDIOS
Joined
Apr 21, 2016
Messages
269
Likes
495
Points
475
Location
London, UK
#9
Interesting. It has been kind of a shame that non-steam users can't manually take the png image from the items's workshop page. But it looks like that decision worked out for the best then.
We have no control over how Steam presents workshop items so that would have never been an option really, but I get what you're saying about it being nice if you could grab the png's straight from the workshop browser if you wanted to.
Convenience on one hand, but also sidestepping community involvement.

Can you do something similar with the Steam tab thumbnail loading process too..?
This is on the plans, but not the highest of priorities I'm afraid.
 
Likes: ZeroGravitas

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#10
Twitter is still slow, but I'm looking at pushing most of that slowdown off of the main thread. It will still take longer to load the snapshots, but it won't cause the game to slow down as it does currently.
I've just reconnected twitter to test this in 1.0.3.7, before you guys push the next stable.

I'd already seen improvements in loading time and less FPS loss (though still bad) for the Steam thumbnails. Twitter too, it seem. But this *only* aapplies when in Creative and R&D.

In campaign mode my (fairly powerful desktop) virtually hangs for 20 seconds when opening either of these tabs (I've cut down my workshop subs to ~100):


Its taking ~3-4 times longer per thumbnail in campaign vs creative, and the FPS loss is far worst. Does this mean that you didn't change the threading in campaign (yet)?

There's no hang on loading campaign saves though.:) Would I be correct in guessing that it no longer checks twitter at all for invaders?

It also clicked that the ~1s (~830ms) full-on hangs I get repeatedly in campaign could be the game decoding regular enemies, too? Or will that just be the auto-save function? How frequent is that?


I think this issue with opening the workshop (and Twitter) tab in campaign is going to cause a fair bit of consternation (and reports). I think many would argue that it makes the workshop virtually unusable in campaign. :(

Presuming there won't magically be a fix for this in the stable, I'd strongly suggest mentioning it properly in the patch notes @saiwun. The current note certainly doesn't cover it:
If you have a large number of snapshots (e.g. 300), the initial load of this update may take slightly longer than usual.
  • All subsequent load times should be quicker.
 
Likes: Aardvark123

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#11
There's no hang on loading campaign saves though.:) Would I be correct in guessing that it no longer checks twitter at all for invaders?
Seems like this is the case, given that Lathrix got the exact same invader at least 3 times in a row, here in today's episode.
 
Last edited:
Likes: Aardvark123

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#12
Twitter is still slow, but I'm looking at pushing most of that slowdown off of the main thread. It will still take longer to load the snapshots, but it won't cause the game to slow down as it does currently.
Just realised that the Twitter thumbnails load a lot faster again, now. In campaign, too. Don't know since when, but in 1.1.0.5 for sure.

So, has this snapshot processing bottleneck been worked on and somehow universally resolved?

Are invaders direct from Twitter now definitely working?

Are the ~1 second hangs during campaign play ever related to looking up invaders from twitter? Or just auto-saving the world or current tech?
 

Django

PAYLOAD STUDIOS
Joined
Apr 21, 2016
Messages
269
Likes
495
Points
475
Location
London, UK
#13
So, has this snapshot processing bottleneck been worked on and somehow universally resolved?

Are invaders direct from Twitter now definitely working?

Are the ~1 second hangs during campaign play ever related to looking up invaders from twitter? Or just auto-saving the world or current tech?
It's possbile some of the snapshots being loaded are now in the new format, thus not requiring slow conversion.

I can't sat what is causing the infrequent hangs without getting profiling data. It could be saving (every minute?), maybe snapshot decoding, could also be garbage collection due to general GC buildup.
 
Likes: ZeroGravitas

ZeroGravitas

Breaker of Games
Joined
Jun 29, 2017
Messages
3,711
Likes
5,057
Points
675
Age
36
Location
UK
Website
www.youtube.com
#14
Oh, thanks for replying!:D
It's possbile some of the snapshots being loaded are now in the new format
Ooooh.... Right. I see.:) So all the tweeted snapshots since the code changes (1.1 stable) are in the new format... So probably the Twitter pipeline is pretty much clear of old format snapshots by now... Is there a time cut off with how far back the loader looks, or just up to a certain number?

It could be saving (every minute?)
I think that must be the main thing. I'd have to comb over footage more deliberately and see if it is exactly(?) a minute and if there are other more minor hangs too.

Assuming it is saving, do you think we might ask for (and potentially get) an option to disable auto-saving? These little hangs are pretty persistently irritating. No chance of substantial optimisations to magic them away?