@Nadeo : Gathering informations about the big dedicated server issues : lags, crash

[Palteza]

We're lauching this brainstorming-topic to collect the more informations we can from the community : event organizers, servers admins, and everyone who has feedbacks.
In background, we DDE are working on a new 24h race edition and this is a nice occasion to try to collect, especially intended to Nadeo, all the parameters of these server issues.

It has been known that the servers have problems with memory charge, leading to crashes at some point, and when not crashing, performing very badly with huge latency and lags.
Here are the parameters we know : seems to happen in non-TA modes (see Shorty' speech below) /// depending on an amount of CPs crossed (see our tests below).

If you know something about that :
What was the configuration of your server when you met problems ? Number of divers ? TA/Rounds/Other ? Any other special parameter ?
What the controller (uaseco, pyplanet, maniacontroll) ? No controller ?
What were the problems, server crash, lag, other ?
Did you find any solutions to solve these problems ?
Any food for thought to explore ?
Add anything you want.

Of course we are hoping to share knowledge with you Nadeo about all this stuff.

Thanks everyone for you feedbacks !

[Palteza]

1st feedback : Shorty about a competition




2nd feedback : our internal tests

test made on a mini map : 17secs for 61CPs (small loop with multi CPs trick, TA mode)
lag startafter ~10minutes - after 20minutes the server get this error ( screenshot )
we reached more or less ~4000CPs before the crash
the server is runing on a virtual machine and use a dedicated before 4.1

[HaagseSmurf]

In the screenshots i see the use of a controller but i dont see it mentioned here.

Important is to be absolutely sure the issues are caused by the dedicated server and not by the controller. In all fairness you should do the same tests with uaseco, pyplanet and maniacontrol (the most used controllers atm) or do a test without a controller.
If the issues as stated here are happening with each controller and appearing at the the same moment its fair to say the problem is caused by the dedicated server software. Also save the serverlogfiles and eventual other reportmethods to give as much info as possible to Nadeo so they can narrow down hopefully the cause of the issues.

Good luck!

Greetz

[Miss]

use a dedicated before 4.1

Why? There was a recent memory leak fix in 4.1 dediserver, so updating is probably worth a shot.

I suggest you first update the dediserver, and then attach a debugger to it and try again. This way you can gather a lot more information (eg. a potential call stack or location in the code where the crash occurs.

You also failed to mention which OS you're running the server on. Is it Windows? Linux?

[Jonniboy]

The second screenshot is actually kind of misleading, and that it was shared is partly my fault to begin with. This test was from an old DS build, and it seems to be fixed now. The new tests which did not cause any problems were tested on a server of mine running on Ubuntu 18.04.1 Server with PyPlanet installed. We will do further tests about the checkpoint bug soon, and if we experience problems again we will post a more detailed feedback.

But please feel free to add your own experiences and tests that you may've made already. The topic was more meant as a place for everyone to gather information about this topic, because there is little to no overview about the "small" problems and the unawareness of the dedicated server's performance leading event hosters to not host big events anymore (Spam's Weekly Race, possibly 24H Race, some HuntMania events etc.).

[Miss]

There should be more details to your tests though. Things like - how much RAM is the server using? Do you have a graph of the RAM usage before the crash? etc.

Is anything printed to the log?

What happens when it crashes while you have gdb attached to the dedicated server? What's the crash address etc? Does it segfault, or something else?

[Jonniboy]

There should be more details to your tests though. Things like - how much RAM is the server using? Do you have a graph of the RAM usage before the crash? etc.

I did not do the tests with the old DS version, that's why I have no details about these. Because there were no problems with the new patch, it was not necessary to further investigate this issue. But I will do some more tests regarding that topic, though I don't think there will be much to see since it seems to be fixed.

[Miss]

Sounds good!

[jonthekiller]

Since the latest version of the dedicated server, memory is no more an issue, for example with 40 players I use 140Mb with ManiaControl, or 300Mb with a max to 100 players.

For me the issue you can have is the Laps gamemode, who still have some issues, but maybe the Rounds gamemode can be use instead (it's what I have done for the Trackmania Champions League qualifications).

In the next months I will be able to give more metrics and details with a huge number of players to see the limit we have if some are interested.

memory.PNG (92.87 KiB) Viewed 1260 times