[B3.1][DedicatedServer] Memory leak

Moderator: NADEO

Bueddl
Posts: 165
Joined: 27 Jun 2010, 11:39

[B3.1][DedicatedServer] Memory leak

Post by Bueddl » 24 Mar 2013, 12:37

Hi there,

as suggested many time sin the "main thread" of the dedicated server release ManiaPlanet 2 Beta Server 2013-02-27b ( http://forum.maniaplanet.com/viewtopic. ... 4&start=80 ) I am opening now an report thread for the memory leak issue here.
I'll try to collect all the important information here in one post to keep it more clean and more easy to get debug data for the team.

Status:
Probably FIXED since Release 2013-04-11.
See here for more details: http://forum.maniaplanet.com/viewtopic. ... 61&t=18545

Thanks Xavier! :thumbsup:

2013-04-13: the memory handling is ways better now and the largest memory leak has been solved, but there is still a leaky code snippet. See: http://i.imgur.com/99GJz1h.jpg

Error:

Basically, this all is about the memory caused crash with the log message like that:

Code: Select all

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Platform:

So, there seems to be a memory leak under *nix systems,
as far as we know this is not concerning windows based systems.
Some of us are using CentOS (reported OS version: 6 x86_64) and some Debian/GNU Linux (reported OS version: 6.0.6 squeeze x86_64). In both cases the memory leak shows up.
  • Debian 6.0.6 squeeze, x86_64 -> 64-Bit
  • CentOS 6, x86_64 -> 64 Bit
  • Debian 5.0 Highmem patch -> 32-Bit
Not affected:
  • Windows XP, 32 Bit
  • maybe: Ubuntu Server Edition 12.04.2, x86_64 -> 64bit
Confirmed Software versions with the issue:
  • 2013-02-27
  • 2013-02-27b
Misbehavior:

We first reported this on Feb 28th 2013. The basic around this seems to be some missing delete instructions.

The memory increases when players are joining (well that works as expected^^),
but it does not decrease on player leave,
so I think it is keeping back all the player objects and never deleting one of it.

Same behavior about the maps.
As usual, on next map there is a allocation of memory, so it increases,
but there is no delete processed at the end of a map.

If there are only players leaving, say until zero players on the server, even though there are map changes (on an empty server), there won't be any new memory allocated, but also none freed.

The third abnormality we detected is, that a freshly started game server in a windows environment takes about 90-110MB of memory. Running on a *nix system, it takes 300-400MB from start up.

This problems seems not to occur with less than 10 players, but as my graphs are showing it would occur also then, as you can see the memory also increasing on low player counts, but it would take more time.

Testing for it:

If you want to test your system and log the memory behaviur, you can have a look here for linux systems:
http://forum.maniaplanet.com/viewtopic. ... 27#p154927

Additional information:

As analyzed the binary uses a 32 Bit architecture, so i386, as detected with "objdump", what is also the reason for the crash. The program gets out of address space what is in 32 Bit binaries 2^32 = 4096MB. Even though that's the reason for the crash, it is just the indicator that there is something terribly running wrong.

Code: Select all

ManiaPlanetServer
architecture: i386, flags 0x00000102:
EXEC_P, D_PAGED
start address 0x08048140
The compiling systems seems to be an Ubuntu (maybe the reason for the bug not occurring on Ubuntu has its reason here, even though it is debian-based, what itself is affected)

There are some images out showing the malfunctions in memory handling.
http://img15.hostingpics.net/pics/15809 ... 396348.jpg
http://tm.bueddl.de//upload/crash-2013-03-06_1900.jpg

A live memory and player-count tracking for some of our servers is available here:
http://tm.bueddl.de/zero-stats/Very_Short

______________________________________________

Please add only a post if I was missing something really important. Otherwise I don't think that NADEO will carefully read that post, so keep it clean please ;-)

Kind regards,
Bueddl
Last edited by Bueddl on 13 Apr 2013, 21:20, edited 12 times in total.
http://www.tmserver.org | Mania Exchange
═╦═╔╦╗² | MP4 Beta

User avatar
Rots
Translator
Translator
Posts: 1164
Joined: 01 Sep 2010, 14:39
Location: España, Barcelona

Re: [B3.1][DedicatedServer] Memory leak

Post by Rots » 25 Mar 2013, 14:56

I'll keep an eye on a machine we got running ubuntu just in case... atm we didn't get any error of this kind... it's true that ram usage doesn't decrease as fast as it grows when some players join but it does not go thaaaat high on graphs without stop :|
Image
Image
Image
TrackMania Original brought me here... Here I am since 2006

nocturne
Posts: 221
Joined: 23 Jun 2010, 21:31

Re: [B3.1][DedicatedServer] Memory leak

Post by nocturne » 25 Mar 2013, 15:18

Nothing really to add.. just noting that I suffer the same issue on CentOS 6 x86_64. Server continuously consumes more memory as players connect and maps change, to the point it crashes the entire webserver.

Bueddl
Posts: 165
Joined: 27 Jun 2010, 11:39

Re: [B3.1][DedicatedServer] Memory leak

Post by Bueddl » 25 Mar 2013, 18:35

Rots wrote:I'll keep an eye on a machine we got running ubuntu just in case... atm we didn't get any error of this kind... it's true that ram usage doesn't decrease as fast as it grows when some players join but it does not go thaaaat high on graphs without stop :|
32 or 64 Bit and what version of Ubuntu? Please provide us with some more detailed information about your environment.
http://www.tmserver.org | Mania Exchange
═╦═╔╦╗² | MP4 Beta

User avatar
Rots
Translator
Translator
Posts: 1164
Joined: 01 Sep 2010, 14:39
Location: España, Barcelona

Re: [B3.1][DedicatedServer] Memory leak

Post by Rots » 25 Mar 2013, 19:31

Bueddl wrote: 32 or 64 Bit and what version of Ubuntu? Please provide us with some more detailed information about your environment.
Ubuntu server 12.04.2 lts 64bit
CPU intel Xeon 2ghz (x2)
2gb ram

It's true that from time to time ping goes over 2000ms :mrgreen: but we imagine its a net problem... As i said i can't apreciate ram leak...
Image
Image
Image
TrackMania Original brought me here... Here I am since 2006

Bueddl
Posts: 165
Joined: 27 Jun 2010, 11:39

Re: [B3.1][DedicatedServer] Memory leak

Post by Bueddl » 25 Mar 2013, 23:56

Hmm, so we all are working on 64 Bit Linux. No real common attribute as we have the fallowing. Some are Linux derivated, some are Windows. On some we have it and on some we don't have it^^

no mem leak:
Ubuntu , with 64 Bit, which is a derived Debian which is GNU/Linux derived.
Windows XP ( what architecture??)

mem leak:
Debian, also 64 Bit , which is a derived GNU/Linux and
CentOS 64 Bit, which is a derived RedHat what itself is derived from GNU/Linux.

Is everyone of us using the current build?

@Rots: How many players do you have on your servers at a time? How intense is the fluctuation? Are there large freeing parts to see , so is the server really giving large memory amounts away or is it just never allocating it?
http://www.tmserver.org | Mania Exchange
═╦═╔╦╗² | MP4 Beta

User avatar
fng_thatoneguy
Posts: 713
Joined: 05 Jan 2011, 18:06
Location: That One Place
Contact:

Re: [B3.1][DedicatedServer] Memory leak

Post by fng_thatoneguy » 26 Mar 2013, 00:20

My Windows XP is 32-bit.

EDIT: I'm also using the latest build. 2013-02-27b, I believe.
Image

User avatar
Rots
Translator
Translator
Posts: 1164
Joined: 01 Sep 2010, 14:39
Location: España, Barcelona

Re: [B3.1][DedicatedServer] Memory leak

Post by Rots » 26 Mar 2013, 12:18

Bueddl wrote: @Rots: How many players do you have on your servers at a time? How intense is the fluctuation? Are there large freeing parts to see , so is the server really giving large memory amounts away or is it just never allocating it?
The server handles in the evenings an average of 25 players ( in 1 or 2 servers with xaseco and many plugins) + ts3 server + website ... there's free swap space and we don't reach 2gb ram so not a busy server

From time to time i'm checking ram usage and it is released... Your graphs are insane :O it keeps always growing even if 0 players
Image
Image
Image
TrackMania Original brought me here... Here I am since 2006

Pesky
Posts: 39
Joined: 22 Jun 2010, 12:11

Re: [B3.1][DedicatedServer] Memory leak

Post by Pesky » 26 Mar 2013, 13:46

Just for information

Debian 5.0 32-Bit Highmem

We have the Problem as well.

nocturne
Posts: 221
Joined: 23 Jun 2010, 21:31

Re: [B3.1][DedicatedServer] Memory leak

Post by nocturne » 28 Mar 2013, 10:47

Just for sake of posterity.. back when we had this same problem in TMCanyon before ultimately abandoning it, we were running CentOS v5 64b, but after experiencing this issue tried CentOS v6 32b, Gentoo 32b, Ubuntu (whatever the current version was.. 10..?) 64b, before ultimately settling on CentOS v6 64b just for simplicity of configuration and interest of familiarity. No matter the distrib, no matter the arch.. same problem every time.

If I'm not mistaken, aren't all of the aforementioned (besides Gentoo) Debian based to begin with..? Also if I'm not mistaken, judging by the various botched start-up scripts Nadeo has released in the past for the ded server and ManiaLive -- I'd say they are coding for/on Debian themselves.

I'd really like to know how/if Rots is not suffering from this.. Anybody on here with decent enough *nix skills to give him a command to output the mem usage of a pid into a log file..? Never keep in practice long enough to remember syntax, hehe.

Locked

Return to “Maniaplanet Reports”

Who is online

Users browsing this forum: No registered users and 0 guests