Measuring the persistent storage limit

You can talk about ManiaScript for ManiaPlanet here

Moderator: English Moderator

Post Reply
User avatar
BigBang1112
Posts: 377
Joined: 20 Jul 2015, 12:44
Location: Český Krumlov, Czech Republic
Contact:

Measuring the persistent storage limit

Post by BigBang1112 » 18 Feb 2020, 14:10

I was always curious about how much we can store with persistent variables, but we didn't really get to know the exact limit. So I tried to involve in it myself and measure everything as in depth as possible.

For the text measurements, I always used the character "0". Persistent storage was always cleared as much as possible before every measurement.

Maximal persistent Text length is most likely 2^16 (65 536) characters. My measurements were moving from 65040 to 65287, so the realistic maximum length is slightly above 65 thousand.
Measurement of 65287 caused a complete crash of persistent data reading, resulting in script crashes just when declaring the persistent - only deleting the profile helped there.
I'm not able to tell if these values can be understood as bytes. I tried putting just ASCII characters and also UTF8 characters and the results were always above 65 thousand, even tho the UTF8 character combination can take more than 1 byte for each character. But I don't claim I'm a text encoding expert either. But if yes, we can say the persistent storage limit is 2^16 bytes (aka 64kB).

Maximum count of elements in arrays of empty Text, Integer and Real seem to be most likely 2^14 (16 384), realistically I got into 16221-16239 range. An array of Vec2 is probably 2^13 (8 192) and the weirdest: an array of Boolean is 2^12 (4 096) for a rather unknown reason. Booleans can be expressed in one byte but in Maniaplanet, they take up more space than a Vec3. :D The rest of my results don't seem to match any computer-related number.

The measurement list ordered from largest to smallest count:
  • Text max length: 65040-65287
  • Array of empty Text max length: 16239
  • Array of Integer max length: 16230
  • Array of Real max length: 16221
  • Array of 1 char Text max length: 13008
  • Array of Vec2 max length: 8106
  • Array of Vec3 max length: 5398
  • Array of Int3 max length: 5396
  • Array of 10 char Text max length: 4645
  • Array of Boolean max length: 3940 (can anyone explain why tho? xd)
  • Array of 50 char Text max length: 1204
  • Array of 10 char Text with 10 char Text key max length: 877
  • Array of 1000 char Text max length: 64
The very similar but not the same numbers of some measurements may happen because of the variable name additional small storage taken throughout the test.

After I measured all of these, I ran the Text length measure again, and this time I got a value 64754, 533 less than the very first measure which was 65287. So the persistent variable name surely also takes up space in the persistent storage.

So we can probably tell the list of values in the cleanest way like this:
  • Text max length: 2^16 (65 536)
  • Array of empty Text max length: 2^14 (16 384)
  • Array of Integer max length: 2^14 (16 384)
  • Array of Real max length: 2^14 (16 384)
  • Array of Vec2 max length: 2^13 (8 192)
  • Array of Boolean max length: 2^12 (4 096)
  • Array of 1 char Text max length: ?
  • Array of 10 char Text max length: ?
  • Array of 50 char Text max length: ?
  • Array of 10 char Text with 10 char Text key max length: ?
  • Array of 1000 char Text max length: ?
  • Array of Vec3 max length: ?
  • Array of Int3 max length: ?
:!: I didn't measure structs because these shouldn't be ever used as a persistent in the first place. For structs, it's always better to convert them to JSON and store them as Text so you won't come across unfixable (or very hard to fix) conflicts.
I may try this later though just in case to see how structs are managed in bytes.

By this data information, we are able to calculate the storage size of each type.

Persistent data sums up in the storage, so if I would have two persistent Text variables with 32 thousand characters, I'm just about to reach the persistent limit. Therefore I highly don't recommend going close to the actual limits I measured with your variables, as the amount of persistent data left differs from each user.

Let me know if you're interested in measuring the limits of metadata (if there are any). :D
Last edited by BigBang1112 on 18 Feb 2020, 16:27, edited 3 times in total.
Creator and competent racer. YouTube Discord
- Nadeo Envimix (remake coming 2020)
- Videomaker (coming 2020)
- Challenge (OUT NOW)
- Leaderboards (OUT NOW)
- Compute Shadows

reaby
Posts: 1022
Joined: 29 Dec 2010, 23:26
Location: Eastern Finland
Contact:

Re: Measuring the persistent storage limit

Post by reaby » 18 Feb 2020, 15:43

:thumbsup: Very good, very good.

User avatar
Nerpson
Translator
Translator
Posts: 1549
Joined: 27 Sep 2013, 18:02
Location: France
Contact:

Re: Measuring the persistent storage limit

Post by Nerpson » 18 Feb 2020, 18:03

Interesting to know, thanks for the study.
ImageImageImageImage

User avatar
Miss
Posts: 2082
Joined: 05 Jan 2016, 11:34
Location: The Netherlands
Contact:

Re: Measuring the persistent storage limit

Post by Miss » 18 Feb 2020, 19:35

BigBang1112 wrote:
18 Feb 2020, 14:10
Measurement of 65287 caused a complete crash of persistent data reading, resulting in script crashes just when declaring the persistent - only deleting the profile helped there.
That's really bad, Nadeo should fix this if that's indeed possible. A single manialink could destroy a player's entire profile? That's bad.
1070 GTX, Ryzen 3700X, 16GB RAM, Windows 10
Forum moderator, opinions are my own. :thx:
Check out Image openplanet, the alternative ManiaPlanet & Turbo scripting platform! (Openplanet subforum)
I also stream on Twitch and tweet on Twitter.

User avatar
BigBang1112
Posts: 377
Joined: 20 Jul 2015, 12:44
Location: Český Krumlov, Czech Republic
Contact:

Re: Measuring the persistent storage limit

Post by BigBang1112 » 18 Feb 2020, 19:46

Miss wrote:
18 Feb 2020, 19:35
BigBang1112 wrote:
18 Feb 2020, 14:10
Measurement of 65287 caused a complete crash of persistent data reading, resulting in script crashes just when declaring the persistent - only deleting the profile helped there.
That's really bad, Nadeo should fix this if that's indeed possible. A single manialink could destroy a player's entire profile? That's bad.
I didn't manage to reproduce it after then though, not sure what has really happened, but it probably managed to save the data over the 64kB limit.
Creator and competent racer. YouTube Discord
- Nadeo Envimix (remake coming 2020)
- Videomaker (coming 2020)
- Challenge (OUT NOW)
- Leaderboards (OUT NOW)
- Compute Shadows

Post Reply

Return to “ManiaScript”

Who is online

Users browsing this forum: No registered users and 1 guest