List sizes break > 2518 items

sausage

This is a follow on from https://forum.orx-project.org/discussion/4937#Comment_4937 where the list limit was raised from 255 to 65535.

Seems to the limit actually breaks at 2518 items in the list.

[Lists]
MyBigList = A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
	A #B #C #D #E #F #G #H #I #J #K #L #M #N #O #P #Q #R #S #T #
... etc to 2518 items and no more...

orxConfig_PushSection("Lists");
	
orxS32 count = orxConfig_GetListCounter("MyBigList");
	
orxLOG("Total in the list: %d", count);

If you add one more item to make it 2519 items in the list, the count will result in 0 and some of the list will spit out into the log.

I discovered this issue as I started increasing the size of my level map.

output:

[15:51:56] [CONFIG] [orxConfig.c:orxConfig_Load():2771] [game.ini]: Begins th
e processing of included file @../game.ini@.
[15:51:56] [CONFIG] [orxConfig.c:orxConfig_Load():2944] Key <#T #> has no value,
 assign character '=' not found.
[15:51:56] [CONFIG] [orxConfig.c:orxConfig_Load():2944] Key <A #B #C #D #E #F #G
 #H #I #J #K #L #M #N #O #P #Q #R > has no value, assign character '=' not found
.
[15:51:56] [CONFIG] [orxConfig.c:orxConfig_Load():2777] [game.ini]: Ends the
processing of included file @../game.ini@.
[15:51:56] [LOG] Total in the list: 0

iarwain

Well, actually you haven't reached the limit of items in a list but you have exceeded the size of the config processing buffer which is 8kb.

I can of course increase the buffer size, but that will only push the problem further.
Even if we could end up with large enough buffers, the problem will then become performances.

For example, every time you need to access an item at index with orxConfig_GetList*(), unless you reuse the same index as last call, we need to go through the whole list from the beginning till we find the matching index, and this is probably already pretty expensive with a list that contains 2500 items! (Config values are simply char arrays in memory, there's no hashing/indexing.)

I'd advise either breaking down your list in a more hierarchical structure or just use your own data structure at that point.

sausage

Ah yep... I'll break my list it into several lists, and my routine will load them once and concat them in code.

I'll update the map exporter to do the same.

iarwain

Nice!

I'll think about doing some indexing if it turns out to be a problem for some people, but that'll come with an extra load with dynamic memory allocations.