It looks like you're new here. If you want to get involved, click one of these buttons!
Hi, everyone! I've been playing with ORX for some time and I'm quite fond of the data-driven design and how the low-level code is arranged. The config system has many merits. It's very powerful and flexible. However, I found that robustness and scalability is less desirable. Actually I've stumbled on quite a few problems when trying to run the outdated examples on github. In these cases, The deprecated C API can be easily noticed when typing "make", but the config part is quite the opposite. For example, in some examples the "Position" entry is set to "top right" or something alike which is no longer supported, but the engine don't complain. Same problem can be experienced when an entry is mistyped or no longer supported. Let along the lack of transparence for the parsing process.
These are just a few aspects of the problem. ORX's config system is too free for non-trivial games where there could be thousands of sections weaving upon each other. If one mistyped just one word in a complicating config system, it would be a pain in the ass to find it out. More so if commands are abusively used in the config system, but I guess that would be another topic. I wonder if anyone feels the same about this, because I feel that even a tiny improvement can do me a lot of good given the current situation.
The engine and the config system are simple and effective and I think we would all like to keep it that way. So the task would be how to improve the robustness and scalability without introducing too much complicity. In my opinion, given that ORX is a data-driven engine, the responsibility of the config system is to correctly organize the data so that the engine can parse them as the user expected or throw errors if otherwise (while runtime variables and game logics should be kept in the code). To achieve this, we would need to set some rules for the data format.
For now, I've got some simple ideas but they might not be very complete at this moment.
First of all, enhancement of explicity.
1) Typed sections
ORX uses config section of free forms to create objects and structures. The engine does not know what a section represents until it is used to create structures. I think that maybe we can give a hint to the engine so that it can know what a section really is and perform checks. For example, we can write the title of a section like this:
[HeroObject:Object]
So that the engine would know that this section is a definition of an object. This way, the engine can throw a warning when a section name of incorrect type appears in where it shouldn't be. This done, the check of entries become possible. For when the type of a section is determined, the engine can query its type to get a list of internal entry names and restrictions on the values (these infomation should be registered on the init of the config module). This way the engine can throw a warning or an error when a neccessary entry is missing or a value is invalid.
2) Distinguish internal entries from external entries
Let's call the hardcoded entries as "internal entry" and the ones defined in the config files as "external entry". To make the data format even stricter, we can provide a way to explicitly specify an internal entry name for use in both assignment and reference. We might want this because sometimes missing an internal entry is not noticeable. To implement this, we can allow the user to choose to prefix an internal entry name with '~' when an internal entry is wanted, or to set a rule that the first letter must be uppercase for internal entry and lowercase for external entry. Actually I personally prefer the latter one for it doesn't need to define more symbols and can completely divide entries into two types. So it would be like this:
; Explicitly declaring HeroGraphic as a Graphic section
[HeroGraphic:Graphic]
; Assigning an internal entry (start with uppercase letter)
Texture = hero.png
; Inheriting a section makes it has both its original type and the type of the inherited section
; In this case maybe we can allow omitting the type of HeroGraphic1 and default it to parent section's type.
[HeroGraphic1:Graphic@HeroGraphic]
TextureSize = (64, 64)
TextureOrigin = (64, 0)
[HeroObject:Object]
Graphic = HeroGraphic1
; Defining & assigning an external entry (start with lowercase letter)
health = 100
This way, any typo or invalid assignment can be discovered at parsing phase (of course this does not apply when a value is a command string).
3) Redefinition of a section throws a warning
Currently, redefining a section defaults to extending the original one. I think it would be more helpful to have it the other way round. That is to say, the engine should throw a warning and wipe the old entries if a section is defined twice without a '+' prefix.
4) Handle collision flags and groups more safely
Typos in collision flags are hard to find out because collision flags are defined on a object section's instancing. Maybe we can have a special section to register all collision flags before using them. The same applies to group names as well. For now, one can mitigate these problems by using data inheritance. For collision flags there is one more thing to be addressed. Collision event might not be emitted if collision flags are not used in pairs. Collision pairs should also be pre-registered to ensure there's no broken pairs. It will look like this
[Collision]
Flags = hero # monster # wall # bullet
; Define collision pairs. All flags must be pre-defined in 'Flags' entry
; (hero <-> monster) and (hero <-> wall)
hero = monster # wall
; (bullet <-> monster) and (monster <-> wall)
monster = bullet # wall
; If there are flags not included in pairs, throw a warning
5) Settings sections are special
Forbid the definition of new (external) entries for settings sections. Forbid type hinting too. So this
[Console:Object]
and this
[Console]
ScrollSize = 1000
abc = 1
should result in errors. This way it doesn't change anything for using it but ensures nothing unexpected would happen when a special settings section is misused as a normal one.
Secondly, enhancement of section prototype design.
1) Section entries should be carefully designed so that coupling between the effects of different entries is diminished to an extreme extent. I propose this because I once encoutered confliction between different entries (although I can't recall exactly which section type and which entries).
2) If confliction can't be avoided, then the engine should check the potentially conflicting entries and their values and throw a warning if neccessary.
Thirdly, data scope.
Currently all sections are referenced by one linked list which allows no means to separate them. Maybe it would nice to have scopes in config systems just as in common programming languages. I'm not quite sure yet.
So, this is it. I would love to hear from the developers and other users. I haven't touch the code yet but would be happy to if this proposal sounds acceptable.
Comments
Hey @Tang,
First of all, thanks for your interest in orx and for taking the time to write such a detailed proposal.
That being said, let's tackle those points, one at a time point.
[Part 1]
Ah interesting, do you have links to those outdated examples? It'd be nice to update them if possible. Usually there's not much depreciation happening on the data-side itself, and when it happens all the cases should be documented in the release notes.
I'm not sure what you mean by the lack of transparency of the parsing process, would you care to elaborate on this one, please?
I cannot speak for other users, of course, but in my personal experience, I've almost never experienced any typing-related issues that would take more than a few seconds to fix. Typos are generally usually easy to avoid, there are dictionary of orx-recognized properties for some popular editors, such as Sublime Text, which can also help on this point. One can easily prepare their favorite editing software to help them with this as well.
I see it differently. The config module's responsibility is to be a data repository. That data can be fetched from .INI files, converted from other formats programmatically and is meant to evolve over the execution of the application.
And that's where its responsibility ends for me. It's also one of the base modules, and as such it should not know of specifics from higher level ones, such as graphic or object, for example.
I think your main concern is to have a safe net for developers, rather than a pure technical scalability issue. I have a handful of projects that use a few dozen thousands of config properties, those are loaded and accessible without any performance hit, so technical scalability doesn't seem to be the issue at hand.
Most of those properties are programmatically generated in my case though, preventing any duplicate effort from the developer or issues like typos.
I feel this has a couple of disadvantages:
it removes some flexibility of the system: it's frequent that a single section is used as a data source for many different types. You'd then need to be able to support:
it's also verbose, you'd then need something to protect/detect erroneous types and you'd simply bring the issue one level higher
[Part 2] (Sorry, the forum had a limit that prevented me to reply in a single post.)
To begin with, what are we trying to achieve by differentiating orx-recognized properties vs user-custom ones? I'm not sure I see the gain of doing such a divide.
Also, how would you handle properties and types that are introduced by extensions to orx, but not by the user themselves, like Scroll, Mod, Movie, etc. ?
Additionally, the current proposal is rather verbose, and as such more error prone.
I'd personally use the following to express what you wrote above:
And have as many inheritance layers that are logical, given the current situation, only overriding/extending what is necessary.
I disagree. Having sections defined partly in multiple files, or overridden in conditionally loaded files is important for the system's flexibility.
A simple example is when porting a game to a different platform.
Taking the example of Little Cells, when we made the Android version, we simply loaded
android.ini
on top of the existing config.This modified only the aspects we wanted different on Android, such as the aspect ratio, the tutorials, the menus, the cursor, etc.
It's also making it very easy to test the Android version on a regular PC by simply loading
android.ini
as the code is almost entirely identical for all versions (the only exception being the Steam vs Google Play integration).Another benefit is to provide additional debugging information, that's local only to the developer. I usually have a
dev.ini
file that only exist on my own machine, in which I redefine some properties for debugging purposes. Things like adding a name tag to display the object's animation, or additional viewports to display the shadow and light intermediate textures. All this without having to touch a single line of code and more often than not, without having to restart the game.In any case, orx will warn you for every single property you redefine, printing a detailed debug message for each line. That should already help the user to find any unwanted/mishandled cases.
There's also the
!
symbol if you want to entirely wipe a section instead of modifying/extending it:Currently orx will output a debug message for each new literal flag, so it should be trivial to find any typing error. That being said, you can fetch the entire list (or even preparing it) as it's stored in config:
One of the behaviors from your suggestion that isn't already present is preventing the extension of this list.
I could add a property to control it, or simply prevent any extension if the property was provided by the user and not created by orx itself. I'm OK with both options.
As for the pairs, I'm not sure I completely understand what you mean.
From my interpretation, you can already do what you suggest if you desire.
In any case, it's common practice to have variations on the SelfFlags & CheckMask for more complex situations and behaviors.
Maybe I missed something here and you could provide more details or a complete example?
[Part 3] (Last one!)
Well, that's an unfortunate example, as
Console
is a settings section that is meant to receive any freeform properties by the user as well, which is the main way to define data-driven aliases.I'd be curious to know which ones. Usually the names of all properties recognized by orx have been carefully chosen to only provide collisions when they made sense, from a logical point of view, like
Color
for both[Object]
and[Graphic]
. There are a handful of exceptions, but maybe we can highlight them in the documentation to prevent confusion.Again, that would require more context. Usually each module is responsible to handle and check the validity of the values they request from the config module. It's possible these checks might not cover all the cases, and we should seek to complete them.
Well, technically all the sections are stored in a hash table, not a linked list, but they do remain at the same level, you're correct.
This is by design, as I've always imagined users could add their own layer of scoping if they need it. They can use a separator for example, or any naming convention that would suit them. For example,
And write a simple wrapper function to handle the scope part. In my experience of using orx extensively for over 15 years, I've never felt I was missing scopes, but if people do, I think the current config system is flexible enough for them to roll their own scoping system, without needing to write much code and without forcing them in one way to do so.
Thank you again for your comments. I appreciate the ideas you're providing to correct issues you've faced in the past, but I wonder how much of this came from using a new system vs pitfalls that everyone will encounter at some point.
For example, orx already checks for case variations when fetching a property, so if you have:
Orx will print a debug message when
Texture
is being fetched by the Graphic module, saying it foundTeXture
instead and that maybe that wasn't on purpose.I'd like to add more safety nets in the future, while preventing extra verbosity or any runtime-cost of non-debug builds, as much as possible.
Here's a card listing some cases that were discussed in the past on Discord: https://trello.com/c/Kuy4acib/219-add-a-orxconfigverify-to-check-for-config-issues-inheritance-etc
Some more advanced tests, like using Levenshtein distance, for example, could be interesting to add for typo detection.
Lastly, the two template files could also be formalized and leveraged for some form of spell checking or even for the base of an editor. It's been discussed a couple of times in the past as well.
As for the typing of sections, itself, we could look at a real world example, Destroy 2 by @krousty_bat and @naroin, where they currently have ~17k, most hand-edited, config properties. It'd be nice to hear if the issues you mentioned are common ground for them and much of an hinderance to their progress or not.
I know they weren't the case on another real world example, Little Cells, made by @krousty_bat and myself, which had ~12k config properties.
I'd also like for other people in the community to chime in with their personal experience. However we might have more luck on Discord instead, as the forum isn't read by many anymore.
Thanks for your reply and concern! I see that I haven't present the problem clear. Sorry about that. In the "LargeWorld" example, a deprecated entry "OnCreate" is been used and eventually caused an assertion error in memory accessing. The effect of this deprecated entry travelled a long distance to be finally catched. This is the backtrace:
#0 _orxDebug_Break () at ../../../src/debug/orxDebug.c:452
#1 0x00007ffff76f618b in orxBank_GetAtIndex (_pstBank=0x7ffff3ec16b0, _u32Index=0) at ../../
../src/memory/orxBank.c:902
#2 0x00007ffff77345d8 in orxStructure_Get (_u64GUID=0) at ../../../src/object/orxStructure.c
:834
#3 0x00007ffff773bf06 in orxObject_CommandGetPosition (_u32ArgNumber=1, _astArgList=0x7fffff
ffd2b0, _pstResult=0x7fffffffd5d0) at ../../../src/object/orxObject.c:855
#4 0x00007ffff77ab6da in orxCommand_Run (_pstCommand=0x7ffff4062c08, _bCheckArgList=0, _u32A
rgNumber=1, _astArgList=0x7fffffffd2b0, _pstResult=0x7fffffffd5d0) at ../../../src/core/orxCo
mmand.c:257
#5 0x00007ffff77ace09 in orxCommand_Process (_zCommandLine=0x7ffff2c60250 "> Get Runtime Cam
era, > Object.GetPosition <, Object.SetText ^ \"[Camera] Position: <\"", _u64GUID=25769805327
, _pstResult=0x7fffffffd5d0, _bSilent=1) at ../../../src/core/orxCommand.c:1075
#6 0x00007ffff77ad1fd in orxCommand_EventHandler (_pstEvent=0x7fffffffd6d0) at ../../../src/
core/orxCommand.c:1177
#7 0x00007ffff77832bf in orxEvent_Send (_pstEvent=0x7fffffffd6d0) at ../../../src/core/orxEv
ent.c:574
#8 0x00007ffff7776023 in orxTimeLine_Update (_pstStructure=0x7fffe6510208, _pstCaller=0x7fff
e6530b48, _pstClockInfo=0x7ffff2ca0908) at ../../../src/object/orxTimeLine.c:626
#9 0x00007ffff77344af in orxStructure_Update (_pStructure=0x7fffe6510208, _pCaller=0x7fffe65
30b48, _pstClockInfo=0x7ffff2ca0908) at ../../../src/object/orxStructure.c:797
#10 0x00007ffff77439b9 in orxObject_UpdateInternal (_pstObject=0x7fffe6530b48, _pstClockInfo=
0x7ffff2ca0908) at ../../../src/object/orxObject.c:3848
#11 0x00007ffff7743e28 in orxObject_UpdateAll (_pstClockInfo=0x7ffff2ca0908, _pContext=0x0) a
t ../../../src/object/orxObject.c:3988
#12 0x00007ffff77bae00 in orxClock_Update () at ../../../src/core/orxClock.c:759
#13 0x0000555555558a48 in orx_Execute (_u32NbParams=1, _azParams=0x7fffffffda48, _pfnInit=0x5
55555559319 , _pfnRun=0x555555559429 , _pfnExit=0x555555559434 ) at /home/tk
w/workspace/orx-1.12/code/include/orx.h:300
#14 0x000055555555949c in main (argc=1, argv=0x7fffffffda48) at ../../../src/LargeWorld.c:257
From the developers' point of view this might directly reflect to something very close but I doubt it will be this obvious for common users. Of course one can rely on the power of debuggers to solve this sort of problems. But shouldn't using a deprecated entry be more easily to notice? The very reason to have typed section is so that entry checking would be possible. This benefits a lot in the sense of rigidty and correctness, quite obviously. So I guess we can agree on that. Then the problem is can this be done without making a big fuss? Let's get to this
Multiple types can be supported, it can look like this
[HeroObject:Object:Graphic]
But then we should throw warnings when two types have conflicting entry names. For example, if a section is both "Object" and "Spawner", many entries will conflict (Position, Rotation ...). The truth is, a section simply cannot be two arbitrary types but there's no explicity rules to prevent one from doing so. And it can become a real problem when manipulating section parenting in runtime.
Sections without type are simply treated as non-typed ones except for some builtin special section names. User-defined/extension-defined ones can be registered in the config module just like the builtin ones.
I don't feel we have to add type checking in every user modules. Type checking should be performed when a config file is loaded to memory. Checking them at using is unnecessary because it's meaningless to check the type of something which is basically volatile (the contents of a section can be changed from time to time, its type won't matter). Maybe it would be more resonable to consider this as "static checking". It's like type hinting in Python (I don't like Python myself, but type hinting can be helpful for detecting errors). This assumes all section types are already registered in the config module. The API might look like this:
orxSTATUS orxConfig_RegisterSection(const orxSTRING _zName, const orxCONFIG_SECTION_PROTOTYPE *_pstSectionPrototype, orxCONFIG_SECTION_CHECK_FUNCTION *_pfnChecker);
This way, the checking will only run once and discover some static errors and report them. This is all I want for this part and I've no intention of bringing this to a higher level than it already is. I don't want to make "a new data language" yet .
OnCreate
isn't a deprecated property, it's actually a new one.According to that trace, the command stored in
OnCreate
is correctly executed but it looks like the GUID stored forCamera
is incorrect.I just compiled the latest version of LargeWorld with the latest version of orx and was able to run it without any trouble. Which version of orx are you using with it?
This is not linked to any config deprecation at all, the property is correctly used, even in your case, something else is happening afterward.
Additionally, you shouldn't just take my word for it, but of common users, which are all already using that system. Dstroy 2, for example, isn't made by an orx's developer (I'm the only developer for orx).
That was one option I did mention, however you'd need to support a variable number of tags and have the burden of what can or cannot be used on the module that's retrieving the config content. You would also break anyone currently using
:
in their config name. Lastly you'd need to handle adding tags on the fly as well, adding an extra layer of complexity.That is true, but I feel this is more common sense than anything else. Objects and Spawners are two types of structures that use a frame internally, and thus use the same properties for their internal frame (Position, Rotation & Scale). They are the exception rather than the rule though. The last ones using frames would be Cameras. Beside those three, mixing is usually safe.
That cannot happen when a file is loaded in memory as types cannot be known by the config module itself at that time.
Config content is loaded before dependent modules like object, graphic and any other higher level module, are initialized.
Additionally, how would you also handle content that is not coming from a file and set property by property using the
orxConfig_Set*()
API?Similarly, extensions that interpret some config data, such as Movie or Mod, can be initialized at any arbitrary later point, much later down the line.
In any case, it should be rather straightforward to run your own validation if you so desire. You can easily implement a similar scheme on the user-side and check for what you feel is your common pitfalls.
I would however suggest to automate your content creation instead, this way you can enforce the consistency of your data offline, through an editor and/or dedicated build tools.
[Part2]
2) Distingushing internal and external entries.
This is merely for entry checking. If we can distinguish the internal ones, then we can check for invalid internal entry names and detect possible conflictions between multiple entries. For instance, if a deprecated entry "OnCreate" or mistyped entry "Poition" is specified for a section of "Object" type then we can throw a warning then immediately we know where went wrong without even running a debugger to trace things down. Otherwise, there's no way to make the engine think this is an error, it would just think them as user-defined entries and ignore them.
Since every thing can be registered in the config module before loading any .ini file, I don't think there sould be any difficulty for 3rd party sections to fit in this.
The example is just for showing how the config files would look like if 1) and 2) is applied.
3) Redefinition of a section throws a warning (defaults to)
What I'm saying is that the current default is not safe. Suppose I have 100 .ini file, and defined a section named "Bar" in one of these files. When I'm writing my 101th .ini file, I forgot that I already have a section named "Bar" and redefined one, and the contents happens to be very similar. So when I run my game, I'll notice something very odd but have no idea where it comes from. After sometime, I guessed there might be an unindented redefinition. Then I'll have to grep all my ini files to find out.
4) Handle collision flags and groups more safely
I am aware that ORX prints all collision flags and that can partially alleviate the issue. But it won't warn me when I have somthing like this
[Hero]
SelfFlags = hero
CheckMask = monster
[Monster]
SelfFlags = monster
CheckMask = not_hero_but_already_defined_flag
The problem with this one is that hero checks against monster, but monster does not check aginst hero. In such a case the collision event between Hero and Monster might not be omitted (When hero is checked with monster, everything's fine, but no collision event if monster is checked with hero).
There is really no deprecation happening. I can't think of a single example of deprecation from the top of my head. Even when not advertised anymore in the template files, config properties are still honored. For example,
TextureCorner
was "renamed" toTextureOrigin
, but the old name is still honored so as to not break any existing data.As mentioned in my previous post,
OnCreate
is a new property.But that's the thing, config data is loaded before other modules are registered and initialized. In the case of extensions, they can even be initialized much later, at the user's discretion, well after objects have been created, etc.
But that's the thing, orx will warn you if you are redefining content. If by some mischance you are not redefining any property, but only extending content, then it is the intended behavior.
It is your responsibility as the person defining content to know what you are doing. You can devise additional tools or process or I can even consider adding an additional warning when a section that previously existed is pushed again during loading time, but that would be the extent of it as extending content (and redefining it) is the intended behavior: this flexibility is paramount to the system as it was designed.
And that might be what you want as you might want to have a monster that does not collide with heroes, but still collide with other things that would recognize a monster, like walls or ground, for example. Or you might want to trigger those collisions only under specific circumstances at runtime. For example, a ghost that would only become solid if you're not looking at them, to cite a popular game franchise.
[Part 3]
5) Settings sections are special
Well, "Console" is certainly not a proper example. However, we certainly don't want to accidentally mistake a setting section as a regular one. Take "Clock" as an example. If I forget that "Clock" is a settings section and defined it as a regular one and happens to defined the "Frequency" to 0.02. Then I wouldn't even notice the change. So this is why I want to make settings sections special. But I'm not sure my current solution is good. Or maybe we should prefix these special sections with a particular symbol?
[Clock]
is both actually. It's the settings for a regular clock, the one of which you can create on the fly, it just happens to be the initial core clock as well.Similarly
[Input]
contains settings for a regular input set, it also happens to be the initial default set.A prefix could have been the way to go, but introducing it now would break all existing content.
Breaking data is something I've been avoiding as most as possible since the system was introduced in 2008, and I'd like to keep with this philosophy.
Well, I didn't notice it. I just checked frame 10 and found out it was the "Camera" object and checked frame 2 and found out the GUID is 0 and suspected something wrong with the "Camera" section, and I find no "OnCreate" in the template so I deemed it a deprecated entry without too much thinking. But reviewing this backtrace carefully, I can sense that something went wrong that passed the wrong GUID. I'll look into it later. By the way, I'm using the 1.12 version in Linux (compiled from source). This assertion failure occurred when I compiled and ran the game before any modification is made.
When you say the 1.12 version, is that from one of the packages? If so, they're almost 1 year old by now. I always recommend using the latest from
master
/default
, as those are considered stable branches: new development usually occur in dedicated feature branches.CreationTemplate.ini
onmaster
will contain the following properties under[Object]
:True indeed that I can implement this without modifying any ORX code. I'm just saying that maybe we should make this as default behaviors of the engine. Any way, I guess I'll just implement this as an external tool and see if things goes well for me. Graphical tools and extra syntax highlighting are just half-solutions, I think I'll stick on making a "config checker" myself.
Wow, that I haven't noticed. I'll go get the lastest source. Thanks!
Hey there
In all projects made on Orx, I am the graphist and help scripting the gameplay and all graphic presentation
I love the scripting part (even if as an artist, a GUI would be faster for me of course )
but i'm now used to it, and It's very clear for me on Sublime Text.
Of course, I have my habits when it comes to the way i organize my .ini
I like to order things when they are parents or children of another object
for instance
I just miss update on the Sublime.text new features and function introduces those last years
that's all
I guess I still haven't made myself clear yet. Well, here's the case. In the "The Beginner's Guide to Orx" example. If I specified "wall" in hero's checkflag but didn't specify "hero" in "wall"'s checkflag, then the walls won't collide with hero, the hero object will just go through.
Yeah, data inheritance is the main approach to mitigate the problems I presented. And I agree with you that sometimes a graphical editor would be nice to work with.