An unorthodox package management scheme

edited November 2015 in General discussions
Hi Everyone,

A part of my brain has been trying to come up with a simple but powerful package management system for the Orx community for a while. A very unorthodox scheme has just occurred to me. I warn you, the idea will sound terrible at first, but give me a chance before you burn me at the stake :)

The idea is actually very simple:
  • We all base our game projects on a common git repository to serve as an empty template. I already have such a cmake-based template. The idea depends on us all starting from the same template, but the template might actually have the infrastructure for multiple build systems.
  • Each package is a branch that builds on that template repo! Yes, you've heard me right. We won't be using git submodules, we will be using branches.
  • Whenever you want to include a package in your game project, you merge that branch into your game.
  • Whenever you want to release a package you just create a new branch. If your package depends on another package, simply merge that branch into your branch!none
So, what do we gain by this?
  • We can share anything, not just C++ code and headers. We can share .ini files, custom tool configurations, weird build system tricks. Just an example, in my Android build, I've integrated SWIG into my build configuration in such a way that you completely forget you're bridging Java and C++. In this package management scheme, I could share it with the community, so that the users of my package wouldn't even know what SWIG is and wonder how their C++ methods are just callable from Java!
  • Package repository is outsourced. Github, or bitbucket does it for us.
  • Package repositories are decentralized. You can maintain your own set of packages trivially. You can also fetch packages from other people's repositories by just "git remote add"ing their repo
  • Diamond dependency problem disappears! If package B and C both depend on A, no problem, they already have it merged in, so when you try to merge them both into your project, only one copy of A appears.
  • Packages are naturally versioned, you can easily see on a commit graph what libraries depend on what.
  • It's trivial to update your packages, you just merge-in a newer version branch.none
What do you think?

Edit: The scheme has a fatal flaw: You can't remove any package dependency, please see my post below.

I'm still hopeful about a similar package management system though, please see the same post :)


  • jimjim
    edited November 2015
    Sounds right to me :) But I am not that experienced in git to find any hole in your proposal. My only concerns if we have hundreds of package wont that make base repo/template huge? Or If I understand correctly to use a package that you shared, I need to add your package repo as a remote to my repo, then pull and merge that branch in my local repo?
  • edited November 2015
    First of all, this proposal leaves the "hundreds of packages" case a bit out of scope. At this point, we're a small community, and our needs are simple. My proposal aims to solve our problems now. When the issue hopefully grows and becomes a problem, we'll have gained more insight into what kind of package management we ACTUALLY need. What things get shared, and what grows to become a scalability problem.

    Still though, the issue you've indicated is not an immediate weakness of the scheme. As you said, no repository has to include all the packages. I'd also like to clarify that, even if the repository contains ALL the packages, that still doesn't mean that it will contain many files. Only those packages that are merged into a project will become a part of its file tree.
  • edited November 2015
    OK, I've been thinking about the scheme and I've spotted a fatal flaw: You can't remove any package dependency...

    Imagine I have a game, and I depend on A and B. B also depends on A, but I'm not aware of this. At one point, someone releases a package C, that does the same thing as A, so I decide to switch to using C instead of A and somehow "unmerge" A. Now B is broken.

    The root of the problem is that git's (or hg's for that matter) commits work best when they're additive. I mean, when: ( A <merge> B ) >= both A and B, for some measure of comparison. So the commits form a partially ordered set. But when you're removing a package, you're essentially going back to a previous commit, and the partial ordering is broken...

    This flaw is not only in game project level either, the same thing happens if a package decides to remove a dependency too.

    So, I guess, using plain git (at least the way I proposed) is a no-go from the beginning.

    I'm still hopeful for a file-based package management system though. What I mean is, dropping the notion of a "packages" folder, but embracing a package system where a package can in essence change any file or folder in your project. Just like how package managers for Linux distros work. Of course that doesn't mean we should go bananas in the packages we author, most of them will just add a folder under a "libs" folder, but the scheme will allow very deep interaction of packages.
  • jimjim
    edited November 2015
    But one thing remains the same that we follow a similar code/project structure. Only then will know where would be lib, src or platform dependent files. Can we all agree to this one thing?
  • edited November 2015
    Standard project structure isn't at all an unusual thing, many build systems and frameworks come with their own standard structure. So I think it would be a good idea for Orx. It would also greatly simplify the task of package management.
  • jimjim
    edited November 2015
    Exactly :) unity, cocos2dx, even flixel maintains a similar project structure. Or we could read from ini/config script to get path for asset, src, lib, other folders. And copy files accordingly like current extern files using rebol script.
  • edited December 2015
    Why don't we do it exactly like a linux distro/package manager?

    Repos can be any directory that has the following:
    .orxrepo - empty plain text file signifying that the directory is an orx repository

    Zip files, but with the extension .orxpkg - The zip file contains the plugin. (What exactly should the internals be? How should they be laid out?) At the root of the zip file would be a text file named pkgdesc that contains the name of the package, the package creator, the version number, the supported platforms, build date/time, web site, contact info, a short description, a long description if necessary, and a list of the files in the package. All of which aside from the package name, version, and file list could be optional.

    Just an idea.
  • edited December 2015
    I appreciate you putting your mind to this, I hope that's how we're going to come up with a package manager that suits everyone's needs.

    The linux package managers have been a big inspiration indeed, but I had other concerns too. First of all, I tried to implement the package manager as a git workflow in order to inherit all the cross-platform git commodities, such as the client tools, visualizers, the existing hosts etc.

    Secondly, I imagine merge conflicts to be an expected and actually welcome aspect of this package management scheme. For instance, you decide to pull the (non-existent as yet) Scroll package into your project. The Scroll package will probably manipulate your main.cpp file in order to reorganize your project into a scroll-based one. If you've modified your main.cpp yourself, you'll get merge conflicts in the places where you've made incompatible changes. Here, the merge conflict is actually very welcome, as it actually acts as a tutorial as to how you should integrate Scroll!

    Third, the automatic dependency retrieval and implicit versioning are also nice things we inherit by using git.

    What do you think is the advantage of the scheme you propose? What problem does it solve, what tasks become simple with it?
  • edited December 2015
    Simplicity and versatility are the two main pros.

    Zip is a well-known, universal format (although now that I think about it, 7z is better, almost as well-known, and open-source).

    The specification can be easily changed and modified to fit our needs. The package manager itself could handle whatever we wanted it to, from dependency management to merging files and displaying readme/help/documentation where necessary. All of this could be based on a metadata format, and there could even be a visual tool to help create metadata.

    Implicit versioning could probably be shoehorned in using MD5 hash sums.

    Neither solution is perfect, unfortunately, but rolling our own gives us more freedom in the end, even if it's a huge pain in the beginning. Maybe we could even use an already existing package manager and modify it for our purposes.

    Lots of options, all of which need to be considered and weighed. Should we focus on the here and now, or work on a solution that could potentially serve us will in the future as well?
  • edited December 2015
    These are all valid points, but I think the initial pain is more important than that. We're a very small community, and any up-front obstacle might quench the enthusiasm. My main drive for the package management is my personal need for a way to share components across my own projects. The current scheme allows me to spend, say 50% more effort and be able to share those components with you as well, and that's some extra effort I'm willing to sacrifice. But if we go with something significantly more complicated and with much more upfront effort, I'll probably not be able to find the motivation, and I'll probably not be able to convince my wife that the effort is worth it :)

    Another important point is this: How do you imagine the .zip (or .7z) based approach will work? What will be in those archive files? I mean, how will the code be organized? Will we just extract the archives under some folder? If so, how will the components integrate with the user's game? I'm asking these, because this point is very VERY important. With the current git based approach, the files get scattered across the user's project folders. So we don't have to fight with existing toolchains and workflows. The files go to the exact folders where they need to be. We can even inject lines of code or configurations into exiting files. Whereas if we put all the package files under some folder, we'll need to write a lot of scripts to get those files to be compiled for all platforms and packaged with the game installations. Imagine the kinds of things one might want to share as a package:

    1. Code, C/C++ libraries (obviously)
    2. Game content (objects, fonts, etc.)
    3. Custom build steps (such as a prebuild step that captures the current repo revision into a config section, I'm actually planning to prepare this package)
    4. Code generation setups (such as SWIG)
    5. 6. 7... and probably a billion wacky things

    So, you see, scattering files across the users repo opens the door for very creative things to be shared as packages. Now, you could think that our archive based package management works by overlaying the archive contents over the user's repository. Then the biggest problem is, how do we make them get along well with the user's VCS? With the current git based approach, they're nicely combined into a single flattened commit, and it integrates beautifully with the user's VCS.
Sign In or Register to comment.