Help request: mercurial, bitbucket and buildbot

edited September 2012 in Help request
Hi all!

I finally finished migrating orx from svn/sourceforge.net to hg/bitbucket.org.

I will only be working on the new mercurial repository as of now. The repository is at https://bitbucket.org/orx/orx, or, if you'd rather have an URL that's maybe easier to remember, you can use http://code.orx-project.org/orx

Scroll has been hosted on the same account for the past few weeks and I believe this will make external proposition/patches easier to integrate.

I suggest only cloning the default branch (aka trunk) (hg clone -b default ...) as all the historical branches might add quite some traffic for nothing otherwise. :)

Now, I'll need to update the whole site/forum/wiki to point to the new place and that's where I'd love to have some help from the community. :)

I've changed the link on the main page to go to bitbucket. I haven't changed the download link yet but that will come in the future.

Actually the future in this case means whenever I get a continuous integration ready for orx, probably using buildbot. I'll setup the master on my own private server at home that should be available 24/7, but I'll need some slaves, even if they're only available from time to time.

I will be providing slaves for doxygen, msvs2008, codelite, mac os (x86/x64) x and iOS builds myself (with quite some downtime), but I'll need some slaves for msvs2010, linux x86/x64 and Android.

The more slaves, the better, this way we should have access to build regularly, including API doxygen doc.

No more 1.5 years between releases anymore! ;)

Now that sounds pretty neat on paper but I have almost no clue on how to setup a buildbot nor how to compile most of the versions from the command line. If anybody wants to help with that, please let me know!

Thanks in advance! :)

Cheers!

Comments

  • edited September 2012
    nice move

    about android, currently orx is compiled when the application is compiled.

    I probably can change that to build orx as a static lib.
    and provide a slave to build the thing (if bandwidth requirement aren't too high)

    about cloning only the default branch, unfortunately, that's not how mercurials is working.

    when you clone a repositoy, you can't clone a part of it, you allways clone the whole repository including all branches.

    if you want to split branches from the main repository, you need to create forks in differents repositories.

    Forks works like branches (more or less)
    If repo A is the main, and repo B is a 'forked-branch', you can pull changes from A into B, and chanes from B into A (when work on B is finish)
  • edited October 2012
    I have no experience with buildbot, but:

    I think codelite generates makefiles for it's own use, and I've used those makefiles successfully from the commandline.

    I can provide a Linux 32 and a Linux 64 (Both Ubuntu 12.04 LTS) as slaves.
  • edited October 2012
    Yes, unfortunately it generates a different set of makefiles per configuration, which is pretty annoying. :)

    There's another post in this forum about supporting PreMake to generate all the project/make files, by AlgoJerVIA.

    Unfortunately 7 months later I haven't been able to look at his work as I've been crunching for 6 months straight at work and had a lot going on in my personal life too. I guess now it's time to get back on it, as a prerequisite of the whole buildbot integration. :)

    I just can't get motivated, I hate writing this kind of scripts. :(
  • edited October 2012
    I had a look at AlgoJerVIA's premake file, and started messing around with it. I must say, it's not a bad system, and it's kind of fun having a programming language at your disposal.

    Anyway, playing around with it, I seem to have gotten it working well on Linux (both with make and codelite) for all applicable build types. The exception being that the plugins themselves are not currently compiled, but plugin builds still compile for the library itself.

    I think I found a way to get embedded and static builds with Xcode, but it requires testing:
    I implemented (hopefully) a command line argument
    --build-type which can be set to dynamic or static for Xcode builds. eg: --build-type=static

    Of course, I have no clue what compiler/linker options are needed for Xcode, so the resultant solution wouldn't be compileable anyways in the current state.

    If someone wants to tell me a few things, I will hack at it further.
    Specifically:
    -What plugins should be built for what platforms/architectures?
    -What compile/linker options/flags etc... are needed for non linux platforms (win-msvc, win-mingw, osx)
    -As above, what libraries do I need to link against?
    -Are the iOS and Android builds even pre-compilable (Lydesik mentions that Android is always built with the app) If so, what settings are needed? https://forum.orx-project.org/uploads/legacy/fbfiles/files/premake4.txt
  • edited October 2012
    I'll change the way orx is build on android soon...

    but dont't worry about it since it's all custom Makefile fragments for the android ndk-build script, I'pretty sure it wont fit with premake.

    I'll integrate it with buildbot and provide the slaves.
    (I still need to investigate on this and how to package this stuff)
  • edited October 2012
    Thanks for your help, 4babce, it's much appreciated!

    For now, I think we can focus on the embedded versions only. That will also motivate me to rewrite all the tutorials that are not using that version.

    I'll check the different compiler options, hopefully tonight, and will post them here, same for the linker/external libraries.

    The iOS build is precompilable, and it actually requires an extra step after compilation to merge all the versions (iPhone, iPad, arm6, arm7, ...) in one universal binary file, using the lipo tool.

    I won't be able to check your work myself before this week end at best, unfortunately, but I'll provide you with all the info in the meantime.

    Thanks again for your help!
  • edited October 2012
    Sorry, fell asleep before reaching the computer last night. Will try harder tonight...
  • edited October 2012
    Ok, here are some info I gathered. Let me know if you have any questions or need any other info.
    I haven't had the opportunity to look into XCode (OS X/iOS) unfortunately.
    VS
    ==
    
     - orxLIB (dynamic):
      . include directories: ........include;..........externdlmalloc;..........externSDL-1.2.14include;..........externglfw-2.7include;..........externBox2D_2.1.3include;..........externSOILinclude;..........externOpenAL-softinclude;..........externlibsndfile-1.0.22include;..........externstb_vorbis
      . link directories: ..........externSDL-1.2.14libvc20XX;..........externglfw-2.7libvc20XX;..........externBox2D_2.1.3libmsvs20XX;..........externSOILlibmsvs20XX;..........externOpenAL-softlibvc20XX;..........externlibsndfile-1.0.22libvc20XX
      . preprocessor: __orxEMBEDDED__;AL_LIBTYPE_STATIC
      . link libraries: box2d(d).lib SDL.lib GLFW.lib OpenAL32.lib libsndfile.lib SOIL.lib OpenGL32.lib winmm.lib
      . import library: ........libdynamicorxd.lib
      . Runtime library: 0/1 (ie. no DLL, all static)
      . Use C++ compiler
    - orx (static):
      . Uses the link values from above
    
    
    Codelite
    ========
    
     - orxLIB (dynamic):
      . include directories: same as VS
      . link directories: ../../../../extern/glfw-2.7/lib/mingw ../../../../extern/SDL-1.2.14/lib/mingw ../../../../extern/SOIL/lib/mingw ../../../../extern/OpenAL-soft/lib/mingw ../../../../extern/libsndfile-1.0.22/lib/mingw ../../../../extern/Box2D_2.1.3/lib/mingw
      . link libraries: glfw sndfile openal32 SOIL SDL Box2D(d) winmm
      . extra link flags: -static -Wl,--out-implib=../../../lib/dynamic/liborx(d|p).a
    
     - orx (static):
      . same as for VS, uses link properties from orxLIB (dynamic)
    
     - plugins:
      . extra link flags (static): -fPIC -Wl,--enable-auto-import
      . extra link flags (dynamic): -fPIC
    
  • edited October 2012
    Thank you Iarwain. The script should be able to spit out compile-able Visual Studio and Codelite solutions now. Only one way to know for sure:

    Would any Windows users care to test?

    Just grab the premake4 file attached, rename it to premake4.lua and put it in the orx/code/build directory. Make sure premake4 is installed and run it to create the projects.

    The command is: premake4 vs2010

    You can use vs2008, vs2005, or codelite instead of vs2010.

    Also, you need to run it from inside the orx/code/build directory.

    If it explodes in your face, come back here and yell at me. https://forum.orx-project.org/uploads/legacy/fbfiles/files/premake4-ce42b8b04f48d74abca33b30a62516c3.txt
  • edited October 2012
    I think I misspelled a library in that previous version. If it doesn't work with VS, this one might work better. https://forum.orx-project.org/uploads/legacy/fbfiles/files/premake4-54821d1b7ae6a73509d25bfe1361df97.txt
  • edited October 2012
    Thanks!

    I'll give it a try tomorrow night. Would you rather have a write access to orx instead of posting the file here?
    Another option would be to fork orx and create some pull requests.
  • edited October 2012
    I suppose that would be a better storage space. Don't give me write access though. I forked orx, and sent a pull request, though I don't know if you want this version in the main, as opposed to waiting until the script is complete and tested.
  • edited October 2012
    No need for a branch as we're not yet using premake to generate any file: while the system is not online you can make changes in the default branch as much as you want.
    I'll remove orx.premake too.
  • edited November 2012
    Hi,

    I recently want to try SublimeTex and remove VS, so I need to compile orx on Windows using another compiler. I just installed MinGW and tried to create a Makefile to compil orxLIB. I asked some questions to Iarwain and he talked to me about that post.

    I have downloaded premake, read really quickly its documentation, ran it in the build directory with the command premak4 gmake. It's magic and really nice, it works and compile find (I have tested orxLIB compilation). Really nice idea to use this tool!

    But, I have a problem at link, external libraries are not found, I suppose it's normal because I haven't compil them (ld cannot find -lBox2D for example).

    It's planned to create a premake file for external library? Or maybe I missed something?
  • edited November 2012
    Hi again !

    I find some mingw library in the extern directory. There's nothing to build them, but the library exist, don't know how :)

    (EDIT: I'm stupid, I know how, using codelite on windows ... but I don't have it)

    So I did some little change in the premake4.lua and now I have a orx.dll (not tested yet).

    I replace some "codelite" to "gmake":

    1/ line 369 : to enable all extern mingw library in the library path
    2/ line 468 : to enable sndfile to be linked

    With that, mingw32-make orxLIB seems to work.

    So, I suppose there's not a lot to do to enable gmake on windows without hacking the premake file like me ;)

    4babce, good job, really appreciate, thanks !
  • edited November 2012
    Thank you Faistoiplaisir, for testing and debugging the windows builds!
    I have made the required changes, so gmake should build properly on windows now.
    And, yes, hopefully the extern dependancies will be automatically built in the future, but for the time being they should be pre-compiled for you.
  • edited November 2012
    You're welcome ;)

    Your modification works! Thanks!

    I will post here some little remarks when I will found some mistake about premake.

    For example, I saw that building the debug configuration for orx build orx.exe instead of orxd.exe.
  • edited November 2012
    Hey there!

    I finally got the time to give it a try, however I ran into some troubles.

    When I run premake4 vs2010 (or any configuration, really), I get an error about "invalid flags". It'll actually stop at the first "flags" found in the script, no matter its value. Any idea?

    PS: Using Premake 4.4-beta4 on windows.
  • edited November 2012
    Meh, nevermind: works fine with premake 4.3
  • edited November 2012
    Ok, so I've been toying with the premake file for about a week and no matter what I do, the configurations are broken in a way or the other (like language C++ not forcing the C++ compiler, static builds not making a static library in vs, etc...).

    I ended up forking premake-dev in the orx's bitbucket account (nice that premake is also hosted on bitbucket), and will be toying around and see if I can make something work using the premake 4.5 dev version.

    If someone wants to help, all users having an access to the orx group account on bitbucket have access to that fork. :)
  • edited November 2012
    Well actually the dev repository is all broken so I cloned their stable repository instead (not sure why they're not using branches instead).

    I made a few changes to premake already and got all the windows versions (gmake, codelite, vs) running perfectly (embedded/non-embedded + static/dynamic + debug/profile/release).

    I'll check linux next as it should rely on the gmake/codelite work and shouldn't bring many surprises.

    Lastly will be mac os x as for now the xcode project generated doesn't allow to specify a SDK (and it defaults to the iOS one on my macbook O_o).
    Erwin Courmans of Bullet Physics fame wrote a patch a year ago to support this (as long as iphone builds), but it hasn't been integrated to premake (at least not in the stable repository).

    I see if I can integrate in our clone and support ipad as well.

    I rewrote most of the premake4.lua file for orx in the process but I didn't commit it yet as I don't want to break the linux support if someone is already using it. Those who are interested can find the latest version on my dropbox: https://www.dropbox.com/s/5kut2g15vh54q7h/premake4.lua
  • edited November 2012
    Well, this sounds like its getting hairier by the minute.

    Anyway, playing around with the file on linux64 with make.
    Some issues:

    Need to add
        configuration {"linux", "*Static*", "x64"}
    	links 
    	{
    	    "pthread",
    	    "rt" 
    	}
    
    Somewhere to the 'orx' project so that static64 builds correctly.

    Second,
    It looks like profile and release 32 builds are wonky, objects (cursor and profile overlay excluded) dissappear shortly after they spawn. To be more specific, the walls fade in, then dissappear. The balls and text spawn for a fraction of a second, then dissappear. It appears the objects are still alive however, based on the profiler.
    Defining __orxDEBUG__ for those builds fixes the problem.

    Have not tested non-embedded builds yet.
  • edited November 2012
    4babce wrote:
    Need to add
        configuration {"linux", "*Static*", "x64"}
    	links 
    	{
    	    "pthread",
    	    "rt" 
    	}
    
    rt is already part of all the linux builds and I didn't experience the need for pthread on my own linux64 (all the static builds were running on it). Out of curiosity, which version of gcc/libc are you using?
    I'll add pthread anyway but that sounds weird as orx doesn't use threads at all. Which file is asking for those symbols?
    Second,
    It looks like profile and release 32 builds are wonky, objects (cursor and profile overlay excluded) dissappear shortly after they spawn. To be more specific, the walls fade in, then dissappear. The balls and text spawn for a fraction of a second, then dissappear. It appears the objects are still alive however, based on the profiler.
    Defining __orxDEBUG__ for those builds fixes the problem.
    It's an optimization bug of some versions of gcc that where reported a few times in the forum. It mostly affects orxFXs and it doesn't happen with all versions of gcc. I guess we'll have to isolate exactly which optimization flags are responsible and manually deactivate them.

    I modified premake tonight as the linux codelite projects were not correctly created. I'll commit the binaries soon.
  • edited November 2012
    Ah, right I see that rt is already included.
    However, I do need pthread.

    gcc 4.7.2
    glibc 2.16.0
    ==== Building orx (embedded_static_debug64) ====
    Creating obj/x64/Embedded_Static_Debug/orx
    orxMain.c
    Linking orx
    /usr/bin/ld: ../../../extern/glfw-2.7/lib/linux64/libglfw.a(x11_init.o): undefined reference to symbol 'pthread_kill@@GLIBC_2.2.5'
    /usr/bin/ld: note: 'pthread_kill@@GLIBC_2.2.5' is defined in DSO /usr/lib/libpthread.so.0 so try adding it to the linker command line
    /usr/lib/libpthread.so.0: could not read symbols: Invalid operation
    collect2: error: ld returned 1 exit status
    make[1]: *** [../../bin/orxd] Error 1
    make: *** [orx] Error 2
    

    I tried rebuilding libglfw to no avail.
  • edited November 2012
    Mmh, glfw heh? I'm still not sure why it's working for me but I'll add pthread to the link dependencies right away (though it's weird it's working on linux32 and on my own linux64/gcc 4.5.2).

    I also installed a new linux32 box on my computer last night, so now I have one linux64 (Linux Mint 11) and two linux32 (Ubuntu 8.04 and Ubuntu 12.10).

    On the linux64, with gcc 4.5.2, all the optimized builds run fine.

    On the Ubuntu 8.04, with gcc 4.2.4, all the optimized builds run fine.

    On the Ubuntu 12.10, with gcc 4.7.2, the optimized builds are broken the way you described (ie. the alpha is reset to 0 when a FX that contain an alpha slot finishes).

    On windows/mingw(gcc) and os x/gcc, I never was able to repro that issue, no matter which version I used.

    With visual studio or llvm/clang, I never encountered that issue either, so I really wonder what rubs those versions of gcc/linux the wrong way.

    I'll track down which optimization flags are responsible, but it's going to be annoying.

    For now, if you remove the Object.AddFX calls from the WallTrack in code/bin/BounceAlt.ini, you shouldn't see any object disappearing anymore.
  • edited November 2012
    I had a look at the optomization flags myself. It's nontrivial, and no single flag is to blame.

    Interestingly, enabling
    schedule-insns 
    
    with -O2 fixes it, whereas just -O2
    its broken. However, -O2 is supposed to imply -fschedule-insns, so what the heck?

    Disabling any one of:
    cse-follow-jumps	
    expensive-optimizations
    regmove
    rerun-cse-after-loop
    fcaller-saves
    tree-pre
    
    with O2 enabled will also cause it work correctly. Enabling all of those above with -O1 will break it; removing one of them and it works again.

    No idea if disabling any of the O1 flags might also cause it to start working.

    EDIT:
    As a thought: does compiling with -O2 and -fno-schedule-insns on any of the other gcc versions cause the problem to appear?
  • edited November 2012
    Thanks for investing it, 4babce.

    Does enabling schedule-insns fix it in addition to -O3 too?
    And yes, you're right, schedule-insns (as well as schedule-insns2) are supposed to be part of -O2.

    I'll look if I can find any info on the gcc mailing lists as it surely doesn't happen only to us.

    As for using no-schedule-insns on other versions, I'll give it a try and let you know.
  • edited November 2012
    Yah, -03 works with -fschedule-insns
  • edited November 2012
    I added that flag to linux non-debug builds but haven't tested it (not on my computer atm). Hope I didn't break anything. :)
  • edited November 2012
    Ok, so apparently the premake script should be working for all platforms beside iOS and Android.

    It only handles the dynamic version for mac at the moment, and no XCode project either (as premake is actually creating 3 separate projects instead of having a single project with 3 targets :().

    Anyway, that's enough for our integrated building needs and I did some buildbot work at home last night (including a server that runs and two (a window and a mac) slaves that can sync and compile code.
    Now on to write scripts for creating the packages and pushing them online.
    Hopefully that means that we should soon be able to witness new binary package releases. :)
  • edited November 2012
    Great news,

    I'll take a look to setup a buildbot slave for android.
    about packaging:

    Is there a preferred layout for content?
    both android versions are static build and require a bunch of files like libBox2D.a, libNvEvent.a, libSOIL.a + some Android.mk required by the ndk build script.

    Lydesik
  • edited November 2012
    Well, I used to use a layout that mirrors orx's internal folder structure, but we can always decide to go with something else if we think it makes more sense.

    Also, why don't you merge all your static libraries together so that users only have to link -lorx in the end?

    EDIT: Something like:
    ar xv libBox2D.a | cut -f3 -d ' ' | xargs ar rvs liborx.a
    ar xv libNvEvent.a | cut -f3 -d ' ' | xargs ar rvs liborx.a
    ar xv libSOIL.a | cut -f3 -d ' ' | xargs ar rvs liborx.a
    ranlib liborx.a
    
  • edited November 2012
    Ta-da!
    buildbot.png
    buildbot 130.6K
  • edited November 2012
    And visual 2008!
    buildbot2.png
  • edited November 2012
    The mac version still requires some work though...
    buildbot3.png
  • edited December 2012
    I had talked about having the output of the compilation in a different folder by platform (win,linux32, linux64) I dont remember what the final decision about it was, but I'm throwing it back out here sinec people are working on premake and buildbot.
  • edited December 2012
    Well that actually means changing both /lib and /bin folder architectures + updating all the doc/wiki for a very limited audience (those who want to cross-compile on linux).

    That also means to duplicate all the config files as it's done in the tutorial folder, and I'm not very fond of the idea.

    That being said, you can easily modify the code/build/premake4.lua script in order to output different platform to different folders and then generate your own makefile/IDE project file.

    The manually created IDE project files will soon disappear except for the XCode ones as premake isn't doing a viable work for it, unfortunately.
  • edited December 2012
    If you sync the latest, there's an option for premake, namely --split-platforms, that will separate the target folders based on the platform.
    Ie: instead of /lib/dynamic, /lib/static, /bin and /bin/plugins/demo you'll have /lib/dynamic/x32, /lib/dynamic/x64, etc...
  • edited December 2012
    that's great, I'll set that up and update the tutorials.
Sign In or Register to comment.