Introducing BuilDj: Software Project Definition Format that GNOME deserves

In the GNOME beer event, I had a nice chat with ebassi about the problems around our build configuration system and how things like CMake, SCons and Waf do not get the full picture and do not cover some of the really strange corncercases that autotools supports and therefore, coming up with a solution would be a 1 year work with a team of really experienced engineers.

As stubborn as I am, I decided to prove him wrong so I give you BuilDj:

The Problem

My main problem with our current Autotools situation are these:

  • It is not portable as it encourages strong use of Bash and command line tools which ends up making it really hard to use on a non POSIX system, you know, like that used by 92% of the desktop users, even if the source code itself is perfectly portable.
  • It gets on your way, a lot. Sometimes I refrained from writing some testcases just to avoid to add the Autotools boilerplate.
  • Really bad documented, still.
  • You need to learn M4/Autoconf, Automake, Make and Bash to use it properly, as if learning C, C++ or Vala and worrying about the problem the programmer is trying to solve is not hard enough.
  • None extends it because it's hard to extend.
  • Its hard to read and understand even if you eventually wrote it yourself, makes code refactoring a big pain.
  • You simply can't parse autotools and have an IDE or a continous integration tool to understand what's there.
  • The most important one, it scares peoples away, damaging our mindshare, making potential contributors go away. It drains the fun out of our platform.

Requirements

We need a human+machine friendly project description format, that it's pleasant to the eye when you read it, that it's intuitive enough to let you understand what's going on even if you never saw it before.
A format that gets out of your way!

It should support all of the common tasks a GNOME maintainer does (in-line .pc and .desktop file definition, mkenums, gobject introspection support, cross-compilation support, pkg-config oriented, xdg mimetype registration/definition, integration with intltool), but it should suppor them in a meaningful and unobtrusive way.

It should not be a programming language, but support embedable programming extensions with a well documented API, so that IDEs can integrate it.

My Proposal

 A JSON description format called BuilDj that is build-system agnostic (although its reference implementation is done with Waf)

{
"project":
{
"name": "BuilDj Test",
"version": "0.0.1",
"url": "http://www.codethink.co.uk"
},
"requires":
{
"gtk+-2.0":
{
"type": "package",
"version": "2.14",
"mandatory": "True"
}
},
"targets":
{
"my_shared_lib":
{
"type": "sharedlib",
"tool": "cc",
"input": ["lib.c"],
"version": "1.2.3"
},
"my_gtk_program":
{
"type": "program",
"tool": "cc",
"input": ["gtk_program.c"],
"uses": ["my_shared_lib"],
"packages": ["gtk+-2.0"]
},
"my_vala_program":
{
"type": "program",
"tool": "vala",
"input": ["vala_program.vala"],
"packages": ["gtk+-2.0"]
}
}

I don't even need to explain what that means right? By the way, this stuff already works, check the git repository.

Currently it is implemented as a waf wscript that parses the project.js json file, I'm not really interested in entering the build system wars but focuing on having a reference implementation of the format.
I choosed waf because it was the only one that offered most of the features I want as an approachable API and it only adds python as a dependency.

Implementations of the format in other systems are more than welcome, but current development will stick to waf in the foreseeable future.

I'm planning to propose and mentor this work as a Summer of Code project so that we can implement the missing basic features and support for a few GNOME apps.

There's a lot of work to do, support for C++, library and function checking, system type sizes, full cross compilation support. We already have some mockups and plans for those, and the waf maintainer has shown himself quite happy to accept patches upstream for the general purpose tools and that make things easier.

Despite the missing features, it surprisingly itches some of my own scratches already.

P.S. Anjuta and MonoDevelop guys, if you're listening, I'd love to get your feedback!

Advertisements

47 thoughts on “Introducing BuilDj: Software Project Definition Format that GNOME deserves

  1. @Jeff:
    I am, they totally ignored the cross compilation use case in their design. Plus the project generation approach (same used by CMake), means you have to deal with Gyp plus the toolchain you generate with, and you can’t do stuff the other way around (you have to start working with Visual Studio or XCode, and you tweak some bits not supported by CMake or Gyp, and you’ll have to manually change that until a solution is found upstream, been there, done that)

    Like

  2. Nice! I can’t wait for the day when I no longer have to use ugly make files.
    Can you add a snippet for how to build the example project above?
    On a minor note. What is with “True” as a string instead of a bool?

    Like

  3. It seems like a good start, but here are some of those corner cases that I think matter:
    * Check for random headers
    * Hooks for checking versioning in said headers
    * Building out of tree
    * Handle regex cleanly (imagine all the escaping in XML!)
    * Execute external scripts
    * Generating documentation?
    * Unit test integration?
    * Release management? (dist-check, etc)
    * Library versioning for installation
    * Integration files for familiarity
    – Makefile for allowing `make’
    – configure for allowing host configuration with `./configure’
    * Target may be written in a different language for different OS
    * Generate files based on build variables (like autogenerated headers)
    * Crank up those compiler warnings as errors by default! make it hard to suck
    * An input might also have been a target
    I’m sure I’ll come up with more as I think about it more.
    Cheers,

    Like

  4. It’s time for Linux in general and Gnome in particular to have a easy to use, IDE integrated build system. I hope this goes somewhere and it’s embraced by at least (most) of the Gnome platform projects. Also, I would like to see some integration with an easy generation of DEB and RPM packages.

    Like

  5. One thing I am a big fan of is the fact that the language isn’t turing complete. I do the build system at work, which is C-centric and solves some different problems. However one thing that is common is that there are files which you can read and parse and know how to build the project. It seems that once you go down the general purpose language road, the metadata starts to say what steps you need to build the project, rather than letting the build system figure out those steps.
    One thing you might want to consider is treating the configuration file as a way of giving metadata to individual files, rather than metadata about projects. I think of my work system as a way of describing what a file can be used for, and then a “project” is just a way of saying what types of files it wants.
    This it can express things like:
    – When foo.c is used in a project, you need to compile with “-pthreads” (actually, it is slightly more abstract, as it handles different compilers and operating systems, so it just makes sure threading stuff will work)
    – When a.c is used in a project, you are also going to need b.c
    – When compiling for platform X, you want Xfoo.c, when compiling for platform Y, you want Yfoo.c
    The other thing which makes this one work nicely is that the file metadata lives next to the file. Instead of having one top level build configuration, there metadata forms a tree structure, with the top level file referencing files at lower levels. This has a few wins – things like restructuring a directory tree are much more pleasant, and the metadata files are also simple and modular.
    Just some thoughts, feel free to email me if you have other questions, I’ve thought a fair bit about this stuff.

    Like

  6. any plans to also support something like yaml?
    i’m tinkering with a semilar kind of thing but just for python and yaml had a few nice degrees of freedom
    of course this might also complicate alternative implementations, but i decided to go with flexibility for a beginning

    Like

  7. Rebuttal time :
    “It is not portable as it encourages strong use of Bash and command line tools which ends up making it really hard to use on a non POSIX system, you know, like that used by 92% of the desktop users, even if the source code itself is perfectly portable.”
    Using dash as /bin/sh will help you find your non-portable shell code. Really.
    “It gets on your way, a lot. Sometimes I refrained from writing some testcases just to avoid to add the Autotools boilerplate.”
    Don’t really know what your complaint is, here. “Make check” isn’t a silver bullet on its own. The glib unit testing system makes it easy.
    “Really bad documented, still.”
    That _used_ to be true. Now the documentation is much cleaner, and one of Automake’s maintainers also maintains a really easy PDF tutorial.
    “You need to learn M4/Autoconf, Automake, Make and Bash to use it properly, as if learning C, C++ or Vala and worrying about the problem the programmer is trying to solve is not hard enough.”
    The aforementioned tutorial offers a simple intro to M4. Again, there’s nothing complex about it. It’s _much_ simpler than learning python (for scons) or CMake’s god-awful language.
    As for learning make and bash… come on.
    “None extends it because it’s hard to extend.”
    Just take a look at Xorg’s macros [1] that are used throughout all Xorg packages. They’re simple, easy to read and really help maintainers.
    “Its hard to read and understand even if you eventually wrote it yourself, makes code refactoring a big pain.”
    That’s partly true, especially if your autotools files were written a long time ago. Now a lot of packages can be cleaned up. Just take a look at the clean ups I wrote for gnome-power-manager [2]
    “You simply can’t parse autotools and have an IDE or a continous integration tool to understand what’s there.”
    That’s an inherent problem with C compilers, which have hundreds of flags and options. Newer languages like C#, python, java, etc don’t have that problem. It’s easier to build programs so it’s easier to build tools around them.
    “The most important one, it scares peoples away, damaging our mindshare, making potential contributors go away. It drains the fun out of our platform.”
    You really need to back this up. I for one started programming with Visual Studio 6 and was _blown_away_ when I started hearing and learning about autotools. It made _sense_ compared to the mess VC++ was (and still is, even with VC2008).
    And having worked with other “enterprise” tools like ClearCase (which does both version control and build), Autotools is a _great_ tool.
    Sure, no tool is perfect, but you’re barking up the wrong tree. There should be a big GnomeGoal to clean up and “standardize” Autotools use within Gnome. Xorg did it and it works fine.
    Cheers 🙂
    [1] http://cgit.freedesktop.org/xorg/util/macros/tree/xorg-macros.m4.in
    [2] https://bugzilla.gnome.org/show_bug.cgi?id=600605

    Like

  8. Another reason: autotools is slow.
    @Remi: I don’t think there is a tool which allows flawless integration with autotools. Most tools don’t allow manual editing of the auto-foo files, or only partial editing. This is reason enough that there is a problem with autotools…
    You may like to write 5 languages in multiple files instead of 1 language in 1 file, but that’s a personal taste. I don’t think it will be that hard to write a autotools generator for this project files. so everybody can be happy 🙂
    and be honest, even if you claim it is simple. It looks like the other part doesn’t find it simple enough…and they want to program, not figure out how the build-system is supposed to work. So this is a nice initiative

    Like

  9. This is a goal worth pursuing! GNOME and, in general, free software really desperately need an easy-to-use build system.
    I’m curious about something, though. How did you manage to learn Waf? I’d really want to give it a try, but I found no usable documentation other than a collection of rants mixed with seemingly unconnected bits and pieces of information. Am I missing something?

    Like

  10. Meh, I don’t think it’s a goal worth pursuing. The autotools are popular because they’re so extensible. But hey at least you put your money where your mouth was.
    It would be interesting to see this project as a front-end to the autotools, so that IDEs could interface with it. Although I recall Anjuta could do autoconf machinery in the past.
    I didn’t agree with the majority of your reasons to create buildj, but I’ll pick on one of the reasons:
    I don’t think the autotools raises the bar too high for prospective developers. Sure it’s not always easy to use, but there are a lot ppl who do find it easy to use. Newbs can go onto google.com/codesearch and find something that works.

    Like

  11. @Ben: I build software for living and am a GNOME dev yet I need to re-learn autotools each and every year. Mostly because some AM_FOO becomes obsoleted by some AC_FOO or build starts to break in mysterious ways.
    Also, learning from code snippets returned by google just makes you replicate the same errors.

    Like

  12. @Alberto: This looks very similar to my blog post “To make, or not to make – QMake and beyond” point 2), from back in October 😉 Good to see something concrete. Not sure about melding all the targets into one single JSON blob though..

    Like

  13. @Ben: That’s exactly the current problem: “Newbs go onto google.com/codesearch and find something”.
    Something that they will copy, with no clue which parts are actually only needed, which parts might not make sense, and what the hell that code is doing.

    Like

  14. And why is CMake actually unsuited for GNOME? It has a very healthy developer community and good tools (cmake-gui) and IDE integration. En plus, some very large projects use it (like KDE, which used Autotools before and managed to switch). I agree that Autotools suck, but why invent yet another wheel?

    Like

  15. I think Christian makes a lot of good points about features that are missing. Except that while he calls them corner cases, I’ll bet that for some people, each of those is a deal breaker.
    I use and support the autotools a lot (xorg), and I concede a lot of the points you’re making about the difficulty of using them when you just want to get something done. What I think is actually the biggest reason for using the autotools is the sheer number of use cases it supports. You may think that people don’t need all these things, but I guarantee that there are people out there who rely on them.
    In other words, good luck emulating all the insane platform support in libtool or having people cursing you if you don’t.

    Like

  16. >What I think is actually the
    >biggest reason for using the
    >autotools is the sheer number of
    >use cases it supports. You may
    >think that people don’t need all
    >these things, but I guarantee
    >that there are people out there
    >who rely on them.
    But flexibility cannot be so high or it will bite you in the ass in the future at some point. Look at mozilla for example, they now depend on some old autotools version because they were using it in such a clunky way that now it’s not compatible with autotools upstream (it should be super clunky for the autotools devs not to care about their use case!).
    Sascha Peilicke
    >And why is CMake actually
    >unsuited for GNOME? It has a
    >very healthy developer
    >community and good tools
    >(cmake-gui) and IDE
    >integration. En plus, some
    >very large projects use it
    >(like KDE, which used
    >Autotools before and managed
    >to switch). I agree that
    >Autotools suck, but why >invent yet another wheel?
    +1 +1 +1
    Please take this in account. Don’t fall in NIH syndrome again.
    It seems scons has a lot of disadvantages but has the advantage of having the python syntax (no need to learn another syntax).
    And CMake has the ugly syntax and some corner cases not covered, but has the community, the maturity, the existing integration with other tools/IDEs and the proof that large projects have switched to it successfully.
    So why not have the best of both worlds without NIH? I.e. build a plugin for CMake that allows you to use python syntax, and fix every corner case that Gnome needs for its usage. And just not be so strict about pkg-config (yes it’s supported by CMake by an addin, why you developers want it so badly not to be an addin? in the end, pkg-config is not cross-platform).

    Like

  17. If this makes it easier to build in Windows (MSys or Visual Studio) and OSX then this would be a big win… everything works fine in Linux, but outside not so much.

    Like

  18. @Andres:
    CMake has serioous desing flaws that are not ease to overcome, I already mentioned I’m not interested in the buildtools war so I’m not getting there.
    By the way, pkg-config is cross platform actually.

    Like

  19. Alberto, I really like this idea. I think a universal description syntax which can then use waf, or generate autotools output, or whatever else, is probably the best solution to this problem.
    One point, have you had a look at Nix? I know it’s crazy and would probably make this a two year project instead of a one year project, but I think it’s worth knowing about. actually it would be possible to make a converter from buildj to nix packages, so we would be a step closer to the utopia I attempted (poorly) to describe here: http://ssam.livejournal.com/9966.html
    Also, Remi: here’s why autotools dissuades people from writing unit tests. To create one for my project, which uses waf, I:
    * write tests/test-case.c
    * run ./waf –check
    Let’s compare with autotools:
    * write tests/test-case.c
    * edit tests/Makefile.am, to add the source file, and the target
    * run ./config.status
    * run automake
    * run make check
    Let’s not forget on big projects, or slow systems, these commands can take a while.

    Like

  20. I’m one developer of Anjuta. I don’t think autotools is so bad.
    * The documentation is not that bad and you can easily find code samples. I think it’s still the standard on all POSIX systems.
    * It’s quite easy to extend if you know M4, quite several packages have their own macros. The syntax is logical , it is just easier for a computer.
    * It’s not true that you can’t parse it. Anjuta is able to parse and modify autotools files since several years. It’s not perfect but I don’t think it’s easier to parse C++ or a scons files.
    The issue is that autotools is designed to allow an user to get your sources and compile them on his own POSIX system. It’s not designed to make the life of the developer easy. I think everybody can agree that it’s a big success :).
    It’s possible to do something better than autotools but it’s much more than just building a few programs on 2 or 3 common systems. Moreover, there is a big advantage to keep compatible data formats. Even if autotools doesn’t always succeed, buildj will have to overcome this hurdle.
    In Anjuta, we are working to improve autotools support to make it easier to use for developers but we can support other project formats.

    Like

  21. Mmh, I find this one description file format to suit all needs a bit simplistic IMHO. The Gnome community usually claims to have superior collaboration practices and yet it fails horribly with the basic things like establishing a solid foundation to built upon.
    What about extending/extracting and *documenting* all common stuff into gnome-common like mm-common, xfce4-dev-tools, xorg-macros first, instead of:
    a) requiring each project to copy/create their own macros and scripts (git-to-changelog, introspection.m4)
    b) duplicating the same stuff all over again (SILENT_RULES, MAINTAINER_MODE)
    c) search individual packages for m4 macros they might provide and figure out what they are supposed to do and whether you should actually use them. (GTK_DOC_CHECK, AM_PATH_GLIB_2_0)
    Unless you have common ground rules and proper documentation, another build description format solves nothing IMHO.

    Like

  22. Alberto: thanks for your answer. But saying just “it has design flaws” doesn’t help that much. Can you give URLs of these issues in their bug tracking system to see what the CMake devs think about them?
    Oh and Karl’s is a good point too. We could make this parseable now (regardless of the solution we choose) so jhbuild can understand dependencies changes without having to do changes in both places (in your project and in jhbuild definition files).

    Like

  23. 1) Please push this idea into autotools upstream! If autotools could create multiple build systems (configure + Makefile, IDE projects, packaging formats etc) from the one project description, while being flexible enough for those who need it to be, that would be awesome and hopefully kill off some of the zillion other build systems out there. pkg-config style stuff would be good here too.
    2) Please use a less verbose format like YAML instead of JSON.
    3) waf is being removed from Debian, so your implementation is not useful there:
    http://lists.debian.org/debian-devel/2010/02/msg00714.html

    Like

  24. Like Ignacius said, this could be used to generate DEB and RPM packages more easily. I think it goes deeper than that, though. I think this format has the potential to specify dependencies in a distribution-neutral way. This could make it dead simple to generate packages for all major distributions really quickly and easily. Furthermore, one could imagine a user downloading the source code, double-clicking an icon, and having a package built for his distribution automatically. As long as his distribution knows which of its packages correspond to the stated dependencies, it wouldn’t even matter if he uses some crazy new distribution that the software developer has never even heard of with a completely unique package management system. Talk about easing the pain of large-scale software distribution for Linux!

    Like

  25. @Alberto Ruiz
    You claim to be “not interested in the buildtools war”, but this *is* a build tool. “CMake config file + Make” = build tool = “BuilDj + waf”.
    This is just a *huge* case of NIH, and you are essentially re-inventing CMake and following the exact same path as CMake. CMake started out a simple config file like this and was expanded based on user demands into what it is now. Despite its flaws CMake is currently is 10-100 times more powerful than BuilDj and used on projects of the same scope as Gnome.
    You efforts would be much better served by expanding CMake in the areas you find lacking. I have looked into adding a python wrapper for it’s internal API, and think it would a good summer of code project. You could also help CMake expand cross-compilation and pkg-config support if that is what you need.
    You would not only create the tool you need with less effort, you would also improve CMake for all it’s existing users.

    Like

  26. Hi,
    Now that I’ve read the whole post, I can give more constructive criticism :
    * I think there is no point in separating shared and static libs (I can go as far as to say we need no distinction between libs and packages). That is, it should be just an option (like static_only), and for the rest they should be equivalent.
    * (not very important) do we really need the toplevel braces? (I guess yes). Removing them would remove a level of indentation.
    * Is the “tool” option really needed? Really, isn’t it clear from source files extension that we need C/Vala? Having a “tools” option containing things like gobject-introspection that won’t be always used makes sense, but for basic things, I think not.
    * I’d rather see packages attached to ‘targets’ rather than directly to the project.
    * A target probably need a “path” option (I hope this is already implemented)
    * Is there a mailinglist/irc channel yet? How are we supposed to discuss this?
    * For the GSoC 2010 idea, it may be a better goal to implement it correctly using C or Vala. Having a working prototype (and I mean by working, at least able to compile something like vala and maybe even more) in Python, and a half finished port (by reimplementing the parts of Waf needed) may be better. Thoughts?

    Like

  27. Hi,
    This is something that I have though about a lot in the past, and actually started to write my own tool for[1]. It sounds as though you are coming to the problem from much the same point as I was, which is also roughly the same approach as the Maven project (which I was considering as a baseline example) – that the important thing to do is wrap all knowledge of how to actually do the build in the tool, and let the end user simply declare what they want.
    My build file format was actually very similar to yours, but in xml, and assuming autodiscovery of source files, and some other minor differences.
    This is still something that, barring a lack of time, I am very interested in. Please feel free to look at my sources, and if you think we are on the same wavelength, I would be more than happy to discuss this further.
    Phil
    [1] https://launchpad.net/realise, and also http://mail.gnome.org/archives/vala-list/2009-September/msg00365.html, http://mail.gnome.org/archives/vala-list/2009-October/msg00144.html

    Like

  28. I would love this for MonoDevelop! Autotools is a nightmare if you’ve never used it before. I’ve been desperately hoping for something like this for a while now.

    Like

  29. +1 For YAML. I have applications that use hand written JSON for configuration, it can be an improvement over XML, but for really large files, readability will eventually become a problem.
    YAML has cleaner syntax, and it has a form of inheritance, so you don’t need to repeat the same settings on a bunch of different objects.

    Like

  30. (I have already tried posting this once, apologies if something similar is in a moderation queue or something.)
    I was thinking about a build tool a while ago, specifically in the context of Vala, but always intended to be general purpose. The build descriptor format I made is actually fairly similar to yours, although in XML – and JSON was suggested on the mailing list as maybe a more succinct option, so the overlap is almost uncanny in places…
    In much the same way, I was thinking about a tool which would be mostly declarative, and make little direct attempt to support non-standard use cases. The thing would be extensible, but focused around conventions more than anything else. In that, Maven was my primary model.
    Sadly, I haven’t done more than poke at it occasionally for a few months, as real life and Java got in the way, but it is something I will return to, hopefully quite soon.
    My project is at https://launchpad.net/realise, and a pre-announcement was at http://mail.gnome.org/archives/vala-list/2009-September/msg00365.html. Please feel free to look at the code[1], and when I get a chance I would like to see if your format could be integrated into it.
    [1] What there is uploaded is at http://bazaar.launchpad.net/~undeconstructed/realise/proto-1/files

    Like

  31. Sir, you are a hero. I was just starting up a new code project and hunting for a build system. Almost used Waf (which was wonderful enough), then I remembered this which is exactly what I have been looking for for AGES! 🙂
    I really look forward to watching it evolve.
    No longer must I be jealous of IDEs for making this stuff easy.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s