Protected: Fleet Commander: production ready!

This content is password protected. To view it please enter your password below:

Advertisements

gnome-class: Integrating Rust and the GNOME object system

(this is a re-post from Niko Matsaki’s blog that I’m sharing here for the benefit of the Planet GNOME readers, you can read the original post here)

I recently participated in the GNOME / Rust “dev sprint” in Mexico City. (A thousand thanks to Federico and Joaquin for organizing!) While there I spent some time working on the gnome-class plugin. The goal of gnome-class was to make it easy to write GObject implementations in Rust which would fully interoperate with C code.

Roughly speaking, my goal was that you should be able to write code that looked and felt like Vala code, but where the method bodies (and types, and so forth) are in Rust. The plugin is in no way done, but I think it’s already letting you do some pretty nice stuff. For example, this little snippet defines a Counterclass offering two methods (add() and get()):

gobject_gen! {
    class Counter {
        struct CounterPrivate {
            f: Cell<u32>
        }

        fn add(&self, x: u32) -> u32 {
            let private = self.private();
            let v = private.f.get() + x;
            private.f.set(v);
            v
        }

        fn get(&self) -> u32 {
            self.private().f.get()
        }
    }
}

You can access these classes from Rust code in a natural way:

let c = Counter::new();
c.add(2);
c.add(20);

Under the hood, this is all hooked up to the GNOME runtime. So, for example, Counter::new()translates to a call to g_object_new(), and the c.add() calls translate into virtual calls passing through the GNOME class structure. We also generate extern "C" functions so you should be able to call the various methods from C code.

Let’s go through this example bit-by-bit and I’ll show you what each part works. Along the way, we can discuss the GNOME object model. Finally, we can cover some of the alternative designs that I considered and discarded, and a few things we could change in Rust to make everything smoother.

Mapping between GNOME and Rust ownership

The basic GNOME object model is that every object is ref-counted. In general, if you are given a Foo*pointer, it is assumed you are borrowing that ref, and it you want to store that Foo* value somewhere, you should increment the ref-count for yourself. However, there are other times when ownership transfer is assumed. (In general, the GNOME has strong conventions here, which is great.)

I’ve debating about how best to mirror this in Rust. My current branch works as follows, using the type Counter as an example.

  • Counter represents an owned reference to a Counter object. This is implicitly heap-allocated and reference-counted, per the object model.
    • Counter implements Clone, which will simply increment the reference count but return the same object.
    • Counter implements Drop, which will decrement the reference count.
    • In terms of its representation, Counter is a newtype’d *mut GObject.
  • &Counter is used for functions that wish to “borrow” a counter; if they want to store a version for themselves, they can call clone().
    • Hence the methods like add() are &self methods.
    • This works more-or-less exactly like passing around an &Rc<T> or &Arc<T> (which, incidentally, is the style I’ve started using all of the time for working with ref-counted data).

Note that since every Counter is implicitly ref-counted data, there isn’t much point to working with an &mut Counter. That is, you may have a unique reference to a single handle, but you can’t really know how many aliases are of Counter are out there from other sources. As a result, when you use gnome_gen!, all of the methods and so forth that you define are always going to be &selfmethods. In other words, you will always get a shared reference to your data.

Because we have only shared references, the fields in your GNOME classes are going to be immutable unless you package them up Cell and RefCell. This is why the counter type, for example, stores its count in a field f: Cell<u32> – the Cell type allows the counter to be incremented and decremented even when aliased. It does imply that it would be unsafe to share the Counter across multiple threads at once; but this is roughly the default in GNOME (things cannot be shared across threads unless they’ve been designed for that).

Private data in GNOME

When it comes to data storage, the GNOME object model works a bit differently than a “traditional” OO language like Java or C++. In those more traditional languages, an object is laid out with the vtable first, and then the fields from each class, concatenated in order:

object --> +-------------------+
           | vtable            |
           | ----------------- |
           | superclass fields |
           | ----------------- |
           | subclass fields   |
           +-------------------+

The nice thing about this is that the object pointer can safely be used as either a Superclasspointer or a Subclass pointer. But there is a catch. If new fields are added to the superclass, then the offset of all my subclass fields will change – this implies that all code using my object as a Subclasshas to be recompiled. What’s worse, this is true even if all I wanted to do is to add a private field to the superclass. In other words, adding fields in this scheme is an ABI-incompatible change – meaning that we have to recompile all downstream code, even if we know that this compilation cannot fail.

Therefore, the GNOME model works a bit differently. While you can have fields allocated inline as I described, the recommendation is instead to use a facility called “private data”. With private data, you define a struct of fields accessible only to your class; these fields are not stored “inline” in your object at some statically predicted offset. Instead, when you allocate your object, the GNOME memory manage will also allocate space for the private data each class needs, and you can ask (dynamically) for the offset. (Appendix A goes into details on the actual memory layout.)

The gobject_gen! macro is setup to always use private data in the recommended fashion. If take another look at the header, we can see the private data struct for the Counter class is defined in the very beginning, and given the name CounterPrivate:

gobject_gen! {
    class Counter {
        struct CounterPrivate {
            f: Cell<u32>
        }
        
        ...
    }
}

In the code, when we want to access the “private” data, we use the private() method. This will return to us a &CounterPrivate reference that we can use. For example, defining the get()method on our counter looks like this:

fn get(&self) -> u32 {
    self.private().f.get()
}

Although the offset of the private data for a particular class is not known statically, it is still always constant in any given execution. It’s just that it can change from run to run if different versions of libraries are in use. Therefore, in C code, most classes will inquire once, during creation time, to find the offset of their private data, and then store this result in a global variable. The current Rust code just inquires dynamically every time.

Object construction

gobject_gen! does not expose traditional OO-style constructors. Instead, you can define a function that produces the initial values for your private struct – if you do not provide anything, then we will usethe Rust Default trait.

The Counter example, in fact, provided no initialization function, and hence it was using the Default trait to initialize the field f to zero. If we wanted to write this explicitly, we could have added an init { } block. For example, the following variant will initialize the counter to 22, not 0:

gobject_gen! {
    class Counter {
        struct CounterPrivate {
            f: Cell<u32>
        }

        init {
            CounterPrivate {
                f: Cell::new(22)
            }
        }
        
        ...
    }
}    

Note that init blocks take no parameters – at the time when it executes, the object’s memory is still not fully initialized, and hence we can’t safely give access it. (Unlike in Java, we don’t necessarily have a “null” value for all types.)

The general consensus at the design sprint was that the Best Practices for writing a GNOME object was to avoid a “custom constructor” but instead to define public properties and have creators specify those properties at construction time. I did not yet model properties, but it seems like that would fit nicely with this initialization setup.There is also a hook that one can define that will execute once all the “initial set” of properties have been initialized – I’d like to expose this too, but didn’t get around to it. This would be similar to init, presumably, except that it would give access to a &self pointer.

Similarly, we could extend gobject_gen! to offer a more “traditional” OO constructor model, similar to the one that Vala offers. This too would layer on top of the existing code: so your init() function would run first, to generate the initial values for the private fields, but then you could come afterwards and update them, making use of the parameters. (You can model this today just by defining an fn initialize(&self) method, effectively.)

What still needs work?

So we’ve seen what does work (or what kind of works, in the case of subclassing). What work is left? Lots, it turns out. =)

Private data support could be smoother

I would prefer if you did not have to type self.private() to access private data. I would rather if you could just do self.f to get access to a private field f. For that to work, though, we’d need to have something like the fields in traits RFC – and probably an expanded version that has a few additional features. In particular, we’d need the ability to map through derefs, or possibly through custom code; read-only fields would likely help too. Now that this blog post is done, I plan to post a comment on that RFC with some observations and try to get it moving again.

Interfacing with C

I haven’t really implemented this yet, but I wanted to sketch how I envision that this macro could interface with C code. We already handle the “Rust” side of this, which is that we generate C-compatible functions for each method that do the ceorrect dispatch; these follow the GNOME naming conventions (e.g., Counter_add() and Counter_get()). I’d also to have the macro to generate a .h file for you (or perhaps this should be done by a build.rs script, I’m not yet sure), so that you can easily have C code include that .h file and seamlessly use your Rust object.

Interfacing with gtk-rs

There has already been a lot of excellent work mirroring the various GNOME APIs through the gtk-rs crates. I’m using some of those APIs already, but we should do some more work to make the crates more intercompatible. I’d love it if you easily subclass existing classes from the GNOME libraries using gnome_gen!. It should be possible to make this work, it’ll just take some coordination.

Making it more convenient to work with shared, mutable data

Since all GNOME objects are shared, it becomes very important to have ergonomic libraries for working with shared, mutable data. The existing types in the standard library – Cell and RefCell – are very general but not always the most pleasant to work with.

If nothing else, we could use some convenient types for other scenarios, such as a Final<T> that corresponds to a “write-once” variable (the name is obviously inspired by final fields in Java, though ivars is another name commonly used in the parallel programming community). Final<T> would be nice for fields that start out as null but which are always initialized during construction and then never changed again. The nice thing would be that Final<T> could implement Deref (it would presumably panic if the value has not yet been assigned).

Supporting more of the GNOME object model

There are also many parts of GNOME that we don’t model yet.

We don’t really support subclassing yet. I have a half-executed plan for supporting it, but this is a topic worthy of a post of its own, so I’ll just leave it at that.

Properties are probably the biggest thing; they are fairly simple conceptually, but there are lots of knobs and whistles to get right.

We don’t support constructing an object with a list of initial property values nor do we support the post-initialization hook. In C code, when constructing a GNOME object, once can use a var-args style API to supply a bunch of initial values:

g_object_new(TYPE_MEDIA,
             "inventory-id", 42,
             "orig-package", FALSE,
             NULL); 

I imagine modeling this in Rust using a builder pattern:

Media::with()
    .inventory_id(42)
    .orig_package(false)
    .new()

We don’t support signals, which are a kind of message bus system that I don’t really understand very well. =)

Procedural macro support on Rust is young

There is still a long ways to before the gnome_gen! plugin is really usable. For one thing, it relies on a number of unstable Rust language features – not the least of them being the new procedural macro system. It also inherits one very annoying facet of the current procedural macros, which is that all source location information is lost. This means that if you have type errors in your code it just gives you an error like “somewhere in this usage of the gnome_gen! macro”, which is approximately useless since that covers the entire class definition. This is obviously something we aim to improve through PRs like #40939.

Conclusion

Overall, I really enjoyed the sprint. It was great to meet so many GNOME contributors in person. I was very impressed with how well thought out the GNOME object system is.

Obviously, this macro is in its early days, but I’m really excited about its current state nonetheless. I think there is a lot of potential for GNOME and Rust to have a truly seamless integration, and I look forward to seeing it come together.

I don’t know how much time I’m going to have to devote to hacking on the macro, but I plan to open up various issues on the repository over the next little while with various ideas for expansions and/or design questions, so if you’re interested in seeing the work proceed, please get involved!

Finally, I want to take a moment to give a shoutout to jseyfried and dtolnay, who have done excellent work pushing forward with procedural macro support in rustc and the quote! libraries. Puttinggobject_gen! together was really an altogether pleasant experience. I can’t wait to see those APIs evolve more: support for spans, first and foremost, but proper hygiene would be nice too, sincegobject_gen! has to generate various names as part of its mapping.


Appendix A: Memory layout of private data

My understanding is that the private data feature evolved over time. When the challenges around ABI compatibility were first discovered, a convention developed of having each object have just a single “inline” field. Each class would then malloc a separate struct for its private fields. So you wound up with something like this:

object --> +--------------------+
           | vtable             |
           | ------------------ |
           | SuperclassPrivate* | ---> +-------------------+
           | ------------------ |      | superclass fields |
           | SubclassPrivate*   | --+  +-------------------+
           +--------------------+   |
                                    +--> +-----------------+
                                         | subclass fields |
                                         +-----------------+

Naturally any class can now add private fields without changing the offset of others’ fields. However, making multiple allocations per object is inefficient, and it’s easy to mess up the manual memory management involved as well. So the GNOME runtime added the “private” feature, which allows each class to request that some amount of additional space be allocated, and provides an API for finding the offset of that space from the main object. The exact memory layout is (I presume) not defined, but as I understand it things are currently laid out with the private data stored at a negative offset:

           +--------------------+
           | subclass fields    |
           | ------------------ |
           | superclass fields  |
object --> + ------------------ +
           | vtable             |
           +--------------------+

Although no longer necessary, it is also still common to include a single “inline” field that points to the private data, setup during initialization time:

           +--------------------+ <---+
           | subclass fields    |     |
           | ------------------ | <-+ |
           | superclass fields  |   | |
object --> + ------------------ +   | |
           | vtable             |   | |
           + ------------------ +   | |
           | SuperclassPrivate* | --+ |
           | ------------------ |     |
           | SubclassPrivate*   | ----+
           +--------------------+

Fedora: Laptop Hardware Enablement (hint: we’re hiring)

So, since September I have transitioned to a new job, from working on Fleet Commander full time with Oliver, to managing a team whose job is to make sure that a number of laptops (to be defined) are well supported in Fedora, and therefore upstream kernel, Wayland, MESA and GNOME. I’m honored to be working among great people in this space: Hans de Goede, Peter Jones, Christian Kellner and Benjamin Berg.

If you want more details as to the plans of the team, please watch the talk Hans gave at DevConf that you can watch on YouTube.

Of course, I’ll still be managing the Fleet Commander project, which is coming along pretty nicely. Currently Oliver is focusing on the FreeIPA integration.

By the way, I’m hiring a Senior Software Engineer for the team, we’re looking for someone with consumer & enterprise  hardware enablement related skills to help with TPM2 support for x86_64, POWER and ARM in Fedora, people willing to work from Munich are preferred, but we’re open to strong candidates working remotely, if you think this is for you get in touch with me and/or apply.

Rust 🖤 GNOME Hackfest – Mexico City, 2017

1487048523_e1b5bcdbe1_b_d
Glorieta del Ángel de la Independencia, Credits: laap mx@flicker cc-by-nc-nd

As you know a bunch of GNOMies have been in touch with Rustaceans to improve the Rust and GNOME binding integration. To accelerate the effort, we have decided to organize a hackfest soon-ish. After some digging and given that the great Federico has been involved in the effort we’ve figured that it was rather poetic to do it in Mexico City, birthplace of GNOME! Check out the wiki page for more details.

We will be meeting at the Red Hat office from the 29th to the 31st of March to try to advance the state of GObject bindings and try to make Rust emit GObject code as well. Get in touch with me or Federico if you want to attend!

 

First Rust+GObject coding session

Last Wednesday, Nicholas Matsakis from Rust/Mozilla and I sat down on a video chat to start getting our hands dirty on moving ahead with making Rust a part of the GNOME ecosystem.

While there are two efforts to produce GNOME bindings for Rust, gi-rust and gtk-rs, however none of them provided the means to emit GObject subclases so we decided to tackle this problem.

During the first half of the session we reviewed how GObjects are created from C, I used Vala’s output to illustrate how this is done. We didn’t dive on too much detail on how properties and signals are emitted nor interfaces, however we focused on detail on how to inherit from a GObject.

After that we went ahead and wrote what, probably, is the first piece of Rust code that emits a GObject in this playground repo. Nothing too fancy, but a starting point to get us somewhere.

The next step is to figure out how to take this and turn it into something that a Rust developer would find Rust-like. One of the options we’re exploring is to use Rust macros and try something like this:

gobject!{
  class Foo : Bar {
    fn method_a () -> u8 {
      ...
    }
  }
}

The other alternative would be to decorate structs, traits and whatnot, but this would be a bit more cumbersome. Turns out that Rust’s macro system is quite powerful and allows you to define a custom syntax.

That’s the progress so far, the next step would be to try to start implementing some bits and pieces for the macro and see how far we can get. I’d like to thank Mozilla and Nicholas specially for the amount of attention and effort they are putting into this, I must say I’m quite excited and looking forward to make progress on this effort.

Stay tuned, and if you’d like to help don’t hesitate to reach out to us!

Thoughts on DX: GNOME and Rust

A few months ago I spent some time to learn some basic Rust, I was interested in getting an informed view of the language, specifically about the safety and concurrency idioms as well as its compatibility with the C ABI and automatic memory management while being non GCed. I must say I was pleasantly surprised with the tooling, cargo is a breeze and crates.io is a wonderful resource. More recently though I’ve been investing time to actually understand the memory ownership model and how it plays with channels/concurrency. I must say that what these guys have achieved is really clever with a language that once you get the grasp of things feels actually really nice to use.

2000px-rust_programming_language_black_logo-svg

For a long while now I’ve been worried that the GNOME project would struggle to grow its contribution and stay attractive if it stuck to C in the long run (i.e. next 10-20 years). GObject hasn’t really caught on beyond the GNOME ecosystem. So we’re basically maintaining a whole low level framework, and we have to come up with something by ourselves everytime a new technology comes in (i.e. JSON, REST…). Given our limited resources, I’ve been wondering if there are better alternatives.

For people wanting to contribute to the core libraries consumed by the ecosystem, the only options are C and Vala. Vala has been a great tool for prototyping, I love it myself, but debugging it is a nightmare, it’s filled with security issues and even if we fixed those really difficult problems, we’d be maintaining our own language on top of everything else. I would like to see us maintaining less stuff other than a desktop and the application development framework, not more.

I trully believe Rust gives us a way out of this situation, with certain nice side benefits, Rust really ticks most boxes: C ABI compatibility, safety, modern syntax, vibrant community, ever growing set of libraries and tools and a culturally aligned organization backing it: Mozilla. So when Federico made the brave step of start using it in one of our libraries I was very encouraged by the notion that this might actually happen. I’ve been pleasantly surprised to see positive reactions to his effort by many core developers on twitter.

Now, imagine for a moment, that we decide to somewhat embrace Rust in our libraries, and we start adopting it in places like GTK+. Suddenly we have the opportunity to engage in the growing enthusiasm around Rust, and we have a channel to technologies and tools being built outside of our own community such as WebRenderer, Servo…

Additionally, we release ourselves from the burden of maintaining core libraries for everything so we can focus on producing a great desktop and application development story.

Ultimately though, there are many challenges, achieving full GObject compatibility can be difficult, we would need to be able to consume GI bindings from Rust and eventually emit GI bindings too, and in the end it would be up to the core of the community to lead an effort like this so there needs to be consensus too. It is quite a bit of work, but I believe it is worth considering as it might give us back a lot resources to focus on other stuff.

Please don’t read this as a formal proposal, I think something like that should come, I’m mostly putting my thoughts on this out there and see what the rest of the community thinks.

Need an ARM board to do GNOME development?

Now that the donations are in, it’s time to start putting them to use. If there’s any cool project you would like to use one of the available ARM boards or if you’re interested in doing enablement of those boards to make GNOME run on them please get in touch with me on IRC or drop me a line at aruiz  gnome org.

I have to note that requests to use them as your home server or for non-GNOME related stuff are discouraged, these boards were donated with the purpose of improving support for the ARM ecosystem in GNOME.

Here’s the list of the HW and who has them.

There are a few Banana Pi available (most with a Mali 400 GPU and two BPi M2s with a PowerVR SGX54MP2), so these are trickier to get GNOME Shell running on, if anyone enjoys reverse engineering GPUs, Malis are very popular these days and they get very little love as the Lima project is a bit silent for some time now. People looking into porting llvmpipe to ARM are also more than welcome!

As per the Qualcomm boards, they run with the freedreno driver (by the way, props to Rob Clark for his amazing work on this driver) and I was able to run a GNOME on Wayland  on them using the official Debian image, so they are more suitable if you want to focus on the upper layers of the stack.

I would like to reiterate my gratitude to Banana Pi and Qualcomm for their generosity for the hardware, as well as ARM and Codethink for the server side stuff that is already being used in our GNOME Continuous efforts.

And of course, I’m going to GUADEC! I’ll be taking the boards with me, so if you think you have something interesting to do with them and you are attending just find me around.

badge-goingto