Category: Hacking (english)

Over the last months I’ve been working in several projects, switched laptop and reinstalled my main distribution several times. Having to replicate the development environment one time after another and reinstall the compiler, tools, editor and all sorts of dependencies that stain a desktop distribution is a major nuisance. In the worst case, things won’t work as before due to incompatibilities of the new distribution. And what about bringing new developers to a complex project with a lot of dependencies which are difficult to track? The new team member can spend days trying to replicate the development environment of the rest of the team.

Fortunately, I learnt to use a tool that solves all these problems: schroot. Now I have one chroot for each of the main projects I work on. I can “transplant” it from one distribution or computer to the next, give it to a newcoming developer so that they can start hacking in the project in minutes, or just archive it when I’m not in the project anymore. It’s true that I’m spending more disk space with this approach but, from my point of view, the advantages are worth. Moreover, working in this way you start from a “bare” distribution and if your project build system is forgetting some dependency (because it “should” be on a desktop distribution) you will notice.

Making it work on a Debian based distribution is as easy as:

sudo apt-get install schroot

After that, you have to initialize the chroot(s) to use. Each chroot can be based on their own distribution. For instance, to create one chroot based on Ubuntu Quantal 64 bit to hack on webkit (so, named after it), you have to initialize it using debootstrap:

sudo debootstrap --arch amd64 quantal /srv/chroot/webkit http://archive.ubuntu.com/ubuntu/

Then you have to configure schroot to find it by editing /etc/schroot/schroot.conf and adding the new entry. There are a several flavours to configure chroots, but my favourite one is directory-based:

[webkit]
description=Quantal for compiling Webkit
type=directory
directory=/srv/chroot/webkit
users=enrique
groups=root
root-groups=root

If you want to have an independent home dir in the chroot, comment out the corresponding line in /etc/schroot/default/fstab (and probably in desktop/fstab too). If you’re going to use WebKit2, you need to enable (uncomment) shared memory support. The resulting fstab should look like this:

/proc           /proc           none    rw,bind         0       0
/sys            /sys            none    rw,bind         0       0
/dev            /dev            none    rw,bind         0       0
/dev/pts        /dev/pts        none    rw,bind         0       0
#/home          /home           none    rw,bind         0       0
/tmp            /tmp            none    rw,bind         0       0
#/run           /run            none    rw,bind         0       0
#/run/lock      /run/lock       none    rw,bind         0       0
/dev/shm        /dev/shm        none    rw,bind         0       0
/run/shm        /run/shm        none    rw,bind         0       0

Now, to enter in the chroot:

schroot -c webkit

Schroot has an automatic session management system that allows creating, reusing and killing sessions but which can be a bit uncomfortable at the begining. If you find yourself with a lot of zombie sessions, just kill them all:

schroot -e --all-sessions

As I’ve got tired of automatic and zombie sessions, now I use a tiny webkit.sh script in $HOME/bin to create, reuse and kill them on demand. Feel free to customize it for your needs if you find it useful.

For more info, here are the original sources I used to learn about schroot:

• https://wiki.ubuntu.com/DebootstrapChroot
• https://help.ubuntu.com/community/DebootstrapChroot

UPDATE: 2014-02-03

If you want to enjoy graphics acceleration (and probably other features), the chroot needs to have the same acceleration packages than the host. In my case (Nvidia card), I needed this in the chroot to avoid a big black rectangle while running Chromium/Blink:

apt-get install nvidia-319

UPDATE: 2015-01-08

If you want to use pulseaudio in the chroot, add these paths to /etc/schroot/default/fstab (my user is “enrique”, with UID 1000, use your owns):

/var/lib/dbus /var/lib/dbus none rw,bind 0 0
/home/enrique/.config/pulse /home/enrique/.config/pulse none rw,bind 0 0
# On other distros: /home/enrique/.pulse /home/enrique/.pulse none rw,bind 0 0
/run/user/1000 /run/user/1000 none rw,bind 0 0

This post explains how to compile C source code for Android using the Native Development Kit (NDK) by using autotools to set up the building infrastructure and using androgenizer to convert that autotools infrastructure into Android.mk files understood by the ndk-build tool. I’ll first review some autotools concepts very quickly trough examples and iterate over them.

Simple program

(Download the testapp.tgz example or browse it online)

We start from a simple helloworld program with its main() function that depends on a sayhello() function. We want to compile it using autotools in the simplest possible way.

To create an autotools template from scratch, follow the instructions of this presentation, which are abridged here:

  autoscan
  mv configure.scan configure.ac
  # Add this line below the AC_INIT line:
  # AM_INIT_AUTOMAKE
  nano configure.ac
  autoheader
  aclocal
  touch NEWS README AUTHORS ChangeLog
  automake --add-missing --copy
  autoconf

Makefile.am should contain:

  bin_PROGRAMS = testapp
  testapp_SOURCES = testapp.c

Then, to compile, just:

  ./configure
  make

The testapp.tgz file is an example of this. Look at the “autoall” script.

Library + program using libtool

(Download the testlib.tgz example or browse it online)

In this case we have to add a new libtool line to configure.ac, apart from the automake one, under the AC_INIT line:

  AM_INIT_AUTOMAKE
  AC_PROG_LIBTOOL

This is how Makefile.am should look like:

  lib_LTLIBRARIES = libtestlib.la
  libtestlib_la_HEADERS = testlib.h
  libtestlib_la_SOURCES = testlib.c
  libtestlib_ladir = $(includedir)
  libtestlib_la_LDFLAGS = -avoid-version

  bin_PROGRAMS = testapp
  testapp_SOURCES = testapp.c
  testapp_LDADD = .libs/libtestlib.so
  testappdir = $(includedir)

The LDFLAGS are the flags passed to libtool, and are documented here and here in the Autobook reference. In this case, the “-avoid-version” forces generation of libtestlib.so instead of libtestlib.so.0.0.0 (versioned libs aren’t supported in Android).

UPDATE: This “-avoid-version” flag isn’t needed anymore in recent versions of the NDK. They will generate unversioned libraries automatically and won’t recognize the flag (will show an error).

Look at the “autoall” script in the example for all the details. Now it’s fully automatic, so you don’t have to do the insertions by hand in configure.ac.

Androgenized lib + program

(Download the androgenized-testapp.tgz example or browse it online)

All the C code must reside in a directory called “jni”, inside the main directory of the Android project we want to create (in the androgenized-testapp example only the jni is included, technically the other ones aren’t needed):

androgenized-testapp
  src (Java source code)
  libs (compiled binary libs and executables)
  obj (intermediate object code for libs and exes)
  jni
    Application.mk (actually not mandatory)
    Android.mk (generated by androgenizer, drives the compilation when using NDK)
    Makefile.am
    testlib.c
    ...

A new target called Android.mk must be created in Makefile.am:

Android.mk: Makefile.am
        androgenizer -:PROJECT testlib 

        -:REL_TOP $(top_srcdir) -:ABS_TOP $(abs_top_srcdir) 
        -:SHARED testlib 
        -:SOURCES $(libtestlib_la_SOURCES) 
        -:LDFLAGS $(libtestlib_la_LDFLAGS) 

        -:PROJECT testapp 
        -:REL_TOP $(top_srcdir) -:ABS_TOP $(abs_top_srcdir) 
        -:EXECUTABLE testapp 
        -:LDFLAGS -ltestlib 
        -:SOURCES $(testapp_SOURCES) 
> $@

Androgenizer must be downloaded and installed somewhere in our PATH from here. It has a very brief parameter documentation, and the best way to understand it is by seeing usage examples, such as all the androgenizer lines added to the Makefile.am in the GStreamer project.

In the Android.mk target shown above two modules are going to be compiled: “testlib” and “testapp”. Each module is declared with a “-:PROJECT” parameter. The rest of the lines belong to the current project (current module) until a new one is declared. I’ve left a separation between projects to illustrate this.

Next line should always be a “-:REL_TOP” and “-:ABS_TOP”, as recommended in README.txt. These two values are used to check all the paths in LDFLAGS, CFLAGS, etc. and substitute local paths with absolute paths. This is important, because normal makefiles usually assume that the rules are executed from the subdirectory the makefile is in, while what is considered as the working directory for Android.mk makefiles is always the directory from which ndk-build is invoked. As you can imagine, compiler directives such as “-I..” are going to have a very different meaning and cause the wrong behaviour when passed to the compiler by ndk-build.

After that, we indicate the type of target for the project (“-:STATIC” for libtestlib.a, “-:SHARED” for libtestlib.so, and “-:EXECUTABLE” for testapp). The extension of the library/app is automatically calculated.

The “-:SOURCES” parameter is used to indicate the files to be compiled. In this case we can use the list of files already calculated by the previous automake rules (see testlib.tgz and testapp.tgz examples) and stored in the “libtestlib_la_SOURCES” variable.

The “-:LDFLAGS” parameter is used to generate the needed library dependencies for the current module. In the case of testlib, those dependencies are already calculated by automake in the “libtestlib_la_LDFLAGS”. In the case of testapp, it depends on testlib, so we indicate it manually with “-ltestlib”. That will translate into this line in the generated Android.mk file:

  LOCAL_SHARED_LIBRARIES:=libtestlib

this will make the NDK build scripts to link against the proper “libtestlib.so” (if the lib was dynamic) or “libtestlib.a” (if it was static). The NDK already knows how testlib was built and how it has to be linked.

The final “> $@” line just means that make has to take the output of the androgenizer command and dump it to a file named just like the target it’s being build, that is, “Android.mk”.

After all the autoscan, configure.ac customization, autoheader, aclocal, libtoolize, automake and autoconf steps we can “./configure” and “make Android.mk” to generate the “Android.mk” file. Note that no full “make” is needed. Then we step back one directory (“cd ..” to the project root) and perform the actual build using NDK (which has to be downloaded from here, installed and be in our PATH; the version I used to write this post was NDK r8):

  ndk-build V=1

The V=1 option just prints out all the executed commands, which is very handy. The compilation result will be placed in the “libs” directory.

To understand the process a bit more, it’s a good idea to look into the generated Android.mk file:

  LOCAL_PATH:=$(call my-dir)
  include $(CLEAR_VARS)

  LOCAL_MODULE:=testlib
  LOCAL_SRC_FILES := 
          testlib.c
  LOCAL_LDFLAGS:=
          -avoid-version
  LOCAL_PRELINK_MODULE := false
  include $(BUILD_SHARED_LIBRARY)
  include $(CLEAR_VARS)

  LOCAL_MODULE:=testapp
  LOCAL_SRC_FILES := 
          testapp.c
  LOCAL_SHARED_LIBRARIES:=
          libtestlib
  LOCAL_PRELINK_MODULE := false
  include $(BUILD_EXECUTABLE)

Apart from some boilerplate (LOCAL_PATH, CLEAR_VARS, LOCAL_PRELINK_MODULE), we can distinguish the two modules (LOCAL_MODULE), its source files (LOCAL_SRC_FILES), inherited LDFLAGS (LOCAL_LDFLAGS), required libs (LOCAL_SHARED_LIBRARIES) and finally, the building instruction (BUILD_SHARED_LIBRARY, BUILD_EXECUTABLE).

Seeing the Application.mk and Android.mk supported directives are documented somewhere in your installation of the NDK. They support more options than the ones explained here. With this basic introductions, that documentation should be more understandable. Application.mk can be edited directly. Android.mk has to be customized using the “-:PASSTHROUGH” androgenizer parameters in Makefile.am. For example:

  -:PASSTHROUGH LOCAL_ARM_MODE:=arm

Unfortunately, real life autotools build scripts are a bit more complex than the examples shown here. Here are some debug tips that can be helpful:

• Use “ndk-build V=1” to get the exact command line used to compile each file. When something fails, execute that command line independently and examine all the flags passed to it trying to look for incorrect compiler flags or missing include paths. Try to figure out what the correct command line flags would be.
• Locate where the offending flags come from in the Android.mk. From there, locate them in the Androgenizer Makefile.am target. Usually some Makefile.am variable has the wrong values. Need to debug more on how Androgenizer is called? Go to the desired directory, delete Android.mk, make Android.mk and see the executed command line.
• Where the Makefile.am variables comes from? What Makefile.am variables should be used? Examine the generated Makefile to find the values of all possible Makefile.am variables. That’s the place to look to decide what to pass to Androgenizer and what not.