Portable GNU-Linux binaries

The builds we distribute for GNU/Linux, including those included in exported "Linux Tarballs" and "Debian Linux Packages" are hopefully portable to other distributions. This page documents how we achieve that, and methods considered.

To compile a portable build, run scons portable=1.

Discussion

I looked at how several Linux games I had were packaged/distributed.

• Almost all of them included separate binaries for x86 and for x64.
• Almost all included .sos for SDL/SDL2 and other SDL libraries if they used them, as well as libraries like GLEW
• A few of the games included libstdc++.so.X or libgcc_s or similiar
• A few included libogg, etc.
• Nobody seemed to have statically linked anything such as libc or other common libraries
• Steam includes heaps of libraries from Ubuntu 12.04 that override many of the system ones, such as SDL

Libraries

It's not necessary to statically link the binaries.

libc

We can't require a recent version of glibc. For creating portable binaries I have seen people recommend building on an old distro such as CentOS 5. But using a too-ancient distro will result in requiring libraries like libstdc++5 (gcc 3.2-3.3.3 from 2002-2004) that some distros like Ubuntu don't bother to include by default (while Slackware does, but not libstdc++4).

An ELF binary includes fine-grained symbol version requirement tags for glibc and some other libraries, and these requirements don't get added unless you actually use a function or global which is annotated with such a requirement. So it is OK to compile against a recent glibc as long as you check the result and fix any problems. Unfortunately, this needs to also be done for any dynamic libraries that we distribute.

We link to glibc_compat.c and pass some magic flags to the linker to avoid dependencies on certain glibc functions - see below.

libstdc++.so

The required version of libstdc++.so depends on the version of g++ used (see the table here, item 4) and the features of that library which are actually used. These can be checked by looking at GLIBCXX in the objdump -p output; see below. New versions of libstdc++.so.X.Y can be used with any program compiled against libstdc++.so.X.Z with Z <= Y.

The last compatibility-breaking release was with GCC 3.4.0 (20040419).

Our options:

1. compile with an old version of GCC
2. ensure that we aren't using any functions in libstdc++.so that are new (by checking GLIBCXX), and provide shims if we are
3. statically link to libstdc++.so
   • CXXLINKFLAGS += ['-static-libstdc++']
4. include libstdc++.so in the packages and link to it with LD_LIBRARY_PATH

On 32-bit x86, statically linking libstdc++.so.6.0.21 adds about 1MB to the binary size (with debug=0), and 400KB after zipping. libstdc++.so itself zips to 500KB. Since unlump, relump, ohrrpgce-game and ohrrpgce-custom all depend on it, it could save a bit to ship libstdc++.so.

We went with option #2. Shims are provided in lib/stdc++compat.cpp, which is adapted from Mozilla. Also we compile C++ code with -D_GLIBCXX_USE_CXX11_ABI=0 which makes GCC avoid some newer parts of libstdc++.

libgcc_s.so

This dependency can be avoided by adding CXXLINKFLAGS += ['-static-libgcc'] (when linking with gcc instead of fbc). It only adds 15KB to the binary!

libtinfo.so

Programs compiled by fbc will depend on libtinfo.so or libncurses.so, which is very unportable*.

We avoid this problem by not linking to these libraries (by using gcc instead of fbc to link objects; or you can pass -nodeflibs to fbc) and instead linking to lib/termcap_stub.c.

* libtinfo.so is a very common incompatibility between distros. ncurses can be compiled either as just libncurses, or as separate libtinfo and libncurses libraries, where libtinfo just has terminfo functions. Various distros compile it differently; Debian compiles as two libraries. If you get an error the workaround is to symlink libtinfo.so to libncurses.so. FB itself requires libtinfo if it exists, and otherwise falls back to libncurses. (This is logic in src/compiler/fbc.bas - well-intentioned but annoying).

Other libraries

Very commonly, a shell script is used to set LD_LIBRARY_PATH, a relative path for the linker for included libraries, although it is possible to set a link-time flag when compiling to look in the same directory as the executable (or some other relative directory) for shared libraries with "gcc -Wl,-rpath,'$ORIGIN'". Using a shell script however makes it easier for people to understand and override this behaviour.

I think we should include just SDL and SDL_mixer libraries (each about 450kB), but not libogg, etc.

MIDI instruments

However, this is not enough to hear MIDI music. Android builds also need to include MIDI instrument patches, and I think we found some relatively small ones for that? It might be possible to include patches, but prefer any that are already installed on the system, but I'm guessing this would require patching SDL_mixer.

Testing compatibility

What's described in this section is checked automatically (by linux_portability_check.py) when compiling with scons portable=1. It's printed at the bottom of the build output and looks like

>>  ohrrpgce-custom requires glibc 2.14 (released 2011-06-01) and libs for gcc 3.4.0 (released before 2008-03-05)

To check a binary's dependencies, use objdump -p. E.g., for a x86 build:

> objdump -p ohrrpgce-game 

ohrrpgce-game:     file format elf32-i386

Program Header:
    PHDR off    0x00000034 vaddr 0x08048034 paddr 0x08048034 align 2**2
         filesz 0x00000100 memsz 0x00000100 flags r-x
  INTERP off    0x00000134 vaddr 0x08048134 paddr 0x08048134 align 2**0
         filesz 0x00000013 memsz 0x00000013 flags r--
    LOAD off    0x00000000 vaddr 0x08048000 paddr 0x08048000 align 2**12
         filesz 0x0020460a memsz 0x0020460a flags r-x
    LOAD off    0x0020460c vaddr 0x0824d60c paddr 0x0824d60c align 2**12
         filesz 0x00008ccc memsz 0x000321e4 flags rw-
 DYNAMIC off    0x00204698 vaddr 0x0824d698 paddr 0x0824d698 align 2**2
         filesz 0x00000160 memsz 0x00000160 flags rw-
    NOTE off    0x00000148 vaddr 0x08048148 paddr 0x08048148 align 2**2
         filesz 0x00000044 memsz 0x00000044 flags r--
EH_FRAME off    0x002021ec vaddr 0x0824a1ec paddr 0x0824a1ec align 2**2
         filesz 0x000004dc memsz 0x000004dc flags r--
   STACK off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**2
         filesz 0x00000000 memsz 0x00000000 flags rwx

Dynamic Section:
  NEEDED               libncurses.so.5
  NEEDED               libtinfo.so.5
  NEEDED               libpthread.so.0
  NEEDED               libSDL-1.2.so.0
  NEEDED               libSDL_mixer-1.2.so.0
  NEEDED               libX11.so.6
  NEEDED               libXext.so.6
  NEEDED               libXpm.so.4
  NEEDED               libXrandr.so.2
  NEEDED               libXrender.so.1
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  NEEDED               libdl.so.2
  INIT                 0x0804c304
  FINI                 0x0823133c
  INIT_ARRAY           0x0824d60c
  INIT_ARRAYSZ         0x0000006c
  FINI_ARRAY           0x0824d678
  FINI_ARRAYSZ         0x0000001c
  HASH                 0x0804818c
  GNU_HASH             0x08048b1c
  STRTAB               0x0804a188
  SYMTAB               0x08048bd8
  STRSZ                0x000012a3
  SYMENT               0x00000010
  DEBUG                0x00000000
  PLTGOT               0x0824d800
  PLTRELSZ             0x00000a48
  PLTREL               0x00000011
  JMPREL               0x0804b8bc
  REL                  0x0804b874
  RELSZ                0x00000048
  RELENT               0x00000008
  VERNEED              0x0804b6e4
  VERNEEDNUM           0x00000006
  VERSYM               0x0804b42c

Version References:
  required from libgcc_s.so.1:
    0x0b792650 0x00 20 GCC_3.0
    0x0d696910 0x00 15 GLIBC_2.0
  required from libdl.so.2:
    0x0d696910 0x00 18 GLIBC_2.0
    0x0d696911 0x00 12 GLIBC_2.1
  required from libm.so.6:
    0x0d696911 0x00 19 GLIBC_2.1
    0x0d696910 0x00 10 GLIBC_2.0
  required from libstdc++.so.6:
    0x056bafd3 0x00 14 CXXABI_1.3
    0x0297f865 0x00 08 GLIBCXX_3.4.15
    0x08922974 0x00 06 GLIBCXX_3.4
  required from libc.so.6:
    0x09691f73 0x00 17 GLIBC_2.1.3
    0x0d696912 0x00 16 GLIBC_2.2
    0x0d696913 0x00 11 GLIBC_2.3
    0x0d696910 0x00 07 GLIBC_2.0
    0x09691a73 0x00 05 GLIBC_2.2.3
    0x0d696911 0x00 03 GLIBC_2.1
  required from libpthread.so.0:
    0x0d696912 0x00 13 GLIBC_2.2
    0x0d696911 0x00 09 GLIBC_2.1
    0x0d696910 0x00 04 GLIBC_2.0
    0x09691972 0x00 02 GLIBC_2.3.2

See [1] for making sense of this. It seems this binary requires glibc 2.3.2+, which was released 2003-02-28. libstdc++.so.6 was first introduced in GCC 3.4.0 GLIBCXX_3.4.15 is GCC 4.6.0+, which was released 20110325. For example Ubuntu 11.10 defaults to installing gcc 4.6 (I don't know what ) but some conservative distros released in 2012 may not support this binary. I guess at present (2016) requiring a 2012 release of a linux distro is acceptable.

Comparing the above to a binary built on slackware64-current with a very recent glibc, the version requirements are identical except that libtinfo.so.5 isn't required and there are no GLIBC_* tags at all for libdl.

Finding and replacing a function requiring a recent library

Once you know that something compiled into a binary has an unwanted requirement, e.g. GLIBC_2.28, run objdump -T ohrrpgce-game | grep GLIBC_2.28 to find the cause.

Requirements for a recent glibc can be fixed by either removing all calls to the function, or wrapping it with a wrapper that calls a specific older version. See fcntl for an example - search for fcntl in SConscript and lib/glibc_compat.c.

Actually, instead of a wrapper, we could put an asm (".symver ...") line in a header and include it in any file calling that function. However this is not possible in general, because it might be called from libfb, or from C code generated by fbc.

Inspect the system library to find out what versioned symbols are available, to figure out the correct asm (".symver ...") line to add to lib/glibc_compat.c. For example:

> objdump -t /lib/lib.so.6 | grep fcntl
...
00101b70 g     F .text	00000215              fcntl@@GLIBC_2.28
00101d90 g     F .text	00000023              fcntl@GLIBC_2.0
...

This shows a new fcntl was added in glibc 2.28 and the older one is available with asm (".symver fcntl, fcntl@GLIBC_2.0");

32-bit and 64-bit builds will probably need different '.symver' lines.

For more information, see here.

Requirements for a recent libstdc++.so may be fixable by updating lib/stdc++compat.cpp by merging in changes from Mozilla: see stdc++compat.cpp changelog (warning, we have local changes).

See Also

• Compiling
• Linux library requirements
• The libstdc++ manual explains glibc versioning
• linux_portability_check.py utility (for glibc/stdc++/supc++/gcc_s only)