A long-delayed Debian PowerPC SPE update

Unfortunately, several internal projects and some unexpected GCC and kernel issues have been blocking progress on the Debian PowerPC SPE port over the last few weeks.

Aside from some internal projects that I can't talk about on this blog, I spent the last few weeks rebasing our board support code onto Linux kernel v3.2-rc1 (now -rc3).  I also had to update several miscellaneous architecture code cleanup patches and fix a few minor bugs introduced upstream since my last rebase.

As far as the GCC compiler issues go, Alan Modra saved the day again and with the new non-crashing kernel I was able to test out his patches for GCC PR target/50906.  Several testcases fixed... Huzzah!

In the next few days I hope to actually get the buildd started up as I wanted to do a few weeks ago.

Cheers,
Kyle Moffett

Debian multiarch is cool, except when it breaks GCC

Ugh...

The last few days have been frustrating.  GCC does not have a nice build-system to play with, and encountering a build error 6 hours in to a 7-hour build really sucks.  Even worse is GCJ (the GCC Java compiler and runtime) failing 6 hours into its 7-hour build with an error that you have to go back and redo the 8-hour GCC build to fix.

I'm afraid I'm a bit burned out from all that, so all I have for you today is a few links:

• An additional e500 libffi patch: #647288
• A totally-untested GCC/GCJ patch to fix Debian multiarch directories: #647324
• Some PowerPC MPIC patches: [RFC PATCH 00/10] powerpc/mpic: General cleanup patch series

Enjoy!

Cheers,

Kyle Moffett

Debian PowerPC e500v2 port, part 8

This is part of the Debian PowerPC e500 porting effort, the series on my blog starts at: "How to bootstrap a new Debian port"

Sorry that it's been so long since the last post... I was able to get my board booted from NFS, but after that things got complicated.

I spent quite a while trying to re-cross-bootstrap more packages; but GCC 4.6 refused to build in a reverse-cross configuration (IE: build a compiler on amd64 that runs on powerpcspe and targets powerpcspe).

Unfortunately, there's a bit of a nasty dependency loop that prevented me from doing a native build either.  In order to build a new GCC 4.6 with multiarch support, I needed to install a new multiarch libc6... except that multiarch libc6 "Breaks" my old GCC 4.4 (which doesn't have multiarch).

I ended up doing a native build of a new multiarch libc6 on my NFS-booted system and forcibly installing that (which leaves my GCC 4.4 broken).  Thankfully libc6 creates a file in "/etc/ld.so.conf.d/" that GCC uses to help find libraries and I could work around the runtime crt*.o objects like this:

  # cd /usr/lib
  # for i in powerpc-linux-gnuspe/*; do ln -s "$i"; done

With that I was able to build a new GCC 4.6.  Unfortunately, I still had an old gcc-defaults package (depending on gcc-4.4) and the new gcc-defaults package would not build unless I could install gcc-4.6 and gcj-jdk and a half-dozen other things I don't have.

Thankfully, with a couple more quick compatibility symlinks I could just remove "gcc" and "g++" and leave "build-essential" technically broken but have a working build environment:

  # cd /usr/bin
  # for i in *g{cc,++}-4.6; do ln -s "$i" "${i%-4.6}"; done

After that, I had a new libc6 installed, along with "multiarch-support"!  Finally!  Woohoo!!!

I upgraded dpkg, apt, aptitude, and about a half-dozen other libraries and tools that all need multiarch these days...

Now I need to figure out what I need to install/build/rebuild in order to make sbuild work again, then I should be able to get a new "buildd" server going.

And of course once I have one going I have enough spare hardware to start about 18 of them.

I'll hopefully post again later today with more progress.

Cheers,
Kyle Moffett

Debian PowerPC e500v2 port, part 3

This is part of the Debian PowerPC e500 porting effort, the series on my blog starts at: "How to bootstrap a new Debian port"

Before we can actually start cross-compiling packages, we need to create a quick set of symlinks so that they can find our compiler.  The "gcc-4.6" packages we built in the previous steps created a set of tools named like this:

powerpc-linux-gnuspe-gcc-4.6
powerpc-linux-gnuspe-g++-4.6

Unfortunately, by default autoconf won't actually find those, and the "gcc-defaults" package does not have a cross-compiler mode (although it can be cross-"compiled", as we will need to do later).  So symlink the relevant binaries to versions without the "-4.6" at the end:

$ cd /usr/bin
$ for i in powerpc-linux-gnuspe-*-4.6; do sudo ln -s "$i" "${i%-4.6}"; done

So the next step is to make sure I have all of the Essential and Build-Essential packages up-to-date for installation into a usable root filesystem for building everything else.  That means I need to figure out a list of those packages first.

To make this easier, we can use the "grep-dctrl" tool to look through the APT package cache on our favorite Debian box.  MAKE SURE YOU USE A BOX WITH DEBIAN TESTING/UNSTABLE.  If you don't then you won't get the right package lists for your new port.

Let's first start with the "Essential: yes" packages:

$ sudo apt-get install grep-dctrl
$ grep-available -Xn -s Package -FEssential 'yes' >new-pkgs.txt

Now add "build-essential" to the list and get ready to list dependencies:

$ echo 'build-essential' >>new-pkgs.txt
$ : >all-pkgs.txt

Then we need to recursively follow dependencies.  It's all basically done with the following scriptlet:

$ while [ -s new-pkgs.txt ]; do \
      cat new-pkgs.txt >>all-pkgs.txt; \
      for i in $(cat new-pkgs.txt); do \
          grep-available -PXn -s Depends,PreDepends "$i"; \
      done | sed -e 's/([^)]\+)//g' -e 's/, /\n/g' \
          -e 's/ *|[^\n]*//g' | grep . | sort | uniq >dep-pkgs.txt; \
      cat dep-pkgs.txt all-pkgs.txt all-pkgs.txt \
          | sort | uniq -u >new-pkgs.txt; \
  done

Once that's done running (it should take ~30 seconds), you will have the complete binary package list in all-pkgs.txt.  Now we need to find all the source packages necessary to build those binary packages:

$ for i in $(cat all-pkgs.txt); do \
      grep-available -PXn -sSource:Package "$i"; \
  done | sed -e 's/ (.*//' | sort | uniq >all-srcs.txt

Go ahead and use the resulting all-srcs.txt file to make yourself a checklist, because the rest of this is going to be a painfully long and manual process.  I've included my list below, wrapped onto a single line to minimize wasted space:

base-files base-passwd bash binutils build-essential bzip2 coreutils dash db debconf debianutils diffutils dpkg e2fsprogs eglibc findutils gcc-4.6 gcc-defaults gdbm gmp grep gzip hostname insserv libselinux libsepol libtimedate-perl linux-2.6 lsb make-dfsg mpclib mpfr4 ncurses patch perl sed sensible-utils shadow sysvinit tar tzdata util-linux xz-utils zlib

Not too unreasonable, right?  What you will rapidly realize is that build-dependencies are the key, and we have not actually tried to list those out yet.  The biggest problem tends to be docs-generation tools (EG: LaTeX, Doxygen, etc) which are entirely unnecessary for bootstrap purposes but have lots of dependencies of their own (EG: xorg, qt4, etc).

So let's pick one of these and get started.  I've included the precise versions I used below, but you can just leave off the "=1.4-1" bit to get the latest unstable version assuming your APT is set up correctly.

$ mkdir gzip && cd gzip && apt-get source gzip=1.4-1

First you should check the "Build-Depends" field in the "debian/control" file, to make sure you have all of the necessary dependencies.  Note that some dependencies (automake, autoconf, texinfo, etc) should be satisfied by your build-system architecture (EG: amd64), while others (libc6, etc) should be satisfied by "-cross" packages for your target architecture.

$ grep Build-Depends: gzip-1.4/debian/rules
Build-Depends: debhelper (>= 5), texinfo, autoconf, automake, autotools-dev

 Ok, everything looks good there.  Now check the "debian/rules" file to check for the following few issues:

1. The rules should reference DEB_HOST_GNU_TYPE to talk about the target system.  If it seems to use DEB_BUILD_GNU_TYPE for that then it's probably broken.
2. Any testsuites should be disabled when DEB_HOST_GNU_TYPE != DEB_BUILD_GNU_TYPE, otherwise the build will likely fail.  You may be able to work around these kinds of issues by setting DEB_BUILD_OPTIONS=nocheck.

There are certainly lots of other ways that packages can fail to cross-compile correctly, but if it uses autoconf/automake and either cdbs or debhelper then it should be mostly OK.  The next step is to try a cross-compile:

$ ( cd gzip-1.4 && dpkg-buildpackage -a"${MYARCH}" -us -uc -B ) 2>&1 | tee build-gzip.log

Even if the build finishes successfully, you should use "lintian" and "dpkg-deb -c" to verify that the package files look correct.  If you have any errors or unexpected package contents then you have some serious work to do to figure out why and fix the package.  In the "gzip" case I identified a minor bug (Debian bug #644785) in the packaging which resulted in one particular setting coming from the build-system instead of from the host.

NOTE: If you have to make any changes at all, make sure you generate a nice patch and submit it back to Debian with the "reportbug" tool.

Once you have the package built, you should check to see if it resulted in any library packages (lib* and lib*-dev).  If so, you should try to get those installed with "dpkg-cross" the same way that the eglibc ones were installed previously, as you may need them to satisfy other cross-build dependencies later on.

That's it for tonight!

Cheers,
Kyle Moffett

Debian PowerPC e500v2 port, part 2

Well, I've been working again on the Debian powerpcspe port and there is a lot more progress to be had! (This series starts at: "How to bootstrap a new Debian port")

I left off last time after EGLIBC failed to build with errors about undefined references to "__stack_chk_guard".  It turns out the problem is that EGLIBC does not correctly detect the presence of support for the "-fstack-protector" GCC option, as per Debian bug #644662.  The fix is to download and apply this patch, which fixes the configure script test.

The second issue I ran into is that dpkg-gensymbols is not very good at finding all the relevant multi-arch libraries while trying to build packages, as per Debian bug #595144.  The fix there is an existing patch to the "dpkg" source package by Steve Langasek; it changes the "libdpkg-perl" binary package to fix the dpkg-gensymbols tool.

As a side note, it turns out that my previous gcc-4.6-cross-stage1-dep-fix-v2.patch broke non-bootstrap-stage builds of GCC.  As a result, I've posted a new version to Debian bug #644439; this one should handle native, cross, and cross-bootstrap builds fine although it's still undergoing compile tests here.

The final issue with the EGLIBC build is that it tries to build the "locales-all" package by executing the built binaries, which of course fails for cross-compiled EGLIBC (see Debian bug #644771).  The fix is trivial, simply don't build "locales-all" when cross-compiling, found in this patch.

So before we can retry building EGLIBC, we need to build an updated libdpkg-perl for our build host:

Step 6.1 - Building and installing a patched libdpkg-perl:

$ sudo apt-get build-dep dpkg=1.16.1
$ sudo apt-get source dpkg=1.16.1
$ curl -o 'Dpkg-Shlibs.pm-multiarch-search-paths.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=595144;msg=37;att=1'
$ patch -d dpkg-1.16.1 -p1 <Dpkg-Shlibs.pm-multiarch-search-paths.patch
$ ( cd dpkg-1.16.1 && dpkg-buildpackage -b -us -uc ) 2>&1 | tee dpkg.log
$ sudo dpkg -i libdpkg-perl_1.16.1_all.deb

Going back to the EGLIBC commands I listed previously for step 6, here is an updated version:

Step 6.2 - Commands to build a stage2 EGLIBC:

$ sudo apt-get build-dep eglibc=2.13-21
$ apt-get source eglibc=2.13-21
$ curl -o 'eglibc-2.13-cross-stage1-support.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644546;msg=5;att=1'
$ curl -o 'eglibc-fix-ssp-detection.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644662;msg=5;att=1'
$ curl -o 'eglibc-disable-cross-locales-all.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644771;msg=5;att=1'
$ patch -d eglibc-2.13 -p1 <eglibc-2.13-cross-stage1-support.patch
$ patch -d eglibc-2.13 -p1 <eglibc-fix-ssp-detection.patch
$ patch -d eglibc-2.13 -p1 <eglibc-disable-cross-locales-all.patch... MAKE ANY OTHER CHANGES TO EGLIBC HERE ...
$ ( cd eglibc-2.13 && DEB_GCC_VERSION=-4.6 \
      dpkg-buildpackage -a"${MYARCH}" -B -us -uc ) 2>&1 \
      | tee eglibc-stage2.log

Once the build finishes, it will have produced a bunch of deb files, but you only actually need the libc6 and libc6-dev packages.  As before, we will eventually be able to install foreign architecture packages but for now we have to forcibly dpkg-cross them (with a few ignored dependencies).

Step 6.3 - Commands to install the stage2 EGLIBC:

$ sudo dpkg-cross -M -a "${MYARCH}" -X libc-bin -X libc-dev-bin -i \
      "libc6_2.13-21_${MYARCH}.deb" 
"libc6-dev_2.13-21_${MYARCH}.deb"

After this completes we have a fully usable C library for our target architecture, and it's time for the final GCC cross-compiler build with all features enabled.

Step 7.1 - Commands to build a final full-featured GCC:

$ sudo apt-get build-dep gcc-4.6=4.6.1-13
$ apt-get source gcc-4.6=4.6.1-13
... MAKE ANY CHANGES TO GCC HERE ...
$ ( cd gcc-4.6-4.6.1 && DEB_GCC_TARGET="${MYARCH}" DEB_STAGE=stage2 \
  dpkg-buildpackage -b -us -uc ) 2>&1 | tee gcc-4.6-stage2.log

The final GCC build leaves about 23 packages lying around in the directory, and you should probably install all of them.

Step 7.2 - Command to install the final GCC:

$ sudo dpkg -i $(dcmd --deb gcc-4.6_4.6.1-13_*.changes)

NOTE: DO NOT UPLOAD ANY OF THE PACKAGES YOU BUILT!!!  They intentionally deviate from standard in various ways due to the cross-building process and you should wait until you can rebuild them natively before uploading.

Cheers,
Kyle Moffett

How to bootstrap a new Debian port

Ok... This is is kind of a funny way to start a blog, but it's the first really serious content that I felt like posting, so here goes!

I've been working on a Debian port for the e500v2 architecture "powerpcspe", which has a rather odd way of doing floating point compared to normal PowerPC systems.

Just recently I've submitted a bunch of Debian bugs for different pieces of the bootstrap process that were rather tedious, in the hope that someone doing a new architecture port in the future might benefit. To have a central place for the entire procedure (ideally one that Google can find easily), I'm posting it all here.


The first step is to figure out your architecture triplet and get that added to dpkg as a new target.  For example, the "powerpcspe" architecture uses the target triplet "powerpc-linux-gnuspe".

Temporarily you can modify the following files to get a local system to recognize a new architecture, but before you ship you should get the dpkg maintainers to add it upstream:

/usr/share/dpkg/archtable
/usr/share/dpkg/cputable
/usr/share/dpkg/triplettable

So let's set these shell variables:

$ unset MYARCH MYTRIPLET TARGET
$ MYARCH=powerpcspe
$ MYTRIPLET="$(dpkg-architecture -a"$MYARCH" -qDEB_HOST_GNU_TYPE)"

NOTE: You will see a warning that your GNU system type does not match your GCC system type.  This is OK, and part of cross-compiling.

Then you need to bootstrap a cross-compiler toolchain with that triplet:

1. Build binutils (assembler, linker, etc)
2. Perform a headers-only Linux Kernel build
3. Build a minimal C-only GCC using just the target's kernel headers
4. Build headers and run-time objects using the minimal GCC
5. Build a secondary C-only GCC using the target's EGLIBC headers
6. Build a full EGLIBC using the secondary C-only GCC
7. Build a final GCC using the full EGLIBC

Later, if you need to rebuild the tools you can skip steps 3-5, as you will
already have a usable target GLIBC and target headers to start from.

Step 1 - Commands to build and install a cross-binutils:

$ sudo apt-get build-dep binutils=2.21.90.20111004-1
$ apt-get source binutils=2.21.90.20111004-1
... MAKE ANY CHANGES TO BINUTILS HERE ...
$ ( cd binutils-2.21.90.20111004 && TARGET="${MYTRIPLET}" \
          dpkg-buildpackage -b -us -uc ) 2>&1 | tee binutils-build.log
$ sudo dpkg -i "binutils-${MYTRIPLET}_2.21.90.20111004-1_amd64.deb"

Step 2 - Commands to download and modify Linux Kernel headers:

$ sudo apt-get build-dep linux-2.6=3.0.0-3
$ apt-get source linux-2.6=3.0.0-3
$ pushd linux-2.6-3.0.0
$ vim debian/config/defines ## Add "${MYARCH}" to list
$ cp -a debian/config/{powerpc,"${MYARCH}"}
$ vim debian/config/"${MYARCH}"/* ## Make it work
... MAKE ANY OTHER CHANGES TO THE LINUX KERNEL HERE ...
## THE NEXT COMMAND FAILS WITH A WARNING MESSAGE, THAT IS OK!!!
$ dpkg-buildpackage -a"${MYARCH}" -b -us -uc -T"debian/control-real"

Step 2.1 - You will need to edit the "debian/rules
" Makefile in the Debianized linux-2.6 sources and append the following text at the end:

.PHONY: FORCE
binary-%: FORCE
dh_testdir
$(MAKE) -f debian/rules.gen binary-$*_$(DEB_HOST_ARCH)

Step 2.2 - Commands to actually build Linux Kernel headers:

$ dpkg-buildpackage -a"${MYARCH}" -b -us -uc -Tsource
$ dpkg-buildpackage -a"${MYARCH}" -b -us -uc -Tbinary-libc-dev \
          --as-root -rfakeroot
$ popd

FIXME: Right now 2011/10/05 dpkg in testing/unstable does not support installing packages from foreign architectures, so we have to run the following command to forcibly convert the linux-libc-dev package with dpkg-cross.  Eventually "foreign-architecture ${MYARCH}" in your /etc/dpkg/dpkg.cfg file will be enough to make "dpkg -i" just work.

Step 2.3 - Command to install the Linux Kernel Headers:

$ sudo dpkg-cross -M -a "${MYARCH}" -i \
        "linux-libc-dev_3.0.0-3_${MYARCH}.deb"

Alternate Step 2.3 (UNTESTED) - Eventually (with full multiarch dpkg, already in some Ubuntu) you will just use this command:

$ sudo dpkg -i "linux-libc-dev_3.0.0-3_${MYARCH}.deb"

FIXME: The GCC stage1 build has a bug right now (Debian Bug #644439) which causes the gcc-4.6-powerpc-linux-gnuspe package to falsely depend on a libgcc-powerpcspe-cross package which is not part of a stage1 build.

Until that bug is fixed upstream, you will need to apply this patch to the GCC sources or it won't install (included in commands below)  (UPDATE!)  The old patch breaks native builds, use this one instead.

Step 3 - Commands to build a minimal stage1 GCC:

$ sudo apt-get build-dep gcc-4.6=4.6.1-13
$ apt-get source gcc-4.6=4.6.1-13
$ curl -o 'gcc-4.6-cross-stage1-dep-fix-v2.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644439;msg=30;att=1'
$ patch -d gcc-4.6-4.6.1 -p1 <gcc-4.6-cross-stage1-dep-fix-v3.patch
... MAKE ANY OTHER CHANGES TO GCC HERE ...
$ ( cd gcc-4.6-4.6.1 && DEB_GCC_TARGET="${MYARCH}" DEB_STAGE=stage1 \
    dpkg-buildpackage -b -us -uc ) 2>&1 | tee gcc-4.6-stage1.log
$ sudo dpkg -i "cpp-4.6-${MYTRIPLET}_4.6.1-13_amd64.deb"
$ sudo dpkg -i "gcc-4.6-${MYTRIPLET}_4.6.1-13_amd64.deb"

FIXME: The EGLIBC debian package does not actually have a way to build this as a part of a minimal package file (see Debian Bug #644546), so we need to apply this patch that lets us build stage1 packages.

Step 4 - Commands to build EGLIBC headers and run-time objects

$ sudo apt-get build-dep eglibc=2.13-21
$ apt-get source eglibc=2.13-21
$ curl -o 'eglibc-2.13-cross-stage1-support.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644546;msg=5;att=1'
$ patch -d eglibc-2.13 -p1 <eglibc-2.13-cross-stage1-support.patch
... MAKE ANY OTHER CHANGES TO GLIBC HERE ...
$ ( cd eglibc-2.13 && DEB_GCC_VERSION=-4.6 DEB_STAGE=stage1 \
    dpkg-buildpackage -a"${MYARCH}" -b -us -uc ) 2>&1 \
    | tee eglibc-stage1.log

FIXME: Right now 2011/10/05 dpkg in testing/unstable does not support installing packages from foreign architectures, so we have to run the following command to forcibly convert the libc6-dev package with dpkg-cross.  Eventually "foreign-architecture ${MYARCH}" in your /etc/dpkg/dpkg.cfg file will be enough to make "dpkg -i" just work.

FIXME2: Additionally, this package currently has excess dependencies on nonexistent libc6 and libc-dev-bin packages.  Hopefully when the Debian EGLIBC maintainers merge the stage1-support patch, they will also fix the dependencies for stage1 packages.  Until then, the dpkg-cross command includes "-X libc6 -X libc-dev-bin".

Step 4.1 - Command to install the EGLIBC headers and run-time objects:

$ sudo dpkg-cross -M -a "${MYARCH}" -X libc6 -X libc-dev-bin \
-i "libc6-dev_2.13-21_${MYARCH}.deb"

Alternate Step 4.1 (UNTESTED, BROKEN) - Eventually (with full multiarch dpkg, and the dependencies fixed) you will just use this command:

$ sudo dpkg -i "libc6-dev_2.13-21_${MYARCH}.deb"

Step 5 - Commands to build and install a stage2 GCC:

$ sudo apt-get build-dep gcc-4.6=4.6.1-13
$ apt-get source gcc-4.6=4.6.1-13
... MAKE ANY CHANGES TO GCC HERE ...
$ ( cd gcc-4.6-4.6.1 && DEB_GCC_TARGET="${MYARCH}" DEB_STAGE=stage2 \
  dpkg-buildpackage -b -us -uc ) 2>&1 | tee gcc-4.6-stage2.log
$ sudo dpkg -i "gcc-4.6-${MYTRIPLET}-base_4.6.1-13_amd64.deb"
$ sudo dpkg -i "libgcc1-${MYARCH}-cross_4.6.1-13_all.deb"
$ sudo dpkg -i "libgcc1-dbg-${MYARCH}-cross_4.6.1-13_all.deb"
$ sudo dpkg -i "gcc-4.6-${MYTRIPLET}_4.6.1-13_amd64.deb"
$ sudo dpkg -i "cpp-4.6-${MYTRIPLET}_4.6.1-13_amd64.deb"

FIXME: The same EGLIBC patch as is used above still needs to be applied or the DEB_GCC_VERSION variable will not work (see Debian Bug #644546).

Step 6 - Commands to build a final EGLIBC (UPDATE: Known to be broken):

$ sudo apt-get build-dep eglibc=2.13-21
$ apt-get source eglibc=2.13-21
$ curl -o 'eglibc-2.13-cross-stage1-support.patch' \
  'http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644546;msg=5;att=1'
$ patch -d eglibc-2.13 -p1 <eglibc-2.13-cross-stage1-support.patch
... MAKE ANY OTHER CHANGES TO GLIBC HERE ...
$ ( cd eglibc-2.13 && DEB_GCC_VERSION=-4.6 \
      dpkg-buildpackage -a"${MYARCH}" -b -us -uc ) 2>&1 \
      | tee eglibc-final.log

FIXME:  Unfortunately I'm stuck here for today...  This EGLIBC build fails with errors about undefined references to "__stack_chk_guard", presumably because it's being built with a stripped-down GCC that does not have those features enabled.  I can only assume that either the stage2 GCC needs to have some additional features turned on or there needs to be an extra round of EGLIBC build after the full GCC is working.

(UPDATE: See "Debian PowerPC e500v2 port, part 2" for further progress.)

Cheers,

Kyle Moffett