View on GitHub

iMatix Base

An archive and tribute to the iMatix Corporation base tooling.

Download this project as a .zip file Download this project as a tar.gz file

Reroute

27 Mar 2019 - cookrn

In the last example, the errors were still copied and pasted from the previous on-system attempts making them a bit confusing in our new Docker-based world.

Now, to build the project, there are 1.5 prerequisites: Docker and Docker Compose if your Docker distribution didn’t ship with it. Then, just run make to rebuild the dev container specified in the Dockerfile.

Having an easily rebuildable dev environment for this type of task has proved very helpful because of the amount of trial and error needed. On a laptop, it can be challenging to constantly download, compile, edit, reconfigure, etc… but with the container you get a fresh start at every rebuild.

So, in our new world, let’s reroute. The good news is, after some tinkering, the foreign and base2 tooling compiles. The other news is that it took some wild detours and digging into interesting and old code. I don’t have a background or familiarity with C infrastructure, so it took some searching. Here’s the highlights:

Uncovering Boom, GSL, and C

After lots of reading through the code in the various projects, there are a few tools that they all seem to depend on. Thankfully, they were already in a decent build state. Once I realized that, running a make install for them succeeded right away.

Most of the other projects in the repository had incomplete Makefiles or had partially generated source files. Playing with Boom a bit after it was installed, I realized that they weren’t meant to be configured with Boom and then used with Make. Instead, they were meant to be used fully with Boom. After poking around a bit more, the boom all command was most helpful at cleaning, generating, and compiling.

root@84e70aa81915:/usr/src/base# boom --help
'boom' prepares and executes the build process.

Usage: boom [OPTION]... [COMMAND]...

Configuration:
  -h, --help              Display this help information
  -v, --verbose           Trace build commands
  -V, --version           Display version information
  
Commands:
  configure               Configure project (generates boomake)
  version                 Show project version
  build                   Build project
  test                    Test project
  compile                 Build project w/o regenerating source
  install                 Install project
  clean                   Clean project
  distsrc                 Create project source package
  all                     clean, configure, build, test, install
  help                    Show list of allowed commands

Compiling APR

We left off in the last post being stuck unable to compile the base2 tooling because the compiler was unable to locate the base_apr.h header file. Since we can see find that file in the foreign libraries, let’s try to compile them instead. Foreign is structured similar to other iMatix libraries with five sub projects: C, APR, PCRE, ZLIB, and Tests. These can be seen as directories within the project but are also specified in the project.pdl file.

The first challenge in compiling was around libtool. The APR wrapper library has two vendored sets of source for libapr and libaprutil. Both were many versions behind what is now current and don’t work with the version of libtool that now ships with Debian Stretch. The approach that ended up working was to upgrade both libraries to the current stable versions. That brought libapr up from 1.3.3 to 1.6.5 and libaprutil up from 1.3.4 to 1.6.1. While fetching the updated sources and attempting to build them a few additional tweaks had to be made.

First, based on the sh code in APR’s project.pdl, it looks like previously the library had been setup to build statically. The new sources needed to be run through the GNU Autotools suite so that they were able to compile in the Dockerized environment. This took us into edits of APR’s project.pdl to add a ./buildconf step prior to running ./configure.

Next, during the ./configure step, there were a few options around the static build that needed removed: --without-libtool and --disable-shared. This allowed both libapr to compile successfully, but libaprutil still failed due to missing headers for a library called Expat. Expat is a C library for streaming XML parsing that was previously packaged with libaprutil but now is a separate dependency. Following standard GNU installation procedures with Expat placed the required dependencies in /usr/local and then allowed libaprutil to compile as well.

Lastly, the wrapper still failed due to attempting to use the ranlib tool on Libtool files instead of on ar archives. I’m not quite sure why the existing scripts would have been copying Libtool .la files to be named as .a library archives, but as best I could tell it was only a bug. Copying the appropriate .a files instead and updating the build scripts appropriately allowed all of the Foreign package to build.

Where has APR_SO_ERROR gone?

Once moving onto compiling the Base2 tools, things went fairly smooth with Foreign being already compiled and installed. The only build error that occurred was around the symbol named APR_SO_ERROR. Digging into mid-00s APR history, it seems to be a short lived symbol for fetching options from a socket. As none of the current symbols seem to be replacements, I ended up replacing the calls to apr_socket_opt_get using the error symbol with calls to apr_get_netos_error. The replacement function returns the last socket error as a known APR status, which should be of the expected value type, but given how no specific socket is specified, I’m not sure if this will introduce bugs. After the last fix mentioned below, it did allow all of Base2 to compile though.

Gurudoc

One Base2 tool had a PDL declaration specifying a documentation set for PAL which was supposed to be compiled. Instead of getting Base1 or Gurudoc to compile, I simply commented out the directive for now to proceed. If it weren’t documentation, it could have turned into another rabbit hole.

Conclusion

Now that the toolchain is compiled, the goal is to start factoring out the libraries back into top-level source projects in this repository so that each can be utilized or maintained in a cleaner way. I’ll also attempt to start producing examples as well as documentation on how to leverage the packages, tools, and various XML dialects. Onwards and sideways!