Swimming against the tide

This is an old article of 2018 that I forgot to publish, about how argh-woops-dang the programming experience can be sometimes, maybe the title should have been swimming in bitumen.

HDF5 is an essential format for exchanging large chunks of data. I don't much like it. It's overly complex. But not having an interface to it is a severe handicap for a language aiming to be generalist, and crunching data is rather a generalist task these days.

Of course, I could be using Python like anyone else on earth, but I would prefer some more productive and lively environment. So a few months ago, I've started to write such an interface for exchanging data between HDF5 and Smalltalk (Visualworks for a beginning). I'll publish on public store when having minimal usable core features, and it will be licensed MIT. But it's not yet ready. It's an activity taken on my free time.

I've started development in windows because having a not-supported-by-apple library working on Macosx is rarely an "out of the box" experience. This week end, I decided to give it a kick and to switch back to my preferred platform.

Building the library with the cmake build configuration provided by the hdfgroup was fairly easy. Just follow the instructions from http://support.hdfgroup.org/HDF5/release/cmakebuild.html.... and produce what you don't need: static libraries.

Building the dynamic libraries has been more involved, because neither setting the STATIC_ONLY=NO nor NO_MAC_FORTRAN=YES in build-unix.sh did the trick, whatever the recommendations found in http://support.hdfgroup.org/HDF5/release/chgcmkbuild.html.
The worse is that I don't know why. Did I forget to remove the previous build?
It's certainly not the right way, but brute force hardcoding of set(ADD_BUILD_OPTIONS "${ADD_BUILD_OPTIONS} -DBUILD_SHARED_LIBS:BOOL=ON") in the various cmake files produced the libhdf5.dylib that I wanted...

Or not... These were x86_64 libraries.  With my dated Smalltalk environment, I wanted an i386, or better a fat 96bits universal version. Not a problem. As usual, Stackoverflow has THE solution: http://stackoverflow.com/questions/5334095/cmake-multiarchitecture-compilation. I stupidely passed option "-DCMAKE_OSX_ARCHITECTURES=x86_64;i386" to ctest, which was not the right incantation, and without any flashing illumination, but rather after laborious attempts, finally added it as build option inside CTestScript.cmake

set (BUILD_OPTIONS "${BUILD_OPTIONS} \"-DCMAKE_OSX_ARCHITECTURES=x86_64;i386\" "). 

This was the right syntax, but not the right way...
My move was anticipated for a long time by another chess player, see those lines in
 hdf5-1.10.1/config/cmake_ext_mod/ConfigureChecks.cmake:

if (APPLE)
  list (LENGTH CMAKE_OSX_ARCHITECTURES ARCH_LENGTH)
  if (ARCH_LENGTH GREATER 1)
    set (CMAKE_OSX_ARCHITECTURES "" CACHE STRING "" FORCE)
    message(FATAL_ERROR "Building Universal Binaries on OS X is NOT supported by the HDF5 project. This is"
    "due to technical reasons. The best approach would be build each architecture in separate directories"
    "and use the 'lipo' tool to combine them into a single executable or library. The 'CMAKE_OSX_ARCHITECTURES'"
    "variable has been set to a blank value which will build the default architecture for this system.")
  endif ()
  set (${HDF_PREFIX}_AC_APPLE_UNIVERSAL_BUILD 0)
endif ()

Ouch! I don't care much of the best approach. I just want something that works! 
And I don't even agree. The best approach is to follow the guidelines of the target platform. 
Everything else is swimming against the tide...
For the time being, it's me who is swimming against the tide, too far away from the mainstream.
Even if I don't build for universal but just i386 architecture, the option will be blanked.
So what are these technical reasons exactly? Where can I read about that?

Without a clue, I started to hack those lines if (ARCH_LENGTH GREATER 1000), and also  set (${HDF_PREFIX}_AC_APPLE_UNIVERSAL_BUILD 1) but of course, as advertised this did not work. OK, with -VV verbose verbose option, we know that the link failed because ZLib and SZip were compiled for x86_64 only too. Untar-ing the provided ZLib.tar.gz and SZip.tar.gz, modifying the ConfigureChecks.cmake inside which had the very same guard as above, re-taring-re-gzipping ... equally failed! Same problem, the external libraries were NOT compiled with universal support. Adding the DCMAKE_OSX_ARCHITECTURES=x86_64;i386 to various CMAKE_ARGS definitions that I found did not help either...

Hacking randomly like this is vain and probably doomed to fail, but how are you supposed to understand what happens and where it comes from exactly among those thousand lines scattered all around the .cmake files?

You think I'm exagerating? 

find . -name '*cmake*' | wc -l
-> 101

wc -l `find . -name '*cmake*'`
-> 23729

More than twenty thousand lines in a hundred files! With all that stuff, HDF5 should certainly compile on your android tablet and your connected fridge! But not on my Mac...

If you count the generated build directory, this roughly doubles.
But if you remove the duplicates (and blanks) it's only half:

cat `find . -name '*cmake*'` | sort |  uniq | wc -l
-> 12799

Believe me now?

Back to the technical reasons they spoke about, I now have a clue: could it be the choice of the right tools? Ugly scripts, ugly environment variables, ugly macros, duplicated code and no debugger for getting a chance to dive inside the machinery. Welcome in the 21st century.

That's what sucks with configure/cmake and all those meta levels that generates configuration files for yet another tool (make): you gradually loose control on the lower levels where things has to be solved. The problem is fairly simple: compile/link ZLib and SZip with the universal build flags. The solution of going across all these layers is really involved, even if I don't even want to make it work for others, but just for me!

I could have talked about Smalltalk and the implementation, but half the week-end is gone in smoke now without the slightest progress.