IBM MQ MacOS Toolkit for ARM

MQ has had a developer’s toolkit for MacOS available for a while. Although MacOS is not a fully-supported platform for MQ, the toolkit allows developers to write and test applications on that system, using a client connection to a queue manager.

Apple have gone through one of their periodic hardware-replacement exercises. MacOS machines were based on the x64 architecture for a number of years, but new machines are built on Arm64 (aka Apple Silicon or M1) chips. And so right at the end of 2022 we released a version of the MQ toolkit that can run natively on Arm64 systems. Doing this had some unique challenges, different from ports of the MQ codebase to other machines.

Dealing with two architectures

Because of the mixture of hardware architectures that will exist for some considerable time, Apple decided on a two-fold migration strategy:

1. An emulator allows x64 code to run on Arm hardware. This emulator, known as Rosetta, means that existing programs do not need recompilation. They run, albeit with some overhead, transparently.
2. Binary programs can contain both x64 and Arm instructions, with the “correct” set of code chosen to execute at runtime. This means that a single program can be distributed and run on both types of hardware with full native performance.

The availablity of (1) meant that the existing MQ Developer Toolkit, which was a purely x64 package, could continue in use on the newer machines. We published an article on setting up a Rosetta environment and running there.

While not ideal, this allows you to develop and test applications. But any programs you write using it would themselves have to be x64 code. That’s because the system chooses the effective architecture when the program starts, and either executes it natively or under the Rosetta emulator. You can’t switch a program’s mode half-way through its run. So your program has to use the same architecture as the available libraries.

For running queue managers on MacOS hardware, that the client programs can connect to, there is information on using virtual Linux machines. Right now, there’s not an Arm64 image available to go into a VM, but that might change. Until then it is possible to use the Rosetta emulator to run x64 images.

Universal Binaries

What we decided to do was to create a “universal” MacOS Toolkit. That means having a single copy of the programs and libraries, with both sets of instructions embedded. You only need to download one package file regardless of the machine you want to develop on. And it allows you to develop universal binaries for your own applications.

Example

As a very simple example of a universal program, this shows a trivial C program as we compile it in 3 ways – once for each explicit architecture, and then for a combined output. The arch flags to the compiler tell it what to build for. The file command reports on what kind of program is in each file.

You can see that the merged file is larger than the sum of the individual binaries; this is not always the case as there will be fixed overheads that manage the partitions regardless of the original size. But a reasonable rule-of-thumb was that the merged files were twice the size of a single architecture.

% cc -o hellox64   -arch x86_64 hello.c
% cc -o helloarm64 -arch arm64  hello.c
% cc -o hellouni   -arch x86_64 -arch arm64 hello.c
% file *
hello.c:     c program text, ASCII text
helloarm64:  Mach-O 64-bit executable arm64
hellouni: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64Mach-O 64-bit executable arm64]
helloarmuni (for architecture x86_64):  Mach-O 64-bit executable x86_64
helloarmuni (for architecture arm64):   Mach-O 64-bit executable arm64
hellox64:    Mach-O 64-bit executable x86_64
% ls -l
total 416
-rw-r--r--  1 metaylor  staff      73 30 Jan 10:25 hello.c
-rwxr-xr-x  1 metaylor  staff   33431 30 Jan 10:27 helloarm64
-rwxr-xr-x  1 metaylor  staff  115352 30 Jan 10:27 hellouni
-rwxr-xr-x  1 metaylor  staff   49424 30 Jan 10:27 hellox64

Porting MQ to be a universal package

As always, the thing that took the most time before we could work on the project was actually getting hold of suitable hardware. There’s the usual financial approval processes, ordering the kit, getting it installed in the datacentre, installing suitable development tools such as compilers and so on.

Aside: I’m reminded of an old IBM document that defined The Process for developing software products. It had all the pieces from initial concept to managing end-of-life, including testing, documentation and maintenance activites. Only one tiny box in the middle was “write code”. While you might express it differently today, such as agile iterations, it still feels fundamentally true.

Once the machines came available for developers to work on, the coding exercise was MOSTLY a matter of adding appropriate -arch flags for building the libraries and commands. But there was one unique challenge, that shows up in how the Toolkit deals with TLS connections to the queue manager.

On most platforms, MQ’s TLS implementation comes from the gskit component. That’s an IBM-developed package made available for bundling within a number of products. They have a version built for MacOS on x64 hardware that we could epxloit in the original MQ Toolkit. But there is (currently) no gskit for MacOS Arm64.

Because we are building universal packages, everything has to compile and link in both modes. How could we compile the TLS piece and have it work?

The answer is dynamic switching

The answer turned out to be a combination of compile-time flags, runtime checks, dynamic loading with dlopen, and using OpenSSL for the Arm64 execution while keeping gskitin the x64 leg and for other platforms. And writing the equivalent layer in the communications component for calling the OpenSSL routines. While similar, it is not a simple or direct mapping between the TLS providers.

The code now has blocks that look something like this:

#ifdef MACOS
if (runningOnArm())
  openSslFunction();
else // Note how the 'else' is inside the ifdef bracket
#endif
gskitFunction();

with one thought being that if/when there is a gskit for MacOS Arm systems, then it should be easy to pull out the OpenSSL unique pieces. We also play a couple of tricks in the code so that a couple of components get dummy implementations for the Arm portion. There is just enough to allow compile/link to work even if the pieces never actually run. Commands like lipo let us manipulate libraries to merge different architectures into a single file.

OpenSSL differences

Using OpenSSL for TLS and related activity is interesting because it has some fundamentally different behaviour than gskit in a few places:

• There is no “stash” file for keystore passwords
• Keystores for personal certificates and truststores for signer information have separate definitions; they commonly sit in different files.
• The keystore and truststore are different formats than the kdb normally used by gskit The PEM and P12 formats are typically in use here.

We chose to make the minimum changes in interfaces to allow use of OpenSSL for most development tasks, without providing a complete equivalent for all circumstances. The client package would not be appropriate for some production-quality environments, even if it were formally supported.

So, for example, there is no ability to use FIPS-certified algorithms or most CRL verification. We do make some assumptions about the names of files to simplify configuration. We considered using OpenSSL for both hardware variants but that would introduce incompatibilities with any existing applications.

There is also no support for AMS in the Arm implementation. It would be technically possible, with the same kind of runtime switching as the TLS branches, but was more work than justified for now.

The README in the installation package shows how to set additional environment variables, and which files to set up for the different purposes. It also has examples of the openssl commands to manage the stores and certificates. There are also some examples in the dev-patterns repository, to help with the slightly different configuration.

We hope that these mechanisms are good enough to allow applications to work on this platform for now. The MQI, for example, allows you to specify the name of a keystore but not a separate truststore. The assumption there is that they are the same file. While this is not necessarily true with OpenSSL, the rules in the README about their relationships should be adequate. And these rules do not need to be permanent if a gskit package comes available for us to exploit in future.

Summary

Working on this took me back to my earliest days in MQ. My original job was to port MQ to a range of non-IBM platforms. And the MacOS activity fits nicely into that. Although there are things I found I didn’t like about the MacOS environment (like any other system) it was still fun to get to work on yet another machine.

You can find the MQ Developer Toolkit for MacOS linked from here. It’s one more part of trying to make it easy for developers to work with MQ.

This post was last updated on February 13th, 2023 at 04:50 pm