MQ application compatibility across a quarter century

I was working on something recently where I had to upgrade various components in the tooling. And I was getting more and more annoyed that the upgrades broke my existing programs and scripts. None of that was MQ’s fault and I’ll write more about the project once it’s available alongside the newly-announced MQ 9.3. [That article is now published here.] But it got me thinking about the efforts we’ve made to keep MQ application compatibility across its lifetime. I wanted to show how we’ve achieved that across a quarter century (and more). And how that has preserved the work that developers have put into their MQ programs. In particular, I want to see if old compiled programs can still work with a current queue manager.

Source compatibility

Most people probably understand how the MQ API makes it possible to add new function without fundamentally changing the interface. Using versioned structures means that new fields get added to a structure, and the queue manager code can use the version to decide whether or not to look at or fill in those fields. There are some aspects of the original API design which are hard or near-impossible to extend, and which I wish had been done differently. But the basic idea has held up very well.

It means that you can take application source code written many years ago, and recompile it on a new system, knowing it still builds. Not bumping the “default” version for a structure means that you get the same behaviour as before – you only have to move to an MQCNOv6 if you want to use the CcdtUrl attribute. And that requires that you use a particular version of MQ to recognise it. If you use an MQCNOv6 against an old queue manager, the application will fail. So you can continue to use an older structure version without changing the code as the defaults have not changed.

Application problems on migrating to new versions have typically happened when the app has not been written to expect new message formats or receiving newer error codes, or when MQ has tightened up its validation to catch applications that have not completely followed rules..

Binary compatibility

What I was more interested in for this exercise however was binary compatibility. Can I take an old compiled application program and still run it against today’s queue manager.

I visited the Hursley Museum to see if they could help. They could. Their catalogue listed a copy of MQSeries V2.2.1 for AIX. (Actually the first box they dug out from the shelves had an empty CD case, but they found another unopened package that did have the CD.) It’s not quite the oldest we ever did for AIX, which was Version 2.1, but I couldn’t locate that level on a medium that I could still read. Using AIX was ideal for this experiment because it’s a system I have access to and where we still develop MQ. But none of my machines have an 8mm or QIC tape drive attached these days.

The only other still-supported platforms from that time are z/OS and iSeries. All the other early platforms either died away or lost enough support that we don’t do current versions of MQ for them. Other operating systems such as Windows (starting with NT) and Linux came to the queue manager world later.

The box

Unboxing – reveals REAL BOOKS and lots of the dancers

I put the CD into the system and looked at the dates:

$ cd /cdrom
$ ls -al 
total 46300
-r-xr-xr-x. 1 root root  5683200 Oct 17  1995 dtextbrw.obj
-r-xr-xr-x. 1 root root 41728000 Oct 17  1995 mqm.obj
-r-xr-xr-x. 1 root root     5438 Oct 17  1995 .toc

I considered doing a “proper” install of the product, but that would mess up too much of my development environment. Being able to isolate MQ installations was a feature from many years later. And I wasn’t bothered about trying to run the queue manager code anyway. Noone should be running that kind of unsupported level. Though I would not be surprised to find there are still unlabelled boxes running a business-critical process hidden in a cupboard somewhere. Something to try another day, if I can find a disposable AIX image to play with.

It was the applications that were more interesting. Those are the things that you – the users – have developed across many years. And for which you may have lost the source by now, but the programs still have value.

The sample programs

So I extracted the sample programs from the installation image and put them in a local directory.

$ ls -l    
total 672
-rwxrwxrwx    1 metaylor system        10891 13 Nov 1995  amqsbcg
-rwxrwxrwx    1 metaylor system        14236 13 Nov 1995  amqscic0
-rwxrwxrwx    1 metaylor system        13654 13 Nov 1995  amqscic21
-rwxrwxrwx    1 metaylor system       135980 13 Nov 1995  amqsdlq
-rwxrwxrwx    1 metaylor system         7341 13 Nov 1995  amqsech
-rwxrwxrwx    1 metaylor system         7133 13 Nov 1995  amqsgbr
-rwxrwxrwx    1 metaylor system         6591 13 Nov 1995  amqsget
-rwxrwxrwx    1 metaylor system         6590 13 Nov 1995  amqsgetc
-rwxrwxrwx    1 metaylor system         9135 13 Nov 1995  amqsinq
-rwxrwxrwx    1 metaylor system         7741 13 Nov 1995  amqsput
-rwxrwxrwx    1 metaylor system         7740 13 Nov 1995  amqsputc
-rwxrwxrwx    1 metaylor system         9123 13 Nov 1995  amqsreq
-rwxrwxrwx    1 metaylor system         8907 13 Nov 1995  amqsset
-rwxrwxrwx    1 metaylor system         7785 13 Nov 1995  amqstrg
-rwxrwxrwx    1 metaylor system        30653 13 Nov 1995  amqsxa4x
-rwxrwxrwx    1 metaylor system        30971 13 Nov 1995  amqsxaex

This machine has MQ 9.3 installed as its primary version, so those will be the shared libraries (libmqm etc) that the sample programs try to load.

Trying the most basic experiment, of putting and getting messages:

$ echo "Hello from 1995" | ./amqsput Q V9300_A
Sample AMQSPUT0 start
target queue is Q
Sample AMQSPUT0 end
$ ./amqsget Q V9300_A                         
Sample AMQSGET0 start
message <Hello from 1995>
no more messages
Sample AMQSGET0 end

And doing the same again but using the client connectivity …

$ export MQSERVER="SYSTEM.DEF.SVRCONN/TCP/localhost(3820)"
$ echo "Hello from 1995 Client" | ./amqsputc Q V9300_A           
Sample AMQSPUT0 start
target queue is Q
Sample AMQSPUT0 end
$ ./amqsgetc Q V9300_A     
Sample AMQSGET0 start
message <Hello from 1995 Client>
no more messages
Sample AMQSGET0 end

The next experiment was to put a message using a current application program which sets the GroupId. Doing that requires an MQMDv2. The v2 structure was not available in the V2.2.1 days, so what would happen?

$ ./amqsbcg Q V9300_A                          
AMQSBCG0 - starts here
 
MQCONN to V9300_A
MQOPEN - 'Q'
MQGET of message number 1 
**** Message descriptor ****
StrucId  : 'MD  '  Version : 1
  Report   : 0  MsgType : 8
  Expiry   : -1  Feedback : 0
  Encoding : 273  CodedCharSetId : 819
  Format : 'MQHMDE  '
  Priority : 0  Persistence : 0
  MsgId : X'414D512056393330305F412020202020625930D040003A01'
  CorrelId : X'000000000000000000000000000000000000000000000000'
  BackoutCount : 0
  ReplyToQ       : '                                 '
  ReplyToQMgr    : 'V9300_A                          '
  ** Identity Context
  UserIdentifier : 'metaylor    '
  AccountingToken : 
   X'04363530350000000000000000000000000000000000000000000006'
  ApplIdentityData : '                               '
  ** Origin Context
  PutApplType    : '6'
  PutApplName    : 'amqsput                     '
  PutDate  : '20220422'    PutTime  : '06225952'
  ApplOriginData : '    '

****   Message      ****
length - 85 bytes
00: 4D44 4520 0000 0002 0000 0048 0000 0222 'MDE .......H...'
10: 0000 04B8 4D51 5354 5220 2020 0000 0000 '....MQSTR  ....'
20: 2021 2223 2425 2627 2829 2A2B 2C2D 2E2F ' !"#$%&'()*+,-./'
30: 3031 3233 3435 3637 0000 0001 0000 0000 '01234567........'
40: 0000 0018 FFFF FFFF 4865 6C6C 6F20 6672 '........Hello fr'
50: 6F6D 2047 6F                            'om Go           '  

No more messages 
MQCLOSE
MQDISC

$  ./amqsget Q V9300_A
Sample AMQSGET0 start
message <MDE >
no more messages
Sample AMQSGET0 end

It’s done its best – the MQMDE that is the alternative to the MQMDv2 extension has been read. But it doesn’t know to skip past that to find the real message data.

The V2 client libraries and PROtocol

The final test was to try to use the V2.2.1 client libraries properly, completely bypassing the installed V9.3 shared libraries. While I had been fairly confident the previous tests would work, I was more sceptical about this last one.

I had to set LIBPATH to point into my extracted tree instead of the default directories. I also had to set NLSPATH so that the message catalog containing error messages could be read, when things didn’t work.

$ export MQSERVER='SYSTEM.DEF.SVRCONN/TCP/localhost(3820)'
$ export NLSPATH=/mq221/usr/lib/nls/msg/prime/%N
$ export LIBPATH=/mq221/usr/lpp/mqm/lib
$ echo "Hello again" | ./amqsputc Q V9300_A
Sample AMQSPUT0 start
target queue is Q
Sample AMQSPUT0 end
$ ./amqsgetc Q V9300_A
Sample AMQSGET0 start
message <Hello again>
no more messages
Sample AMQSGET0 end
$ 

And it works – the client library located and dynamically loaded what it needed to for the communications, and the channel protocol correctly negotiated and connected to the queue manager. I also used system tools such as ldd and truss to verify that the right libraries were indeed being found. The DIS CHS(*) on the queue manager showed – as expected – empty RVERSION and RPRODUCT values, another indication that older libraries were in use.

Summary

It’s not always been possible to maintain that binary compatibility for old applications in all environments, even where we’ve continued to deliver queue manager code. Any breakage has tended to be driven by operating system changes rather than MQ itself. For example, AIX changed some of how to compile threaded programs round about that V2.2 timeframe.

MQ has continued to evolve significantly since that early release. All the new capabilities – functional, environments, APIs – that we talk about elsewhere and which people need for modern applications and deployments are there. If you want to write a NodeJS program in a “serverless” environment that has messages going through MQ, then you can do that. But we bring in those enhancements without ignoring where we’ve been.

I found the basic proof that we can also still run the old stuff to be quite impressive. And I hope you do too.

This post was last updated on June 24th, 2022 at 04:51 pm

Leave a Reply

Your email address will not be published. Required fields are marked *