MQI Node.js library: an experimental reimplementation

The MQI library for Node.js applications first appeared in 2017. One of the first articles about it is here. Regular maintenance and updates to the library ensured it kept up with newer MQ and NodeJS features. But it seemed time to do a more extensive rewrite of the package, and that’s why I’m writing this article. I’ve put an experimental reimplementation on GitHub of the library for you to try out, before it goes into the mainstream release. Originally in the “napi” branch, it’s been promoted to the master branch.

Update: This new version is now on npm as the ibmmq 2.0.0 release and has been merged into the main GitHub branch.

What and Why?

Essentially, the library is now in two related pieces – one provides the Node.js API for applications, and a second C++ layer handles the link to the underlying C MQI libraries. The original implementation relied on external packages to do that C++ work; but I wanted to remove those.

This new version has at least one definite benefit. And there are several further possible benefits, that will hopefully prove to be true.

• The external dependency packages (various libraries with ffi and/or ref in their name) were collecting a number of issues on their GitHub repositories. It is clear that, although they work for now, there is not a lot of attention to their maintenance. It would not surprise me if they stop working at some point. I know I reported a bug several years ago that still exists. I could have taken a fork of those libraries to do my own updates, but decided it was cleaner to start over with something specific to MQ requirements. Reducing dependencies is always a good thing anyway, even though Node.js seems to encourage proliferation.
• Better performance. This remains to be tested. But it feels plausible that copying values into and out of JavaScript objects and variables could be faster when we know exactly what we need to do. Generic transform methods might appear simpler, but they have a couple of layers of additional translation.
• Support for more platforms. With the libffi approach, it was impossible to support some other platforms. In particular z/OS, where I know there has been some interest in writing Node.js applications for MQ. Rewriting to use a core Node.js interface, known as Node-API (or N-API), at least makes it plausible that we could extend platforms.
• Truly asynchronous MQGET processing. Again, this is not in the new library today. But it was another impossible thing in the old library. The way that threads work in the Node.js engine was incompatible with the threading model for the MQCTL/MQCB callback approach. So the current pseudo-asynchronous MQGET uses a set of polling loops and heuristics to pretend it’s using those services. These help to a) share resources fairly and b) allow other work to be done in the application. But with more complete control of the C++ work, and true multi-threading capabilities outside of the application thread, I can imagine ways in which I might be able to get rid of the polling.

How to try it out

One thing I’ve been careful about is to keep the same public API as the older implementation. So if you have existing MQ applications using the ibmmqlibrary, then they ought to continue to work unchanged. To use this trial version, all you need to do is change your package.json file to point directly at a GitHub repository and branch. So instead of

  "dependencies": {
    "ibmmq": ">=1.0.5"
  }

you would, for now, have

  "dependencies": {
    "ibmmq": "github:ibmmqmet/mq-mqi-nodejs"
  }

I’ve not put the new library into the npm repository yet, but npm install recognises this dependency format and will pull in the tree directly (albeit sometimes slowly) from the GitHub branch.

Note that this style of dependency naming with a branch does not always pull in the latest version, because the current git commit level gets embedded in local meta-data. On a “reinstall”, that same commit level is reused by default. To force the newest commit to the branch to be used, you should clean out cached information:

rm -f package-lock.json
rm -rf node_modules
npm cache ls | grep ibmmq | xargs npm cache clean --force
# And now we can do the reinstall.
npm install

I have tested on Linux(x64), Windows, MacOS and AIX. All work provided you have appropriate tools installed for the C++ compilations required. Using the older library probably required the compilers too, but not in all circumstances. The sample programs in the repository have all been tried, and they ran without change.

The future

Assuming this new version works as intended, there are a number of things I hope to do with it or collaborate on:

• Release it to npm, probably as a V2.0 level even though it’s the same public API as the current V1.
• Use something like prebuildifyto reduce the need to compile C++ code during the npm install. This might be an chance to also play with GitHub Actions, so another learning opportunity.
• More extensive testing including performance checks, and making sure there are no memory leaks. One concern with the ref packages was that they might have leakage; hard to verify but I think this new version is at least easier to debug as it does not need to provide all the generic function.
• Test the Typescript layer – since there’s been no change to the Node.JS API, the current Typescript bindings ought to work. But it needs verification.
• Look more at a true async MQGET – the code’s been restructured to at least make that a bit simpler to work on and replace.

Summary

I’ve released this library early in its development to encourage people to try it out, and report issues or provide enhancements. Please give it a go. Comments can be added to this post, or as issues on the main library repository.

[This post also made available here on the IBM MQ Community site, for a broader audience.]

This post was last updated on June 22nd, 2023 at 08:46 am