MQ API Exits – FAQ

Exit

In the past few months I seem to have been been sent a bunch of questions about MQ API Exits from a variety of developers. Some of the questions were repeated so I thought I’d try to collect them, tidy them up, and turn them into an FAQ article.

API Exits were originally made available for one platform in MQSeries V5.2 and then extended to the rest of the Distributed family in V5.3 so they have been around for a while.

Exits are an advanced topic and I’m not planning on going into much general information about these exits. I will assume that someone interested in this article already knows the basic principles and is comfortable working in this environment. For more information on the interfaces see the KnowledgeCentre and the sample amqsaxe0.c exit.

Contents

General

Why might I use API Exits?

The main reasons for using an API Exit are firstly to monitor what an
application is doing:

  • Auditing
  • Problem determination
  • Performance monitoring
  • Reviewing application conformance eg correct use of flags in API calls

Or for modifying application requests:

  • Enforcing standards such as always using FAIL_IF_QUIESCING
  • Adding metadata to messages
  • Security eg encryption

What platforms can I use API Exits on?

API Exits (and other exits I might mention in this FAQ) are features on the Distributed platforms. Although there is an API Crossing Exit for MQ on z/OS, it is very different and only available in the CICS environment. So we will ignore that.

How do API Exits compare to other approaches to managing what applications do?

The main alternative to API Exits is the Application Activity Trace. They both give the ability to see what applications are doing, but in very different ways.

  • Application Activity Trace is asynchronous, processed some time after the application has run
  • Application Activity Trace does not require any additional installation, or specialist code. Sample programs are provided with MQ that can format the generated event messages.
  • Application Activity Trace is “read-only”: it cannot affect what the applications are doing
  • API Exits are synchronous, called in-line with the application code
  • API Exits can modify the MQI parameters as they pass between the application and the queue manager

What languages can I use to write API Exits

Only C is a supported language for writing these exits.

Environment

My exit is only needed for specific applications. How do I stop my exit being called for all apps?

The exit is loaded for all applications. During the initialisation phase, you can look at the pExitContext->Environment and the pExitParms->APICallerType to get a lot of information about the type of caller.

At the highest level, the APICallerType is split between INTERNAL (which is only set by some of the MQ product processes) and EXTERNAL (all user applications and some other product processes). The Environment field is another filter you can apply to your decision.

For example, one of my exits that wants to intercept all user-invoked calls has

  if (pExitContext->Environment == MQXE_MCA_SVRCONN)
    interceptCalls()
  else if (pExitContext->Environment == MQXE_OTHER && pExitParms->APICallerType == MQXACT_EXTERNAL)
    interceptCalls()
  else
    /* not interested in these qmgr connections */

That gets all direct connections and all those coming from clients trapped including clients written in languages such as Java and .Net which cannot support API exits running inside their own processes. The interceptCalls function then calls MQXEP to register the MQI verbs I want to work with.

Remember that a few queue manager processes have special MQXE_ values. I used that in one exit where I was only interested in trapping operations done by the command server.

A further filter that you may want to use is based on the application name. Before MQ 9.1.2, that was available during the exit’s initialisation function. However now that applications can set their own names as one of the MQCONNX parameters, the name passed to the exit may start off as blank and only be available later in the processing. So you may wish to always register the MQI verbs to be intercepted during connection initialisation, and then unregister them later when you know that the application name is not one you want.

What is the link between thread and connection handle? Might one thread process other connections? Or might one connection handle be processed by more than one thread?

There is no relationship between thread and hConn.

Each thread may be handling multiple hConns; each operation on an hConn could be issued by different threads. The exit is running directly under the application’s stack so follows whatever behaviour the application has coded.

Can two application MQI calls be active simultaneously on the same thread? What is the serialisation?

Once an MQI call has been started, it remains on the same thread for the rest of its execution. That includes the calls to BEFORE and AFTER exit points. So if your exit is called for, say, PutBefore you know that that the next thing seen in that thread of your exit will be the PutAfter for the same queue. That means you can use techniques such as stashing
some information in thread-local-storage during the BEFORE, and using it in the AFTER. A common reason for doing that is to restore parameters such as pointers to buffers that your exit might have replaced for the MQPUT, but which need to be given back to the application unchanged. The next operation for that queue’s object handle may be on a different thread of course.

You can also use the MQAXP->ExitUserArea as a space to hold temporary stashed pointers as that area is maintained across all calls to the exit for a given hConn – different hConns will have a different space. Usually that space holds a single pointer to a larger area allocated during your exit’s initialisation (perhaps a structure containing control data you need to maintain), and it can then be deallocated during the termination phase.

A variation on this happens in the MQ CallBack/MQCB operation for asynchronous delivery of messages. The exit is called by the queue manager on the same thread with which the application will get the message data. The stack can be thought of as inverted here – the thread is “locked” until the return from the exits and the application code. While it’s possible to call MQI verbs from within the callback processing, this must be done on the same thread. I’ve expanded on CallBack in a later Q&A.

What version of a structure does an exit see?

The exit sees the structures such as the MQOD or MQMD exactly as the application has used them. So you may not see the most recent version of a structure. If you want to make use of fields that were added to later versions of the structure, then you should define a local copy of the structure in your exit, copy the application’s values into your copy, make sure you have the newer version set in the structure, and then reset the parameter that is passed down into the queue manager.

That is why a lot of parameters to the exit are defined as pointers-to-pointers – it allows replacement of parameters. Then on return from the queue manager (the AFTER phase) you have to copy anything returned back to the application and restore the original structure pointer so that the application can work with what it needs to.

You cannot rely on the application having allocated sufficient space for the newer version of the structure.

Can an exit call the MQI itself?

Yes. But it should not use the verb directly by its usual name. Instead the exit should call via indirect function pointers passed to it. This
is recommended because of how MQ libraries may get dynamically loaded underneath the application as part of the support for multiple product installations.

So, do not call MQOPEN. Instead use pExitParms->Hconfig->MQOPEN_Call. The parameters are the same as the regular MQI verbs. API Exits are not recursive – calls from one into the MQI are not intercepted by any exits including itself.

Can I configure multiple exits? What order are they called in?

You can configure multiple exits with stanzas in the qm.ini and mqs.ini files. One of the attributes in the stanza is called Sequence. This is a number that controls which order exits are invoked.  During Before operations, the exits are invoked in ascending order; during After operations, the order is reversed. That means that do/undo processing can take place in a predictable sequence.

For example, consider two exits where one does encryption/decryption and one does compress/decompress. Clearly you want the compression to happen before encryption. During an MQPUT, the PutBefore works through the exits in numeric order so we want the compression exit to have a lower Sequence than the crypto package. During  an MQGET, to restore the original message content, the decompress must happen after the decryption. Because the work is done in a GetAfter call, the reversed numeric sequence is applied and everything works as it should.

 Will two exits interfere with each other?

The exits will not really interfere with each other but one will see the effects of an exit called before it. So with the previous example, the encryption exit will see the message body that has already been modified by the compression exit, and can encrypt the smaller amount of data.

Of course, the two exits might conflict in specific ways – for example if both wanted to set the same named property in a message. But a properly-designed exit should ensure that kind of thing is either configurable or given a unique scoping name.

Can I tell which version of queue manager the application is connected to?

You can get the queue manager CMDLEVEL and VERSION via MQINQ. The VERSION is a string corresponding to the VRMF. Do an MQOPEN/MQINQ/MQCLOSE for the qmgr object from within the exit. That could be issued in the ConnxAfter phase. It’s possible that the MQINQ might fail in some apps because of authorisations, but should not in the case of channel programs.

Using MQINQ to get information about the queue manager is very standard. Even the MQ JMS implementation uses it internally. If the call fails, then there should be suitable fallbacks and defaults taken, but it is very common to use this check.

How should I provide configuration for my exit? Where should output be sent?

Logs and configuration files probably should be in a directory associated with the queue manager name to allow separation. It’s probably best to avoid anything going into the /var/mqm tree, again to keep it separated from product data. Remember that there are some limits to fields in the API Exit stanza in the qm.ini file so you can’t expect to put a full path name in there eg to name an output directory unless it’s somewhere short like /var/log/apix. Although the amqsaxe0 example exit uses environment variables to name some of these attributes, I don’t like that as a more general approach as the variables would have to be set globally for all applications.

One thing to remember is that, in general, because an API exit is running under application context, output directories may need to be world-writable and configurations may need to be world-readable. Perhaps output gets sent via a proxy process which has separate user configuration. This becomes less of a concern when all user applications are client-connected; then the exit is only ever run inside queue manager processes, and has its level of operating system access.

Puts & Gets; Queues & Topics

When an MQPUT happens, how do I know the name of the queue?

Just as in normal application code, the only direct knowledge of the queue is via the object handle. Mapping that to the queue name requires you to trap MQOPEN calls and store the queue name along with the returned object handle. You can then do the lookup during MQPUT/MQGET.

Your exit will, of course, see the name of the queue as specified by the application code. That could be an alias or remote queue, or one to be resolved by a cluster. On the AFTER flow, you may also see the name of the resolved queue, depending on the options specified in the MQOPEN by the application.

You will also see in the MQOD ResolvedType field an indication of whether the MQOPEN resulted in a queue or topic being opened (for example in the case of an alias queue pointing to a topic). Since the ResolvedType field was added in the MQOD version 4, you may need to temporarily swap out the MQOD supplied by the application with your own copy if the app is passing a lower level of the structure.

How do I know if a message has been retrieved because of a subscription? From which queue?

If you want to know that an MQGET is a result of a publication arriving at a queue, then you’d need to monitor the MQSUB calls to know the hObj in use for the MQGET. That object handle may have been returned from a previous MQOPEN. For managed subscription handles, there will be a real queue assigned but you will not have seen an MQOPEN for it and only
get the hObj returned. However, that hObj is opened with permissions for you to use it in an MQINQ so it is possible to call that from within the exit if you really want to know the name of the dynamic queue.

Does an exit see every put/get of messages?

You see all the verbs issued by applications, so that includes all normal puts and gets. But the exit is not called for internally-generated messages put by the queue manager like triggers or events.

Are there any special considerations when opening a topic?

Opening a topic is much the same as opening a queue. But you may need to look at both the ObjectName and the ObjectString fields in the MQOD. Either or both of the values may be set.

If someone PUTs to a topic, how does the exit know which queues the message is published to?

Basically it does not and cannot know. At least, not using this interface. The fan-out is done inside the queue manager during the MQPUT. Just as applications do not know about subscribers, neither does the exit.

A different exit point, the Publish Exit, does exist. That exit is called once for every message that is being delivered to a subscriber’s queue and allows modification of the message and manipulation of the target queue. But it

  • Is running inside the queue manager core processes so much more dangerous in failures
  • More performance-sensitive
  • Only one exit can be configured for a queue manager

Look at the sample exit amqspse0.c source and the associated structure definitions in cmqxc.h to see some of what it can do.

There is no linkage between an API Exit and the Publish Exit. They are completely separate exits, running in different process spaces and
at different times in the overall flow.

Do GET and PUT support more than 1 message per invocation?

MQPUT/MQPUT1/MQGET only deal with a single message at a time. As does MQCALLBACK.

What is the difference between BufferLength and pDataLength in the MQGET processing? Which is best used to measure message payload size?

BufferLength is the space provided by the application into which message data can be stored. DataLength is the actual message length that is being returned and can be no longer than the BufferLength.

What happens if there is not enough space in a buffer for the message data?

In normal MQI processing, the application will see an MQRC_TRUNCATED_MESSAGE_ACCEPTED (with MQCC_WARNING) or
MQRC_TRUNCATED_MESSAGE_FAILED (with MQCC_FAILED) depending on its options. The exit will also see the same return values.

Where it gets a little tricky is if the exit is further manipulating the message content and buffers. There’s no clear rule that should be applied for exit design, but remember that once the queue manager has returned MQCC_WARNING or MQCC_OK then the message has been removed from the queue – if an exit then goes on to indicate that the message retrieval has failed because it could not fit the exit-modified data in the application’s buffer, then the application cannot retry the MQGET to the queue manager. Remember too that if an application has failed to get a message because the buffer is too small and then retries the MQGET, the next-returned message might not be the same one as previously seen. Another application using the same queue (assuming there is non-exclusive input) might have already successfully removed the message and now you cannot rely on any knowledge of the failed buffer sizes to determine how to proceed.

If the application is using transactions, then in theory you could force an MQBACK to restore the message to the queue but it’s not normally a good idea to interfere with the application’s view of transaction boundaries.

When do the Before and After calls for MQ CallBack occur in relation to the client application?

For a regular MQGET with data conversion the API Exit gets:

  - Function=MQXF_GET, ExitReason=Before
  - Function=MQXF_DATA_CONV_ON_GET, ExitReason=Before
  - Function=MQXF_GET, ExitReason=After

For CallBack with data conversion, the API Exit gets:

 - Function=MQXF_DATA_CONV_ON_GET, ExitReason=Before
 - Function=MQXF_CALLBACK, ExitReason=Before
 - Function=MQXF_CALLBACK, ExitReason=After

For MQCallBack without data conversion, the API Exit gets:

 - Function=MQXF_CALLBACK, ExitReason=Before
 - Function=MQXF_CALLBACK, ExitReason=After

The ordering of these flows for Callback seems odd

It may seem strange that the data conversion call is done first for a Callback when it is between the BEFORE and AFTER in the synchronous MQGET case. But realise that when you are using callbacks, the message has been removed from the queue before the callback function is invoked. It’s logically pushed into the app, rather than the synchronous pull in MQGET processing.

So the logic works that the BEFORE phase needs to be given the message data, unlike during MQGET where you’d see that in the AFTER phase. And the data conversion point is also done before the user’s callback function is invoked, so the API Exit can see both unconverted, and then converted, message data – potentially changing them before the user’s function sees it.

Think of it like

* MQGET --> Exit(MQXF_GET/BEFORE) --> qmgr
*                                     message passed to app process
*      <-- Exit(MQXF_DATA_CONV)
*             data converted
*      <-- Exit(MQXF_GET/AFTER)

MQGET completes with message contents

With callbacks …

* qmgr --> message pushed to application process --> Exit(MQXF_DATA_CONV)
*        data converted --> Exit(MQXF_CALLBACK/BEFORE)
*                       --> user's callback function invoked
*                       <-- Exit(MQXF_CALLBACK/AFTER)

How can I investigate further into when and how exits are called?

Traces of the application process will show the flow – look for functions starting with zutCallApiExits. While tracing, in general, is intended for IBM service to look at, there’s often a reasonable amount of information you can deduce from looking through one.

Message Properties

One thing I’ve started to see more of is the use of exits to manipulate message properties. A common idea is that a property gets added to a message by the exit, for inspection and reporting by further instances of the exit running in other applications on other queue managers. It enables things like performance monitoring and tracking of application end-to-end flows. The property is then ideally removed from the message before the intended application sees it.

There is no ideal design for these situations. There can be all kinds of pathological cases, with different application designs and system configurations, but it does work well-enough for the vast majority of real-world topologies and architectures.

A sample exit amqsaem is shipped with MQ to show some ways of working with properties.

With that in mind, here are some specific questions and comments I’ve received about working with properties in API Exits.

My exit usually reads some values from the MQMD but occasionally, the MQMD is NULL. Is this correct? How do I proceed?

It is possible for the MQMD for a message to be NULL on entry to the MQPUT intercept point. One way to demonstrate this is to use the amqsstm sample program that shows how to set message properties.

For example

  echo  "prop1\ngreen\n\nMy message body\n" | /opt/mqm/samp/bin/amqsstm QUEUE1 QMA

To see the effect, you can check for null on PutBefore and PutAfter intercept points:

  if (ppMsgDesc == NULL || *ppMsgDesc == NULL) {
     // print to log
  }

If you browse the message on the queue, it has a fully-fledged MQMD.

This happens because amqsstm does explicitly set the MQMD to NULL. It’s making use of a very unusual case (and not something I’ve ever seen used in regular applications) where the MQMD is essentially held instead in message properties. The idea is that an app can easily copy an inbound message+MQMD to outbound while just providing deltas to properties including the MQMD.

So if the MQMD is NULL, you should be able to find the equivalent information in properties such as “Root.MQMD.Format” from one of the message handle references passed to the exit in the MQPMO structure just as it has been set by amqsstm. If a property is not discoverable through any of the message handles, then it would have the default value associated with the MQMD structure.

If both NewMsgHandle and OriginalMsgHandle are set, then properties in the NewMsgHandle take precedence over the same-named property in OriginalMsgHandle.

If two messages are put, one put using a named property, and another put with an RFH2 folder, will they look identical on a binary level?

A message that was put using an RFH2 to store properties, or put using MQSETMP properties, or even with a hybrid of both, will be processed the same internally. The receving program doing an MQGET has no idea how the properties of the message were originally set. The queue manager is even at liberty to do things like combine chained RFH2s into a single block for storage and onward transmission.

If the application has not used properties or RFH2 when putting the message, can (or should) an exit add its own properties.

The answer here really depends on whether you want to add a new property
that is only visible to instances of your exit, or whether you want to use a property that could be visible to application code.

An exit can register a folder name during its initialisation that can be
used for dedicated properties. Assuming that the same exit is used both
for PUT and GET applications, then any properties belonging to that folder and added to the message during the Put phase will be visible only to an exit that has registered use of the same folder during Get. No matter how the application retrieves the message (or settings on the PROPCTL attribute of the queue), then those properties will not be seen in the application.

If the exit may sometimes be modifying properties created outside of the exit, then it would not need to register a folder and instead would likely be referring to something in the usr folder. While the exit can add the property if it doesn’t exist, there is a potential issue that applications that are not expecting to see any properties in the message will now receive it either as a property or in an RFH2 header. Which might break the application. The PROPCTL attribute can be used to always strip properties before the application sees them but that may not be feasible to enforce. An exit may need some kind of configurable policy on whether to insert properties into application-level folders if they are not already in there.

We are trying to use PutBefore to change a message property value that will be put to the queue. How should this work?

In general, an exit can modify an MQPUT with a named property or an RFH2 key-value tag. It is probably easiest to use the MQSETMP approach rather than modifying user-supplied buffers with an RFH2.

If the application is already using properties and there is a message handle passed in, then the exit can exploit that. If there is no
message handle, any existing property will have been set with the RFH2 structure. The exit can use the MQBUFMH verb to convert an RFH2-style message, removing the RFH2 and attaching the properties to a handle which can then be queried or modified.

Conclusion

I’ve tried to summarise a number of questions from software developers about API Exits. While investigating some of the more obscure features particularly around properties that people were asking about, I found it very useful to have a skeleton API Exit that I could modify quickly, and Makefiles and scripts to drive simple tests. I don’t think I wrote any application code for these tests – the standard samples (amqsput, amqsget, amqsbcg) and amqsstm seemed to be all that I needed. Don’t forget that one of the parameters to amqsbcg tells it whether to show properties or use RFH2 formatting for the messages it browses.

I will try to update this article if I receive further questions in this area. I hope this is useful.

This post was last updated on June 12th, 2020 at 03:32 pm

Leave a Reply

Your email address will not be published. Required fields are marked *