OTel Context Propagation: part 7 – Removing effects of the MQ Tracing Exit

This article discusses a new API Exit that might help to solve a problem experienced by a number of users trying out the MQ Tracing Exit. When that exit is configured, it adds properties to the MQ message to carry OpenTelemetry context data. A receiving application may be surprised to see those properties. So this new exit can remove an unexpected RFH2 structure, allowing the application to continue to run unchanged.

The problem

On Distributed platforms, you can implement OTel Tracing by configuring the Tracing Exit. I’ve written a lot more about it in previous articles in this series. One of the things this exit does is to add properties to a message. This is required in order to carry the OTel context through each stage in the message processing.

When an application GETs the message from a queue, those additional properties are still associated with the message. And this is generally desirable, because there you might need to take that context forward into other processing steps outside the MQ domain. See this article where I showed propagation of the context into Go and Node.js systems.

But depending on how the application is written and the MQ APIs in use, these additional properties may disrupt application processing. In particular, I’ve seen a number of complaints about apps that are unable to deal with an “unexpected RFH2” in the message.

Running amqsbcg against a simple text message that was PUT to a queue manager where the Tracing Exit is configured, we see this data:

****   Message      ****

 length - 249 of 249 bytes

5246 4820 0200 0000 D400 0000 2202 0000           'RFH ........"...'
B804 0000 4D51 5354 5220 2020 0000 0000           '....MQSTR   ....'
B804 0000 AC00 0000 3C75 7372 3E3C 7472           '........<usr><tr'
6163 6570 6172 656E 743E 3030 2D30 3030           'aceparent>00-000'
3030 3030 3030 3030 3030 3030 3035 6165           '00000000000005ae'
3230 3831 3464 3537 3938 3636 372D 3338           '20814d5798667-38'
3035 3761 3864 3365 3633 3262 3234 2D30           '057a8d3e632b24-0'
313C 2F74 7261 6365 7061 7265 6E74 3E3C           '1</traceparent><'
7472 6163 6573 7461 7465 3E69 6E3D 3030           'tracestate>in=00'
3030 3030 3030 3030 3030 3030 3030 3561           '000000000000005a'
6532 3038 3134 6435 3739 3836 3637 3B33           'e20814d5798667;3'
3830 3537 6138 6433 6536 3332 6232 343C           '8057a8d3e632b24<'
2F74 7261 6365 7374 6174 653E 3C2F 7573           '/tracestate></us'
723E 2020 4865 6C6C 6F20 6174 2054 6875           'r>  Hello at Thu'
2031 3320 4175 6720 3131 3A34 313A 3335           ' 13 Aug 11:41:35'
2042 5354 2032 3032 36                            ' BST 2026       '

If my application does not know how to handle this additional block, it will cause an error. As if it’s a “poison” message.

Solutions

If your application uses JMS or XMS interface, this does not affect them. Processing of RFH2 or message properties is handled automatically in this components. We are primarily concerned with C-based applications and code built on top of that layer.

Propctl

The quickest solution is to set PROPCTL(NONE) on the queue the application gets from. This will certainly stop the additional data from appearing. But the properties (or RFH2) gets stripped from the message very early, before the Tracing Exit reads it. So that will mean that exit cannot see and report on this Span of an OTel trace. And there is no way to propagate the context to further steps in the transaction flow.

Application changes

The simplest application change, and the one I’d recommend as giving the best future-proofing, is to switch to using Message Properties. An MQRFH2 structure can be transformed by the queue manager into the Properties elements, leaving just the original message body in the main application buffer.

To do this, you need to

Change the application to create a Message Handle with the MQCRTMH call
Set the MQGMO.MsgHandle to that handle
Add MQGMO_PROPERTIES_IN_HANDLE to the MQGMO.Options field
Ensure the MQGMO.Version is set to at least 4
Call MQGET or setup a Callback function with MQCB

You can of course then ignore the returned properties, but they are available if you ever need them in future.

The alternative change for an application is to skip straight past the RFH2 structure:

unsigned char *b = buffer;
PMQRFH2 rfh2
if (!strncmp(mqmd->Format,MQFMT_RF_HEADER_2, MQ_FORMAT_LENGTH)) {
   rfh2 = (PMQRFH2)b;
  b += rfh2->StrucLength;
}
// Now start reading message contents from 'b'. 
// And the rfh2 variable holds the original message Format

This API Exit

I’ve published an API Exit here. There are several other exits of varying types available in that repository. There are instructions included on how to build and configure this exit.

It can be run inside MQ client applications and will remove any RFH2 that contains only the OTel-designated properties. This may help for applications that cannot be readily changed, and when there is no need to propagate the OTel context forward for other stages to recognise.

When I run the same test as previously, I now get just the original message body:

****   Message      **** 

  length - 37 of 37 bytes

4865 6C6C 6F20 6174 2054 6875 2031 3320           'Hello at Thu 13 '
4175 6720 3132 3A30 343A 3230 2042 5354           'Aug 12:04:20 BST'
2032 3032 36                                      ' 2026           '

And the MQMD has been updated to show the correct Format.

Conclusion

Maybe the tracing exit can be modified, or alternative implementations developed, at some future time to avoid this issue. But right now, this exit might be a simple solution for people wanting to use the OTel tracing exit withing changing existing applications.