Previous articles in this series have shown OpenTelemetry (OTel) Context propagation for MQ applications in different languages. You should read the first article for an introduction and explanation of the scenarios and problems that need solutions.
This entry has a couple of additional observations I had when trying different configurations with the MQ Tracing Exit (the Instana-developed component) that’s part of the MQ product.
Standard configuration
All of the previous articles in this series showed examples where the application program that is being instrumented connects as a client to the queue manager. That flow is also intercepted by the MQ Tracing Exit and spans get generated successfully. The application’s spans are linked with the MQ channel operations to give a clean end-to-end picture of the message flow. You can see the results in pictures like this one. The OTEL1 and OTEL2 spans are generated by the exit.
Additional Configurations
Run the tracing exit within client application
Why might I consider generating spans within the client program as well as from the SVRCONN end of the channel? Perhaps it might give me information about the network flow and latency. How long does it take between the application issuing an MQPUT, and the SVRCONN channel proxy actually issuing it to the qmgr?
I ran my Go sample program, with the MQCLNTCF environment variable pointing at an mqclient.ini file that had the same API Exit definition as the queue manager.
This DOES NOT WORK.
The Tracing Exit makes use of one of the features that is only available to exits running in local bindings mode. When it’s loaded into a client program, you will see MQRC_EXIT_PROPS_NOT_SUPPORTEDerrors.
Run the application using local bindings
You might be trying this more because of application topology choices, than because of OTel considerations.
If your instrumented application is C (or C++) then everything seems to be fine. Though subject to my previous comments about using C++ as an API implementation. But if you are running with applications using one of the other language packages – for example Go – then there may be problems or conflicts. In particular, on my system, the Tracing Exit created error messages like
[E 2908361] [INSTANA] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE"
error_message: "failed to connect to all addresses;
last error: UNKNOWN: ipv4:127.0.0.1:4319: Ssl handshake failed
(TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:0A00010B:SSL routines::wrong version number"
The Tracing Exit failed to send any spans to the Jaeger collector.
While I don’t know for certain what’s causing the error, my guess is that there is some conflict within the application program for how the TLS protocol libraries are loaded and interact.
Certainly the Tracing Exit is using an open source TLS package. If the OTel components for Go (or Node or Python) are using the same libraries but perhaps at a different version, or maybe different libraries with similarly named external functions, I can imagine all kinds of internal symbol errors that end up with this kind of error.
Conclusion
Maybe the exit can be modified, or alternative implementations developed, at some future time to avoid these problems. But right now, the only way to use the OTel tracing exit as part of instrumented apps in other languages would appear to be to stick to a client-only implementation.