The first article in this series introduced OpenTelemetry (OTel) tracing with IBM MQ. It showed how the trace context can flow when the application is using the MQ Node.JS or Go interfaces. This article carries on the story for OTel context propagation, this time talking about C++ and C applications.
Instrumenting C and C++ applications
Just as with the other language interfaces, your application should already have OTel instrumentation. With C and C++ applications, that is not as straightforward as some other environments.
For a start, the OTel project does not provide pre-built libraries, though there are some places you might be able to download them from, depending on your operating system version. As another aspect, there are no automatic or standard library integrations. So you will probably have to build the OTel libraries for yourself, and then add the necessary instrumentation to the application directly.
OTel also do not provide any native C support; everything is done via C++. So instrumenting C applications will likely require some wrapper layer to convert C to the object-oriented APIs. That’s how this API Exit is written – using a mixture of languages to convert MQ’s C interfaces to the C++ operations driving the OTel API. (The fact that this made it easier to use constructs such as Maps was certainly a benefit but it wasn’t the primary purpose.)
The MQ language binding and exit
While the Node.js and Go propagation layer sits above the C MQI, manipulating message properties and the MQRFH2 structures, the C variant runs underneath the C MQI. It is implemented as an API Exit. But it is logically doing an identical thing to those other bindings. Your application does not need to do anything different, beyond its own OTel work. Calls to the MQI are transparently picked up and processed.
Outbound messages have properties added that can be picked up by a receiving application; inbound messages with context have that context linked into any existing OTel trace/spans. This exit does not emit any new traces or spans of its own.
This layer may be running inside a Go or Node.js program, but there is no “double accounting”. The wrappers’ OTel work takes precedence if the apps are instrumented.
The exit has been written and tested on Linux platforms only. It MIGHT work on AIX with small tweaks to the Makefile (though getting the OTel SDK libraries might be more challenging); it would require more changes to build and run on Windows. It can work in conjunction with, but does not require, the Instana MQ Tracing exit.
An incidental benefit of this exit is that it demonstrates many of the features of a larger API Exit. In particular, how you can work with message properties.
RemoveRFH2 Option?
One difference from the other MQ OTel binding layers is that there is no RemoveRFH2 option on the MQPMO and MQGMO structures. So your application might receive “unexpected” properties or RFH2 header blocks. Part of the reason for not implementing that feature in this exit is because the MQI cannot be extended in any way: the API Exit is inside the MQI calls, not wrapping it. The application would need to use an alternative mechanism. Perhaps an environment variable, but I don’t like using those for this kind of feature. If it turns out to be necessary, then it could be added in future.
Building the exit
First, you need to get a copy of the MQ API exit. Use git clone to pull down a copy from here. The OTel exit is in the apix/otel subdirectory. Before building the exit itself, you also need the OTel cpp libraries.
OpenTelemetry library dependency
The exits depend on having the opentelemetry cpp libraries already available. Either archive or shared libraries can be used depending on your preference. But using shared libraries is more likely to need additional configuration such as LD_LIBRARY_PATH to be set so that they can be found and loaded at runtime.
While you might be able to find existing pre-built libraries, it’s more likely that you will have to build them yourself. See Getting Started material here for information on creating these libraries. For this exit, you do not need any of the exporters to be built, which means that many of the dependencies (eg grpc) are not necessary.
However you do need to build the OTel libraries with the ABI version 2 option. This is done by setting -DWITH_ABI_VERSION_2=ON -DWITH_ABI_VERSION_1=OFF in the cmake configuration. You also need -DCMAKE_POSITION_INDEPENDENT_CODE=ON.
This is where I might like to have a rant about how inappropriate C++ is for library packages … But I’ll mostly resist that temptation.
The API Exit
Once you are in the correct subdirectory of the MQ Exits repository, execute the Makefile with make! The output from that step contains two things:
- An API exit written in C that loads the main tracing exit. Various MQI calls are then proxied to the tracing exit.
- A tracing exit written in C++ using the OTel C++ tracing libraries. Note that this is really mostly C code, as it needs to work with the C MQI. It only gets into C++ mode when essential.
The Makefile might require editing to point at directories where your OTel libraries and include files live. You might also need to modify the OTELLIBSvariable to match your choice of archive or shared OTel libraries.
When you build the exit, the output includes a 32-bit version that does no real work so that 32-bit applications using the queue manager (if you configure it at that point) do not actually fail to run.
More likely, you would run the exit in an MQ C client, with the mqclient.ini file pointing at the exit.
Installation and Configuration
The doit script copies the binaries to a suitable place in the /var/mqm tree.
If you want to use the exit in an MQ client, then the mqclient.ini file can point at the exit. And your application may in turn need to use the MQCLNTCF environment variable to point at the mqclient.ini file.If you want to use the exit for local bindings applications, use the queue manager’s qm.ini file.
In both ini files, the syntax for defining the exit is the same:
ApiExitLocal:
Sequence=10
Function=EntryPoint
Module=mqiotel
Name=MQOTelExitIf you configure this at the queue manager level and also have the Instana exit installed on the queue manager, then the sequence number associated with this exit should be lower than that associated with the Instana exit. That ensures that the queue manager calls the exits at the right time.
The exit is only effective in application processes, whether using local bindings or client connections. It does not have any effect inside queue manager processes. This includes the amqrmppa processes that handle the SVRCONN
connections.
Logging/debug
Set the APIX_LOGFILE environment variable to see various bits of debug reported during the application’s execution. That variable can point at either a filename, or be set to stdout or stderr to print to the console. Problems getting the exit loaded may be easily diagnosed with this log.
As the exit does not do any work in queue manager processes, printing to the console can be appropriate for applications.
The exit also populates a field used by the MQ service trace to show it has been loaded successfully or not.
Instrumented applications
Instrumenting your C/C++ applications to use OTel tracing is beyond the scope of this document. The Getting Started page referenced earlier has useful information.
One requirement is that the application must also be using the ABI V2 OTel libraries. Mixing both V1 and V2 libraries in the same application process seems to cause confusion. And of course, this exit is only useful for instrumented applications.
Testing
Just as in the previous article, I used the equivalent roll-dice application from the Getting Started exercise. And replaced the rolling with MQPUT and MQGET calls. But this time in C++.
As we would expect, the traces are similar. This time showing the CppPut application and flow through the queue manager network:
Conclusion
The OTel support for C and C++ applications seems less complete than for other application languages and frameworks. Developers need to do more work to instrument each application that they might need in a Node.js application. And the C++ libraries have their own issues around building and distributing.
But perhaps this component will give a boost for these applications to encourage OTel instrumentation, knowing that they can be observed in their MQ integrations.
This post was last updated on November 20th, 2024 at 08:01 pm