An earlier blog entry showed how to integrate MQ with the Prometheus database, capturing statistics that can then be shown in a Grafana dashboard. In this article, I’ll show how that initial work has been extended to work with more databases and collection tools.
The latest updates allow MQ to write directly to InfluxDB and OpenTSDB databases, and also to provide data to collectd.
In a previous blog entry I wrote about using the Go language with MQ. One of the reasons for creating that Go package was to enable the creation of a program that sends MQ statistics to Prometheus and hence to be easily visualised in Grafana. This blog shows how it all fits together.
Introduction
MQ V9 metrics
MQ V9 (and the MQ appliance) makes many statistics available through a pub/sub interface. One huge benefit of the pub/sub model is that this data can be collected without interfering with any other monitoring programs. An early prototype of the MQ exporter for Prometheus used the RESET QSTATS command just to prove the concept, but that is not a good command to use in general when you have any other tools that may also use it. Publish/subscribe gives easy isolation for monitors.
Prometheus is an open-source monitoring and alerting solution, whose particular strength is the collection of time series data, with the ability to easily query that data. For example, the number of MQPUTs to a queue may be of interest, and this kind of database makes it easy to see how many operations occurred in an interval, or calculate averages. Prometheus works by pulling information from exporters such as this MQ program at configured intervals over an HTTP connection. It provides libraries in several languages to enable products to export data to it, but the most commonly used is probably the Go library – hence the need for an MQ Go package.
What is Grafana
Grafana provides a way to create dashboards and visualise data held in time series databases. It has Prometheus as a built-in data source, making this pair of products a natural fit together.
Getting started with the monitor
Building the monitor
The github repository contains the monitoring program, the ibmmq package that links to the core MQ application interface and other prerequisite components.
should pull down the client code and its dependencies. The README file in the root of that package shows how to compile the code, either locally or within a Docker container.
Configuring MQ
It is convenient to run the monitor program as a queue manager service, automatically started and stopped along with the queue manager.
The source code directory contains an MQSC script to define such a service. In fact, the service definition points at a simple script (also provided) which sets up any necessary environment and builds the command line parameters for the real monitor program. As the last line of the script calls “exec”, the process id of the script is inherited by the monitor program, and the queue manager can then check on the status, and can drive a suitable STOP SERVICE operation during queue manager shutdown.
Edit the MQSC script and the shell script to point at appropriate directories where the programs exist, and where you want to put stdout/stderr. Ensure that the mqm id running the queue manager has permission to access the programs and output files.
The monitor listens for calls from Prometheus on a TCP port. The default port, reserved for this use in the Prometheus list, is 9157. If you want to use a different number, then use the -ibmmq.httpListenPort command parameter.
The monitor always collects all of the available queue manager-wide metrics. It can also be configured to collect statistics for specific sets of queues. The sets of queues can be given either directly on the command line with the -ibmmq.monitoredQueues flag, or put into a separate file which is also named on the command line, with the -ibmmq.monitoredQueuesFile flag. An example is included in the startup shell script. For example,
starts the monitor to collect the statistics for all queues whose names begin APPA and APPB.
Note on queue patterns
For now, the queue patterns are expanded only at startup of the monitor program. If you want to change the patterns, or new queues are defined that match an existing pattern, the monitor must be restarted with a STOP SERVICE and START SERVICE pair of commands.
Configuring Prometheus
The Prometheus server has to know how to contact the MQ monitor. The simplest way is just to add a reference to the monitor in the server’s configuration file. For example, add this block to /etc/prometheus/prometheus.yml with any changes needed for your hostnames and ports.
# Adding a reference to an MQ monitor. All we have to do is
# name the host and port on which the monitor is listening.
# Port 9157 is the reserved default port for the MQ monitor.
- job_name: 'ibmmq'
scrape_interval: 15s
static_configs:
- targets: ['hostname.example.com:9157']
The Prometheus documentation has information on more complex configuration options, including the ability to pull information on which hosts should be monitored from a variety of discovery tools.
Once the Prometheus server has picked up the MQ configuration, the metrics can be seen under the jobname of ibmmq. The values are labelled with the queue and queue manager names, to assist with selection. This picture shows some of the available information in the selection drop-down:
Selecting metrics
You can select an item from this panel and see its recent values with the queue and queue manager labels. For example,
However, it is more flexible to work with the graphing and dashboard views from Grafana.
Configuring Grafana
Once the Prometheus system is working, grafana can use it as a datasource – again, only a hostname and portnumber is required when adding this type of datasource. And from there, all of the MQ metrics can be accessed and added to dashboards. As an example, this dashboard is looking at several items including the same queues as above, and CPU and logging information:
This picture shows how the top panel was configured, to select several metrics and show the object name in the legend:
Deployment in Docker containers
All of these components can be configured to run inside Docker containers to simplify deployment. To get started, almost everything in the existing Prometheus and Grafana containers can be left to default, except for the need to add the MQ configuration to prometheus.yml. For example, I have this simple Dockerfile
FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/prometheus.yml
where I’ve added the ibmmq block shown above to the default yml file.
And then this script gets both the Prometheus and Grafana components running, using local directories under /var/docker to hold their persistent data:
The MQ exporter program and its configuration can also of course be baked into a Docker image. The MQ docker image on Github has information on the configuration of MQ. The service definition, the shell script and the actual monitoring program can all be copied into a new image.
Conclusion
This article has shown how the statistics generated by MQ can easily be used in some of the monitoring packages that are commonly used with various cloud and container-based systems. The MQ data can be integrated with other metrics to give a complete view of your environment.
I would welcome feedback on this tool. Please leave any feedback here, or in the GitHub issue tracker, whether bugs, enhancements, or thoughts on the value of the monitoring.
This post was last updated on November 27th, 2021 at 03:00 pm
This article is going to talk about some new code shipped with the V8.0.0.4 FixPack to make it easier to see what a queue manager is doing, simplify investigation of possible problems, and to assist with writing your own MQ applications. Formatting MQ Events is now available as part of the MQ product.
To start with, we need to talk a little about MQ’s event generation.
This week a group of us got together in Hursley and started on the next IBM Redbook for MQ. This time it’s about the newly announced MQ V8. Read more about the new book
I was prompted to write this by some recent conversations and news items about product certifications. Microsoft recently announced that they were ending their top level certification scheme. A more interesting certification process was the story about beer!
We have MQ product certifications. You can become a certified designer or administrator. But what’s the value of that? Is there any value?
My personal view is that there is close to zero practical benefit to getting these accreditations.
One of the hit sessions at this year’s Impact was not listed in the agenda. A bit like an off-off-Broadway show, it was given to a select few groups. This was the snappily-titled “MQ Workload Distribution in a Sysplex”, given by Lyn as part of the z solution suite activities. It talks about how different systems may process different amounts of work, and how this can be changed to more balanced distribution if you need to.
Since the initial run, she’s been asked to repeat the session for a bunch of customers, and there’s probably more requests than available time. So when she was visiting the UK recently, we took the opportunity to record the session and it is now available on youtube as a three-part playlist. (I quite like playlists, as it makes it easy to replace individual pieces without modifying the published URL.)
Now that Impact is out of the way, we are getting to the next round of the Hursley Comes To You and the Connectivity and SOA Summits. This time covering Europe (or countries that IBM seems to think are part of Europe).
The first one is at Stockholm next week, on June 11. Then it moves to June 20 in Manchester, July 1 and 2 in Tel Aviv, July 8 in Paris, July 9 and 10 in Madrid.
And this blog is apparently partly responsible for one of them: the Paris event was requested by someone who had read my writing about the US events. And we were able to oblige.
The agendas are slightly different at each event, depending on the expected audience.
But I know that at Paris we are planning to include some deep-dive technical subjects. It will be done in English – sorry, but my French isn’t up to discussing performance tuning. Hopefully that will be OK. I’m looking forward to doing that one as I’ve not been to Paris for many years. Going to see if I can get the train over.
When I have an early weekday flight from Heathrow, I often stay at one of the nearby Hiltons the night before. It’s convenient, not outrageously expensive (especially with points), means I don’t have to get up at 4am to drive up the M3, and I can get a good breakfast in the lounge before going over to the terminals.
So I was doing that last night, checked in, went to check my email, and the laptop power supply went “pop”. No charging possible. I tried a new cable in case it was the fuse, tried different sockets, but it was clear that something was broken. Not a good thing to happen when you’re about to be away for a week with no real hardware support.
The Impact conference starts in 10 days in Las Vegas. For the speakers that means we need to finish up writing presentations, although that seems to happen later and later at each event.
Many conference regulars will be there: myself, Lyn, Morag, Ralph as well as executives and some others who aren’t allowed out in public so often, and the marketing people. As always, this is a busy week not only for the public talking but also the stream of meetings that we participate in. Going to Vegas is not a jolly, and there’s often attempts to NOT be sent.
There used to be a much stricter deadline for finalised presentations, as material was only distributed in physical formats. So it had to be submitted several weeks in advance.For some subjects that was fine, but it made it harder to deal with late-breaking changes, such as if an expected product announcement was cancelled or a new feature altered its implementation.
I was digging through my cupboard a few weeks ago and found the book of presentations from the 1998 MQSeries Technical Conference in Washington DC – properly published and bound. Very nice, but heavy and for those of us who try to fly with only hand-luggage a bit awkward. Back then, we also often used boxes of real acetate foils, carefully separated and hopefully kept in order; there was no guarantee of projection systems even if we had laptops with us. (That DC event has special memories for me, though it’s not why I kept the book.)
It’s interesting looking back at that old material. Features of MQ that were considered interesting or exciting but which are no longer used; and the converse, features that are still critical to customers. Here is the MQSeries V5.0 presentation from 1997/1998 as an example. The “web admin” feature is long gone in that form, as is the DCE Directory Service support. But transaction coordination is still very heavily used.
Although we try to predict which features are going to have a long-term value when planning new releases of MQ, these “What’s New” presentations do show where we got it right and where we didn’t.
