{"id":1706,"date":"2025-02-27T12:03:30","date_gmt":"2025-02-27T12:03:30","guid":{"rendered":"https:\/\/marketaylor.synology.me\/?p=1706"},"modified":"2025-03-01T09:55:42","modified_gmt":"2025-03-01T09:55:42","slug":"native-ha-crr-metrics","status":"publish","type":"post","link":"https:\/\/marketaylor.synology.me\/?p=1706","title":{"rendered":"Native HA CRR metrics & mqperfck"},"content":{"rendered":"\n

There will, no doubt, be full discussions elsewhere of the new MQ Native HA Cross-region Replication<\/a> feature. But one aspect of it seemed particularly relevant to other articles on this site: monitoring the CRR metrics. <\/p>\n\n\n\n

The MQ 9.4.2 release also includes another tool, mqperfck<\/code>, that introduces some additional metrics. And I’ll talk about how that potentially affects other monitoring tools.<\/p>\n\n\n\n\n\n\n\n

CRR Metrics<\/h3>\n\n\n\n

This older article<\/a> shows the original Native HA statistics that the collectors in the mq-metric-samples<\/a> repository can report.<\/p>\n\n\n\n

The latest release of that package can also deal with the CRR metrics. In fact, if you are using these monitoring solutions, and you start to use MQ 9.4.2, then a very strong recommendation (a definite requirement for Prometheus) is to upgrade to this new version v5.6.2. <\/p>\n\n\n\n

If you do not upgrade, you will likely see warnings or complete failures because the developers of the CRR feature “reused” existing metric names. So there are now two different metrics, both with the same basic name of backlog_bytes<\/code>. To distinguish between these, I had to modify the collector code. You will now see names like ibmmq_nha_backlog_bytes<\/code> (for the local HA replication) and ibmmq_nha_recovery_backlog_bytes<\/code> (for the recovery group).<\/p>\n\n\n\n

The metrics.txt<\/a><\/code> file in the repository shows the complete set of metrics that are available, including the new CRR values which are in the RECOVERY<\/code> part of the NHAREPLICA<\/code> class. The metrics published on system topics are also documented here<\/a>. <\/p>\n\n\n\n

Class: NHAREPLICA\n  Type: RECOVERY\n    Backlog average bytes\n    Backlog bytes\n    Compressed log bytes sent\n    Log data average compression time\n    Log bytes decompressed\n    Log data average decompression time\n    Average network round trip time\n    Rebase count\n    Recovery log sequence number\n    Log bytes sent  \n  Type: REPLICATION\n    Log write average acknowledgement latency\n    Acknowledged log sequence number\n    Log write average acknowledgement size\n    Backlog average bytes\n    Backlog bytes\n    Catch-up log bytes sent\n    Catch-up compressed log bytes sent\n    Catch-up log data average compression time\n    Catch-up log bytes decompressed\n    Catch-up log data average decompression time\n    Catch-up uncompressed log bytes sent\n    MQ FDC file count\n    Log file system - free space\n    Log file system - bytes in use\n    Average network round trip time\n    Queue Manager file system - free space\n    Queue Manager file system - bytes in use\n    Synchronous log bytes sent\n    Synchronous compressed log bytes sent\n    Synchronous log data average compression time\n    Synchronous log bytes decompressed\n    Synchronous log data average decompression time\n    Synchronous uncompressed log bytes sent<\/pre>\n\n\n\n
I’ve also updated the example dashboard Logging.json<\/code>to demonstrate:<\/p>\n\n\n\n
\"CRR<\/a>
Native HA metrics: HA replication and CRR recovery examples<\/figcaption><\/figure>\n\n\n\n
You can see the replication between the local HA instances (the 2 standby instances), and the data transfer to the recovery group.<\/p>\n\n\n\n
This particular dashboard shows a Recovery LSN value. Ideally the panel would displayed it in the same fashion as the queue manager output. For example, as <000:111:222:333><\/code>. Unfortunately Grafana doesn’t have a way to reformat an integer in such a specific way. Maybe it will<\/a>, one day. But at least you can see the 64-bit number moving.<\/p>\n\n\n\n
MQPerfck metrics<\/h3>\n\n\n\n
Although separate from the CRR metrics, another tool arrives at the same time. The mqperfck<\/a><\/code>command is primarily intended to capture some key metrics for analysis by L2\/L3 support engineers. But it also brings in a similar problem to  CRR for monitoring solutions: duplication of metric names. And so the mq-golang<\/a> v5.6.2 release recognises and handles these names.<\/p>\n\n\n\n
One block of published metrics added to the queue manager to support this tool is the EXTENDED type. These are very specialised, and not likely to be of use to most administrators. So, by default, the collectors do not report them. If you find you really need them, then you can set the queueSubscriptionSelector<\/code>filter array in the collector configuration to include these EXTENDED values. Here, we can see the actual metrics that the amqsrua<\/code> sample reports:<\/p>\n\n\n\n
$ amqsrua -m QM1\nCPU : Platform central processing units\nDISK : Platform persistent data stores\nSTATMQI : API usage statistics\nSTATQ : API per-queue usage statistics\nSTATAPP : Per-application usage statistics\nEnter Class selection\n==> STATQ\nOPENCLOSE : MQOPEN and MQCLOSE\nINQSET : MQINQ and MQSET\nPUT : MQPUT and MQPUT1\nGET : MQGET\nGENERAL : General queue metrics\nEXTENDED : Extended queue metrics\nEnter Type selection\n==> EXTENDED<\/span><\/strong>\nAn object name is required for Class(STATQ) Type(EXTENDED)\nEnter object name\n==> Q1\nPublication received PutDate:20250207 PutTime:10110583 Interval:11 days,0 hours,59 minutes,34.944 seconds\nQ1   msg search count 0\nQ1   msg not found count 0\nQ1   msg examine count 0\nQ1   intran get skipped count 0\nQ1   intran put skipped count 0\nQ1   selection mismatch count 0\nQ1   correlid mismatch short count 0\nQ1   correlid mismatch long count 0\nQ1   msgid mismatch count 0\nQ1   load msg dtl count 0<\/pre>\n\n\n\n
Conclusion<\/h3>\n\n\n\n
Two new MQ features introduce corresponding metrics. I’ve shown here how they can be collected and processed. I hope they’re useful.<\/p>\n\n\n\n
Update History<\/h5>\n\n\n\n