{"id":1974,"date":"2026-11-10T09:19:44","date_gmt":"2026-11-10T09:19:44","guid":{"rendered":"https:\/\/marketaylor.synology.me\/?p=1974"},"modified":"2026-11-10T11:30:15","modified_gmt":"2026-11-10T11:30:15","slug":"publish-events-to-a-kafka-topic","status":"publish","type":"post","link":"https:\/\/marketaylor.synology.me\/?p=1974","title":{"rendered":"Publishing MQ Events to Kafka"},"content":{"rendered":"\n

Last year, I released a program<\/a> that would read MQ Event messages, format them, and send them directly to an OpenTelemetry backend. I wrote about it here<\/a>. I’ve now extended it to also publish events to a Kafka topic.<\/p>\n\n\n\n\n\n\n\n

Introduction<\/h3>\n\n\n\n

The amqsevtg<\/code> (‘g’ for Go – the language it’s written in) program was effectively a clone of the product-shipped amqsevt<\/code>, with the added capability of configuring an OTel backend. So you don’t need to use an intermediate procedure like sending output via a named pipe or having the OTel Collector program read from a redirected file.<\/p>\n\n\n\n

But a recent question asked if it was possible to send MQ Events to Kafka. Although the amqsevt<\/code> program could handle it similarly to the OTel situation by writing to stdout <\/em>and then piping that to a Kafka-specific program reading stdin<\/em>, it’s not the simplest path to setup. I got it working using the kafka-console-producer<\/code> program (the equivalent of amqsput<\/code>).<\/p>\n\n\n\n

But how tricky would it be to do a more integrated solution?<\/p>\n\n\n\n

Extending amqsevtg<\/h3>\n\n\n\n

Quite easy, as it turned out. It probably took about an hour to modify the program, and to start to test it.<\/p>\n\n\n\n

There was already a well-encapsulated output path to OTel; a similar path to reach Kafka could go alongside. Configuration options can name the broker address and topic (I didn’t add more features like TLS configurations). And that’s essentially all.<\/p>\n\n\n\n

The Apache Kafka project does not have a native Go client; most things there are done in Java. But there are several alternative<\/a> implementations available. I picked the IBM Sarama<\/a> package as it is a Go-native option. <\/p>\n\n\n\n

Differences in the consumer implementation options were not relevant, as this program only writes to a single topic, and the Confluent-provided Go client required cgo, as it is just a wrapper around the C client library.<\/p>\n\n\n\n

Why send events to Kafaka?<\/h3>\n\n\n\n

Sending events to something billed as an event streaming solution feels like a natural option. And there are then a range of analytics programs that can read the events from Kafka and generate reports or potentially take action. <\/p>\n\n\n\n

It’s not really any different than working with an OTel-based stack, but the choice will probably depend on what you already have in place.<\/p>\n\n\n\n

Running the program<\/h3>\n\n\n\n

Once I had a Kafka broker running, I defined the mqevents<\/code> topic that I was planning to use.<\/p>\n\n\n\n

Then run something to read messages from that topic:<\/p>\n\n\n\n

kafka-console-consumer.sh -bootstrap-server localhost:9092 --topic mqevents<\/pre>\n\n\n\n
And finally, start the formatter against a queue manager. The compact<\/code> option causes single lines of output:<\/p>\n\n\n\n
go build -o amqsevtg .
.\/amqsevtg -kb localhost:9092 -kt mqevents -o compact -m EVENTS<\/pre>\n\n\n\n
The default maximum message size for Kafka is 1MB, which is easily large enough for event messages. <\/p>\n\n\n\n
The output is what you would expect. JSON-formatted event messages for easy parsing by other programs:<\/p>\n\n\n\n
\"eventSource\": {
  \"objectName\": \"SYSTEM.ADMIN.QMGR.EVENT\",
  \"objectType\": \"Queue\",
  \"queueManager\": \"EVENTS\"
},
\"eventType\": {
  \"name\": \"Queue Mgr Event\",
  \"value\": 44
},
\"eventReason\": {
  \"name\": \"Put Inhibited\",
  \"value\": 2051
},
\"eventCreation\": {
  \"timeStamp\": \"2026-11-10T09:06:43.06Z\",
  \"epoch\": 1794301603
},
\"eventData\": {
  \"applName\": \"amqsput\",
  \"applType\": \"Unix\",
  \"qMgrName\": \"EVENTS\",
  \"qName\": \"X\"
}<\/pre>\n\n\n\n
Summary<\/h3>\n\n\n\n
Exploring different options can be fun. I already had a Kafka environment available for other reasons, so this enhancement did not take long to code and test. Maybe it gives you more ideas on how to use these events, which are so important when monitoring an MQ queue manager.<\/p>\n
This post was last updated on November 10th, 2026 at 11:30 am<\/p>","protected":false},"excerpt":{"rendered":"
Last year, I released a program that would read MQ Event messages, format them, and send them directly to an OpenTelemetry backend. I wrote about it here. I’ve now extended it to also publish events to a Kafka topic. This post was last updated on November 10th, 2026 at 11:30 am<\/p>\n","protected":false},"author":1,"featured_media":1977,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5],"tags":[41,35,147,20],"class_list":["post-1974","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mq","tag-events","tag-ibmmq","tag-kafka","tag-mqseries"],"yoast_head":"\nPublishing MQ Events to Kafka - Mark Taylor's Blog<\/title>\n<meta name=\"description\" content=\"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/marketaylor.synology.me\/?p=1974\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Publishing MQ Events to Kafka - Mark Taylor's Blog\" \/>\n<meta property=\"og:description\" content=\"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign\" \/>\n<meta property=\"og:url\" content=\"https:\/\/marketaylor.synology.me\/?p=1974\" \/>\n<meta property=\"og:site_name\" content=\"Mark Taylor's Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-10T09:19:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-11-10T11:30:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png\" \/>\n\t<meta property=\"og:image:width\" content=\"256\" \/>\n\t<meta property=\"og:image:height\" content=\"256\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Mark\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@marketaylor\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mark\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974\"},\"author\":{\"name\":\"Mark\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\"},\"headline\":\"Publishing MQ Events to Kafka\",\"datePublished\":\"2026-11-10T09:19:44+00:00\",\"dateModified\":\"2026-11-10T11:30:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974\"},\"wordCount\":506,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/marketaylor.synology.me\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Apache-Kafka-256.png\",\"keywords\":[\"events\",\"ibmmq\",\"kafka\",\"mqseries\"],\"articleSection\":[\"IBM MQ\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974\",\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974\",\"name\":\"Publishing MQ Events to Kafka - Mark Taylor's Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/marketaylor.synology.me\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Apache-Kafka-256.png\",\"datePublished\":\"2026-11-10T09:19:44+00:00\",\"dateModified\":\"2026-11-10T11:30:15+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\"},\"description\":\"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#primaryimage\",\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Apache-Kafka-256.png\",\"contentUrl\":\"https:\\\/\\\/marketaylor.synology.me\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Apache-Kafka-256.png\",\"width\":256,\"height\":256},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=1974#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/marketaylor.synology.me\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Publishing MQ Events to Kafka\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#website\",\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/\",\"name\":\"Mark Taylor's Blog\",\"description\":\"Messaging, Music and Moving Around\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/marketaylor.synology.me\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\",\"name\":\"Mark\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"caption\":\"Mark\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/marketaylor\"],\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Publishing MQ Events to Kafka - Mark Taylor's Blog","description":"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/marketaylor.synology.me\/?p=1974","og_locale":"en_GB","og_type":"article","og_title":"Publishing MQ Events to Kafka - Mark Taylor's Blog","og_description":"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign","og_url":"https:\/\/marketaylor.synology.me\/?p=1974","og_site_name":"Mark Taylor's Blog","article_published_time":"2026-11-10T09:19:44+00:00","article_modified_time":"2026-11-10T11:30:15+00:00","og_image":[{"width":256,"height":256,"url":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","type":"image\/png"}],"author":"Mark","twitter_card":"summary_large_image","twitter_creator":"@marketaylor","twitter_misc":{"Written by":"Mark","Estimated reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/marketaylor.synology.me\/?p=1974#article","isPartOf":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974"},"author":{"name":"Mark","@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c"},"headline":"Publishing MQ Events to Kafka","datePublished":"2026-11-10T09:19:44+00:00","dateModified":"2026-11-10T11:30:15+00:00","mainEntityOfPage":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974"},"wordCount":506,"commentCount":0,"image":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974#primaryimage"},"thumbnailUrl":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","keywords":["events","ibmmq","kafka","mqseries"],"articleSection":["IBM MQ"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/marketaylor.synology.me\/?p=1974#respond"]}]},{"@type":"WebPage","@id":"https:\/\/marketaylor.synology.me\/?p=1974","url":"https:\/\/marketaylor.synology.me\/?p=1974","name":"Publishing MQ Events to Kafka - Mark Taylor's Blog","isPartOf":{"@id":"https:\/\/marketaylor.synology.me\/#website"},"primaryImageOfPage":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974#primaryimage"},"image":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974#primaryimage"},"thumbnailUrl":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","datePublished":"2026-11-10T09:19:44+00:00","dateModified":"2026-11-10T11:30:15+00:00","author":{"@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c"},"description":"This article shows how a program can format MQ Event messages, and publish the events to a Kafka topic for further processign","breadcrumb":{"@id":"https:\/\/marketaylor.synology.me\/?p=1974#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/marketaylor.synology.me\/?p=1974"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/marketaylor.synology.me\/?p=1974#primaryimage","url":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","contentUrl":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","width":256,"height":256},{"@type":"BreadcrumbList","@id":"https:\/\/marketaylor.synology.me\/?p=1974#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/marketaylor.synology.me\/"},{"@type":"ListItem","position":2,"name":"Publishing MQ Events to Kafka"}]},{"@type":"WebSite","@id":"https:\/\/marketaylor.synology.me\/#website","url":"https:\/\/marketaylor.synology.me\/","name":"Mark Taylor's Blog","description":"Messaging, Music and Moving Around","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/marketaylor.synology.me\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c","name":"Mark","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","caption":"Mark"},"sameAs":["https:\/\/x.com\/marketaylor"],"url":"https:\/\/marketaylor.synology.me\/?author=1"}]}},"jetpack_featured_media_url":"https:\/\/marketaylor.synology.me\/wp-content\/uploads\/2026\/11\/Apache-Kafka-256.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/1974","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1974"}],"version-history":[{"count":8,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/1974\/revisions"}],"predecessor-version":[{"id":1984,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/1974\/revisions\/1984"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/media\/1977"}],"wp:attachment":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1974"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1974"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1974"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}