{"id":396,"date":"2018-05-08T22:44:41","date_gmt":"2018-05-08T21:44:41","guid":{"rendered":"https:\/\/marketaylor.synology.me\/?p=396"},"modified":"2019-11-25T09:48:17","modified_gmt":"2019-11-25T09:48:17","slug":"smf-reports-for-mq-with-json-format","status":"publish","type":"post","link":"https:\/\/marketaylor.synology.me\/?p=396","title":{"rendered":"SMF reports for MQ with JSON format"},"content":{"rendered":"

On z\/OS, MQ produces a lot of information about its usage through SMF records. One tool I published on GitHub<\/a> a couple of years ago was designed to take collected SMF data and process it into a more consumable format. The goal was to create something that did nothing more than format the data, rather than try to analyse it in the manner of SupportPac MP1B<\/a>. The analysis would be done using independent tools.<\/p>\n

The comma-separated-value format is ideal for importing into spreadsheets and databases. Many of our z\/OS customers have been helped by reports created from their data, and analysed through SQL queries.<\/p>\n

While I didn’t write about it in a blog post then, I did publish a short video<\/a> showing it in action. And since the original release, I’ve made regular updates to the code. Mostly these have been to simplify the SQL processing.<\/p>\n

I’ve now updated the program to give a further output format option. It is intended to make it easy to feed SMF information to other analytics tools, many of which work with JSON structures. Other aspects of MQ administration can now be done with JSON-format data – the error logs on Distributed platforms<\/a>, an event formatter<\/a> (which is also available in MQ V9.0.5), the REST APIs for administration<\/a> and messaging<\/a> – and this is one further piece of a consistent story.<\/p>\n

<\/p>\n

Getting the SMF formatter<\/h2>\n

The formatting program is available on GitHub<\/a> with both source code and executable versions in the tree. You can extract the single program for your preferred platform (Linux, AIX and Windows are provided) or you can clone the entire tree and build it yourself. Documentation is also provided.<\/p>\n

Running the program<\/h2>\n

To get the JSON output, download the collected raw SMF data to your Linux or Windows machine. Then run<\/p>\n

mqsmfcsv -i <input file> -f json \n<\/pre>\n
This will write the data to a single file in the current directory. You might prefer to stream directly to another processing program, and using “-o-” as a further parameter sends the output to stdout<\/em> for easy piping.<\/p>\n
Example JSON output<\/h2>\n
This shows the start of formatting one of the sample SMF files provided with the tool. Each record is turned into a single structure.<\/p>\n
{\n\"recordType\" : 116,\n\"recordSubType\" : 0,\n\"structure\" : \"QMAC\",\n\"date\" : \"2015\/11\/23\",\n\"time\" : \"11:00:00.020000\",\n\"lpar\" : \"H019\",\n\"qmgr\" : \"MQPC\",\n\"mqVersion\" : \"800\",\n\"authorisationId\" : \"IMS     \",\n\"correlId\" : \"F0F2F3F6C2C3F1E4C4D6C340\",\n\"connectionName\" : \"PRDC    \",\n\"operatorId\" : \"PLN1231 \",\n\"applicationType\" : \"IMS MPP\/BMP\",\n\"accountingToken\" : \"00000000000000000000000000000000000000000000\",\n\"networkId\" : \"D7D9C4C340404040044E0A0800000001\",\n...\n<\/pre>\n
Future thoughts<\/h2>\n
A recently-added feature in z\/OS is the ability to collect SMF data dynamically<\/a>, to enable more real-time processing. It would be nice to collect the MQ records immediately, to give live status information. I’ve designed the mqsmfcsv tool with that in mind, but it’s something that I want to think more about for the future.<\/p>\n
This post was last updated on November 25th, 2019 at 09:48 am<\/p>","protected":false},"excerpt":{"rendered":"
Turn MQ SMF data into JSON for simple consumption and analytics. Use a free open source tool for Windows and Linux to process your mainframe statistics.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5],"tags":[35,38,20,45,30],"class_list":["post-396","post","type-post","status-publish","format-standard","hentry","category-mq","tag-ibmmq","tag-json","tag-mqseries","tag-smf","tag-z-os"],"yoast_head":"\nSMF reports for MQ with JSON format - Mark Taylor's Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/marketaylor.synology.me\/?p=396\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SMF reports for MQ with JSON format - Mark Taylor's Blog\" \/>\n<meta property=\"og:description\" content=\"Turn MQ SMF data into JSON for simple consumption and analytics. Use a free open source tool for Windows and Linux to process your mainframe statistics.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/marketaylor.synology.me\/?p=396\" \/>\n<meta property=\"og:site_name\" content=\"Mark Taylor's Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-08T21:44:41+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-25T09:48:17+00:00\" \/>\n<meta name=\"author\" content=\"Mark\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@marketaylor\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mark\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396\"},\"author\":{\"name\":\"Mark\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\"},\"headline\":\"SMF reports for MQ with JSON format\",\"datePublished\":\"2018-05-08T21:44:41+00:00\",\"dateModified\":\"2019-11-25T09:48:17+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396\"},\"wordCount\":439,\"commentCount\":0,\"keywords\":[\"ibmmq\",\"json\",\"mqseries\",\"smf\",\"z\\\/os\"],\"articleSection\":[\"IBM MQ\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396\",\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396\",\"name\":\"SMF reports for MQ with JSON format - Mark Taylor's Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#website\"},\"datePublished\":\"2018-05-08T21:44:41+00:00\",\"dateModified\":\"2019-11-25T09:48:17+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/?p=396#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/marketaylor.synology.me\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SMF reports for MQ with JSON format\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#website\",\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/\",\"name\":\"Mark Taylor's Blog\",\"description\":\"Messaging, Music and Moving Around\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/marketaylor.synology.me\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/marketaylor.synology.me\\\/#\\\/schema\\\/person\\\/2d6f4113ff54187023e20c20186bbb3c\",\"name\":\"Mark\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g\",\"caption\":\"Mark\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/marketaylor\"],\"url\":\"https:\\\/\\\/marketaylor.synology.me\\\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SMF reports for MQ with JSON format - Mark Taylor's Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/marketaylor.synology.me\/?p=396","og_locale":"en_GB","og_type":"article","og_title":"SMF reports for MQ with JSON format - Mark Taylor's Blog","og_description":"Turn MQ SMF data into JSON for simple consumption and analytics. Use a free open source tool for Windows and Linux to process your mainframe statistics.","og_url":"https:\/\/marketaylor.synology.me\/?p=396","og_site_name":"Mark Taylor's Blog","article_published_time":"2018-05-08T21:44:41+00:00","article_modified_time":"2019-11-25T09:48:17+00:00","author":"Mark","twitter_card":"summary_large_image","twitter_creator":"@marketaylor","twitter_misc":{"Written by":"Mark","Estimated reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/marketaylor.synology.me\/?p=396#article","isPartOf":{"@id":"https:\/\/marketaylor.synology.me\/?p=396"},"author":{"name":"Mark","@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c"},"headline":"SMF reports for MQ with JSON format","datePublished":"2018-05-08T21:44:41+00:00","dateModified":"2019-11-25T09:48:17+00:00","mainEntityOfPage":{"@id":"https:\/\/marketaylor.synology.me\/?p=396"},"wordCount":439,"commentCount":0,"keywords":["ibmmq","json","mqseries","smf","z\/os"],"articleSection":["IBM MQ"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/marketaylor.synology.me\/?p=396#respond"]}]},{"@type":"WebPage","@id":"https:\/\/marketaylor.synology.me\/?p=396","url":"https:\/\/marketaylor.synology.me\/?p=396","name":"SMF reports for MQ with JSON format - Mark Taylor's Blog","isPartOf":{"@id":"https:\/\/marketaylor.synology.me\/#website"},"datePublished":"2018-05-08T21:44:41+00:00","dateModified":"2019-11-25T09:48:17+00:00","author":{"@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c"},"breadcrumb":{"@id":"https:\/\/marketaylor.synology.me\/?p=396#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/marketaylor.synology.me\/?p=396"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/marketaylor.synology.me\/?p=396#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/marketaylor.synology.me\/"},{"@type":"ListItem","position":2,"name":"SMF reports for MQ with JSON format"}]},{"@type":"WebSite","@id":"https:\/\/marketaylor.synology.me\/#website","url":"https:\/\/marketaylor.synology.me\/","name":"Mark Taylor's Blog","description":"Messaging, Music and Moving Around","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/marketaylor.synology.me\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/marketaylor.synology.me\/#\/schema\/person\/2d6f4113ff54187023e20c20186bbb3c","name":"Mark","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9a5ae091c43730194cba7cabb5d65c1dc3f48d05caaddec6ff2319a1ce66376f?s=96&d=mm&r=g","caption":"Mark"},"sameAs":["https:\/\/x.com\/marketaylor"],"url":"https:\/\/marketaylor.synology.me\/?author=1"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/396","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=396"}],"version-history":[{"count":4,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/396\/revisions"}],"predecessor-version":[{"id":561,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=\/wp\/v2\/posts\/396\/revisions\/561"}],"wp:attachment":[{"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=396"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=396"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marketaylor.synology.me\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=396"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}