After recovering from a site-wide power failure and restart, something then went wrong with my AIX system so that I could no longer log into it. It seemed that the ssh service had died. And none of the other standard Unix services like telnet were enabled for security reasons. I’ll show how I was able to recover, and how an obsolete MQ channel option saved the day.
Since this is a virtual machine, where I don’t have access to the “real” console, how could I recover? Possibly raising a helpdesk ticket would get someone involved who could restart the system in maintenance mode and then solve whatever caused the problem. But that was not guaranteed.
Investigating the pieces
The only normal service I could still use was ftp. That was enough to be able to put and get files. So I thought about editing /etc/inetd.conf to reenable telnetd temporarily. But on AIX, that file is only read at startup or when a command runs to force it to be reread. So not helpful. And I couldn’t issue any commands remotely – just as telnet was disabled, so was rexec.
What I noticed however when I looked at the copied inetd.conf file, was that I did have an old-style MQ channel responder configured. The original MQ design for Unix systems did not have a product-provided listener. Instead, you used inetd.conf to define and start an instance of the amqcrsta program – inetd being the general system-wide listener, and amqcrsta being a real MQ channel program. A new instance of amqcrsta was started by the daemon for each inbound channel. The MQ listener program runmqlsr was introduced to Unix systems after just a couple of releases but some people continued to use the inetd configuration because that was what they were used to. I think that I might have simply been checking that it still worked, even at MQ 7.1 which was what I had defined in the inetd configuration.
So I could do telnet <system> 1212 and get connected to the channel – that connectivity was still working even if seemingly useless from the perspective of restarting sshd. Then I realised that the directory where I had installed that old version of MQ was mountable to another machine over NFS. And because I have a very permissive environment within my own systems, I had root authority over that remote filesystem. I also had sudo configured on the AIX box for passwordless access for the mqm account. All these pieces gave me an idea.
The solution
“All” I needed to do was to replace the amqcrsta binary with another program – a shell script – that contained the command to respond as a telnet session. Basically it looked like
#!/bin/sh sudo /usr/sbin/telnetd $*
Then use telnet to connect to the port, and instead of starting an MQ channel process, my shell script came up, and I got a basic command line that let me log in and do some proper recovery. I still don’t know why sshd originally died, but I’ve added configuration to auto-restart it if it fails in future. And I’ve restored the original version of amqcrsta.
Conclusion
This might sound like a security vulnerability, but it’s not really because of all the things I had setup already that gave me so much remote power across a small number of platforms. But I thought it was interesting how something that was deprecated so many years ago could be subverted.
This post was last updated on February 24th, 2022 at 11:10 am