Message Engine doesn't start after setting up cluster

This issue hit me a few times and always took me some time to figure out what happened. So I thought it's a good idea to note it down.

Symptom:

When setting up cluster environment for Maximo, I will need to setup an integration bus with a message engine for each cluster (IF, UI, Cron etc.)

Each message engine will require its own individual schema (and thus individual user if the Oracle DB is used)

After integration bus are setup and Maximo cluster started, we see a lot of errors in the log file, usually in the Cron or MIF cluster due to message engine is not available.

Issues with Maximo attachment (DOCLINKS)

 Just a bunch of my own personal notes regarding Maximo attachment (DOCLINKS) function.

•  When a file is attached to a record in Maximo, it creates a record in the DOCLINKS and a record in the DOCINFO table to keep the details of the file. The file is copied to a location on disk, usually on a local folder (e.g. D:\DOCLINKS) or on a network shared folder. The path for Maximo to read the file is kept in the URLNAME field of the DOCINFO table.
• To setup this function, refer to this blog post by Bruno on MaximoDev blog (https://bportaluri.com/2014/06/attachments-doclinks-configuration.html)

• When a user uploads an attachment, the location where Maximo puts the file will depend on the Folder selected:

How to test SMTP with PowerShell for setting up Maximo email notification

In an enterprise IT environment, it is sometimes difficult to setup Maximo to talk with SMTP service due to networking and security restrictions. To troubleshoot SMTP configuration, in the past, we can use Telnet from CMD tool. However, in newer versions of Windows Server, Telnet is often not installed by default. In such cases, we can use PowerShell to test and confirm the SMTP and firewall setting is working before trying to configure SMTP in Maximo.

The first thing I would do is checking whether firewall has been opened so that the port used by SMTP service (e.g. port 25) is reachable from the Maximo server.

Messaging engine cannot be started as there is no runtime initialized for it yet

I sometimes have issues with message engine not running. Usually I'll just try to restart the whole system and hope that it goes away.

If it doesn't work, in most cases, it is caused by a corrupted file store used by the message engine and the suggestion from the Internet is to delete these files, which seems to work fine.

Sometimes, with the message engine uses a database store, I had a very similar issue. I find it quite hard to find out the exact root cause. So I chose the easier path by simply deleting the whole message engine, create a new one, giving a new schema name for the data store. This ensures it creates new tables when message engine is initialized the first time. 

Creating a new message engine and re-assigning bus destinations usually take less than 5 minutes, and it seems a lot easier than troubleshooting and finding the root cause of the issue.

Redeploy a single web.xml file

Most Maximo settings or java code can be deployed by copy/pasting the file directly to the installed folder in Websphere without having to rebuild and redeploy the application. However, with web.xml, it doesn't work that way. Sometimes, we need to update this file to increase timeout setting or enable/disable integrated security mode

Sure, we can directly modify the file in Websphere without redeployment, but we will also have update the file in a few temporary folders for which, I find the process quite tedious.

Recently, my colleague told me we can just deploy the single web.xml file instead. Below is the process:

- Update the web.xml file with new settings

Node Agent cannot start after configuring LDAP

Just to document a weird issue I had today. I was attempting to configure LDAP (Microsoft AD) for Maximo/Websphere. After it is configured (and I've tested that it can query AD data), Application Server security was enabled. Then the server is rebooted to refresh the new configuration.

Enabling up HTTP Compression for Maximo

To enable HTTP compression for Maximo, follow the steps below:

• Stop HTTP server
• Make a backup copy of \IBM\HTTPServer\conf\httpd.conf
• Edit httpd.conf:
• Enable the following two lines:

Troubleshoot integration issues when Maximo stops publishing data to external system.

I had to deal with this quite often. Most of the times, I got it right and able to identify the problem quickly. In a few cases, it took some time, and usually very stressful as it mostly occurs in Production. (It occurs in DEV and PRE-PROD all the time, it’s just that people usually don’t care, and it goes unnoticed)

Today I had to deal with it again and it took me some time. The cause was something I dealt with before, was told by a colleague on how to fix it (the easy way), but I forgot. This time around, under panic mode, I restarted a few JVMs before I remembered I should ask around and was reminded by my colleague again that it could be fixed with much less damage. I told myself I should write it down for the next time, so here is the sum of what I learned:

Check Network/Firewall Status using PowerShell

Working with Maximo, we have to deal with network/firewall all the time. I can spend a whole day telling you fun stories (or extremely frustrating experience) I had with IT/Network guys. But that's not the point. If you end up here, I guess you're having an issue with firewall. So below is a quick recap of my tricks:

- Ping: the first thing we try when we want to see if a server is accessible from our current machine. But don't jump to a conclusion too early if it shows the server is unreachable. Sometimes, the Network Admin has disabled ping response to enhance security. 

- Telnet: to check if a port is opened, we can use telnet from Windows Command console (e.g. telnet google.com 80). If it can be connected, means the target server is accessible on the specified port. But if it doesn't, first, make sure the server is listening on that port, before jumping to a conclusion that it's a Firewall issue. I made this mistake a few times, blaming the network, then it turned out it's Websphere/Maximo service is not running on the port that I assumed it should be running on

- PowerShell: in many cases, the server is not connected to the Internet, and Telnet is not installed, (and yes, you don't have permission to install new software either). We can use PowerShell to check network connectivity using the two commands below:

$connection = New-Object System.Net.Sockets.TcpClient("google.com", 80)

Write-Host $connection.Connected

The first line will return some errors if it cannot connect to the server, like the screenshot below: 

 If the server is accessible via the provided IP and port, the 2nd line will return the status = Connected

Mess around with Azure: Migrate Maximo to Cloud

Last week, while attending a call to discuss an integration interface between Maximo and an Azure SQL database, the other consultant mentioned a few Azure terms like “blob storage”; I realized I didn’t know anything about Azure despite the fact that the cloud platform has been adopted by the large number of the clients I work with. So today, I decided I should play around a bit with it by trying to migrate a Maximo instance from my local VM to Azure.

Before I go into the technical details, for those of you non-technical readers, I like to sum up the whole experience I had with this platform in one word: magical. That’s it, I went to the website, signed up for a trial account, followed some wizards or clicked on a few buttons or menus that I think were features I need, and boom, I got a running Maximo instance on the cloud in just a few hours. No training, no muss, no fuss, everything just works. Not much else to say. I’m in love now. I think will spend the next couple of weeks, months or even years to learn more about Microsoft stuff.

Troubleshooting Maximo JVM Out-of-Memory error with Heap Analyzer

Occasionally, Maximo became unavailable for a short period of 5-10 minutes. Alarms were raised, IT help desk was called, and the issue got escalated to the Maximo specialist (you). You logged into the server, checked the log file, and found a Java Out-of-Memory (OOM) issue. Not a big deal, the server usually restarted itself and became available soon after that. You reported back to the business and closed the issue. Does that scenario sound familiar to you?

If such an issue has only occurred to your system once, it was probably treated as a simple problem. But since you had to search for a solution on the web and ended up here, reading this article, probably it has occurred more than once, the business requires it to be treated as a critical incident. As the Maximo specialist, you’ll need to dig deeper to report the root cause of the issue and provide a fix to prevent it from occurring again. Analyzing low level Java issue is not an easy task, and this post describes my process to deal with this issue.

Deploy Maximo 7.6.1 on Websphere 8.5

Out of curiosity, I installed Maximo 7.6.1 on a VM to explore the new features. Then I found that I kind of like it, so I wanted to deploy it on my host OS to get rid of the VM. The trouble is I have Websphere 8.5.5.3 which comes with Maximo 7.6.0.0. I can run maxinst.bat to create demo data with Oracle 11g without a hiccup, but the deployment of the EAR file to Websphere didn't go well.

It turned out Maximo 7.6.1 requires Java 1.8 and Websphere 9. It's not possible to upgrade Websphere from version 8 to 9. We will have to install a new Websphere environment and migrate all of the settings which sounds like a lot of work. I decided to tweak it a little bit to make Maximo 7.6.1 runs on Websphere 8.5 using the following process:

OutOfMemory problem when deploying Maximo

For every client I work with, I always keep a copy of the SMP folder if I can. And every time I need a standard demo instance with exactly the same configuration with the client, I simply deploy it on my local Websphere and Oracle DB.

My current websphere version is 8.5.5.3, and I had problem whenever I deploy a Maximo 7.6 instance with lots of add-ons. I gave up the attempt in the last few times as I didn’t really need it. But recently, as I was working with Maximo Flex (on cloud), having a local instance is a must as Maximo on cloud has lots of limitation in term of what you can do. As such, I when I had the same problem deploying Maximo, I had to investigate and fix the issue. Turned out it is an OutOfMemory issue during the deployment process and it took me quite a bit of time to fix it, so I leave the note here as I’m sure many other will have similar problem.