Troubleshooting Maximo JVM Out-of-Memory error with Heap Analyzer

Occasionally, Maximo became unavailable for a short period of 5-10 minutes. Alarms were raised, IT help desk was called, and the issue got escalated to the Maximo specialist (you). You logged into the server, checked the log file, and found a Java Out-of-Memory (OOM) issue. Not a big deal, the server usually restarted itself and became available soon after that. You reported back to the business and closed the issue. Does that scenario sound familiar to you?

If such an issue has only occurred to your system once, it was probably treated as a simple problem. But since you had to search for a solution on the web and ended up here, reading this article, probably it has occurred more than once, the business requires it to be treated as a critical incident. As the Maximo specialist, you’ll need to dig deeper to report the root cause of the issue and provide a fix to prevent it from occurring again. Analyzing low level Java issue is not an easy task, and this post describes my process to deal with this issue.

On version control and automating the deployment of Maximo configuration

This post includes some of my notes on using DBC for deployment of Maximo configuration. In case you wonder why using DBC, the short answer is, if you’re happy with whatever method you’re using to deploy configuration, whether it is manual or using Migration Manager, ignore this post. But if you’re looking for a way to streamline development process for a large team by collaborating and source controlling using GIT, or if you want to fully automate the deployment process, DBC is the way to go.

IBM has been using DBC script for a long time, but only recently, they published a reference guide so third 3rd party consultants like us can use. DBC Script can be used to automate most of the common configuration for Maximo. It has standard commands to create/modify common low-level objects like table, index, domains etc. For many other configurations which don’t have a specific DBC command, we still can handle the deployment using the <freeform> or <insert> statement to put in anything into Maximo DB. Below are some specific notes on certain type of changes:

Send email from automation script

Simple stuff but I got a few people asked me this same question, so here is how to create an automation script to send email from Maximo:

1 - Create a Communication Template:

  • Description: Test Communication Template
  • Applies To: ASSET

Maximo 7.6 feature - Denial of Service attack

I didn't know about this new feature in Maximo 7.6 until today. Here is the problem: an user reported he's unable to log into Maximo with "BMXAA7901E - You cannot log in at this time" error. Both the Maximo Admin and I could log in using the same userid and password without any problem. After some investigation, it turned out that the user's IP address has been blocked.

This is a new feature in Maximo 7.6 as described by IBM here and here by Mark Robbins.

What interesting is, by looking into the default Maximo's settings, an IP will only be blocked if there are more than 50 failed login attempts made in less than 30 seconds. So it's not possible for a normal user to be blocked by this mechanism.

Manipulating Chain of Java Extensions for the upgrade/installation process

I recently had to upgrade a not too big but quite complex system. The original environment includes Maximo and ICD, and two large customization packages, one extended the other (let’s call them package XXX extended by package YYY). The target system is the latest Maximo + ICD 7.6.1, plus 4 add-ons which include Oil & Gas and Utilities.

Customization were written by 3 different third parties over a long period of time and the source code was lost. This posed some challenges related to preserving customization and I had to spend a bit of time to figure it out. Below are some of the gotchas I learnt after the project:

Problem 1: Ensure customization are preserved after upgrade

After reviewing the SMP folder, I found about 300 extended java class files, but the product.xml files only cover about 20-30% of them; worse, some information are not even up-to-date. After the initial attempt to correct these files, I decided to simply ignore them, and build new product.xml files from scratch. Below are some of the key steps I had to do:

Performance Issue after Upgrade and Installation of Oil & Gas and Utilities Add-ons

Recently I involved in a project where we need to upgrade Maximo to the latest version and install several big add-ons which include SP, HSE (which is Oil & Gas), and Spatial (which is the same as Utilities). This is the first time I see a system with that many add-ons installed. The DB in use is SQL Server.

Although we’ve done all recommended steps for the upgrade like running integrity checks before and after the upgrade; and updating statistics and rebuilt all indexes. After the upgrade, it was still a lot slower compared to the before upgrade version. To analyze the problem, I tested and compared the performance of several different queries on the WORKORDER table and the TICKET, some intended to use indexes, some intended to create a full table scan. With the queries that use more indexes, the performance gap seems to be smaller (upgraded version is 2-3 times slower); and the one which requires full table scan is significantly slower (10-20 times slower).

With that result, we concluded that the slower performance is due to

Blank parameter page when trying to run report from Eclipse/BIRT Designer

The issue with Eclipse/BIRT designer showing a blank parameter page when we try to run a report has been around for a while. This problem is due to newer versions of IE installed on Windows. When I was using an older Windows version, I can simply uninstall IE to restore it to an original, older version. Recently, I had to switch to Windows 10 which comes with IE11 and uninstalling IE is not possible. The problem is described here in this IBM technote

After trying the various solutions found on the web, the following seems to work for me:

Start Center - Result Set not displaying

When creating Start Center Template, I often run into the issue of users not being able to see data in result sets with the error 'This portlet has not been setup'. I often wasted a bit of time to figure it out and then forget about it. And, then will run into the same issue again and again. So I will post a check list here for my own future reference. Hopefully it will help some of you Maximo technies out there.

To allow the user to access the data, and generate start center from the template, the user must be given the following access:

  • 'Read' access to the main application. For example Work Order Tracking if the result set is for listing work order.
  • 'Run Report' and 'Create Report' signature option in the main application. If the main app is a custom app, you'll need to create the same Signature Option in App Design first.
  • Access to 'All Site' or at least one Site.
  • Access to the 'Layout and Configuration' application
  • Access to the Report Object structure used by the result set (which is given in the 'Report Administration' app > Select Action > Set Report Object Structure Security

Process Inbound Twilio SMS message in Maximo

In the last post, I uploaded some demos on how we can process inbound text messages to create SR and route Workflow. There are a few people on LinkedIn asked me how to do it. I also like to explore the option of integrating Maximo with Twilio directly without the need for App Connect. With Maximo 7.6, we can create a simple API using automation script. So, the video below is how we can create a simple script API and direct Twilio HTTP requests to Maximo:

Interacting with Maximo via SMS

Voice call and SMS are being used as a user interaction interface in many systems and services provided by large organizations. However, in the past, this integration is quite complex and expensive, and thus, I never had a chance to play around with. Recently, I posted an article on how we can easily configure Maximo to send notification as SMS message. This is made easy and (free for developers) by using Twilio. 

Sending inbound command to a system from Twilio is more complex as it requires setting up of a Web Server and writing code to parse HTTP POST data and forward the message to an on-premise Maximo system. With App Connect, it becomes easy (and free for developers) to set up an API gateway on cloud and routing message to a system hosted in-house. So I decided to give it a try by imagine a two easy-to-achieve use cases as follow:
  • Field workers sending a text message to Maximo to create a new Work Request, report Incident, or Defect. This could be useful to quickly register an action item without having a smart device or 3G connection. Below is a quick demo of this scenario: