The curious case of the MIA work orders?

Working in IT, we deal with strange issues all the time. However, every once in a while, something would come up that leaves us scratching our heads for days. One such issue happened to us a few years back. It came back to me recently and this time, I thought to myself I should note it down.

The issue was first reported to us when users raised a ticket about missing work orders in TechnologyOne, the Finance Management System used by our client. Without work orders created in TechOne, the users won't be able to report actual labor time or other costs. Thus, this is considered a high priority issue.

Deployment without downtime?

Downtime is costly to the business. As developers, avoiding it can give us a ton of benefits both in term of efficiency and for personal well-being as well. For example, when making changes that require downtime to a shared environment, I have my freedom back since I don’t have to ask or wait to do it at night. 

With the introduction of Automation Script, most of the business logic and front-end changes we need push to production nowadays can be done without downtime. Some of them are:

• Automation Script
• Escalation
• Application Design

How to run SQL query in Maximo without DB access using API Automation Script

From Maximo version 7.6.0.9, we can now build custom API using automation script. This is a powerful new feature yet it looks to be underutilized by the community.

The first obviously use case is it gives us the freedom to build any API we wanted without being restricted to the limitation of the Maximo Integration Framework. For example, we can build an API that returns data in CSV or binary format. Or we can use it to upload data and bypassing the business layer.

Since it allows us to use the browser to interact with Automation script, and the script framework itself has access to all Java functions of the MBO layer, We can exploit it to execute all sort of weird operations.

Hot reset sequence without restarting Maximo

One error we often have to deal with is incorrect sequence when adding new data to Maximo. There are many situations when it might come up, such as:

• When loading data using MXLoader, or inserting data directly via SQL
• Sequence corruption due to unknown cause in Production, probably due to errors caused by cancelled/terminated job
• Restoring database from a copy, or after an upgrade.

When this happens, the user sees an error with duplicated key value such as “BMXAA4211E - Database error number 2601 has occurred…”

Use Maximo webservice with JSON content

While the JSON API in newer version of Maximo is quite useful, for many integration scenarios, I still prefer to use the old API infrastructure with Publish Channel and Web Service. However, the native format for this feature is XML.

To send or receive JSON with Publish Channel or Enterprise Service, we can translate the default to JSON format before it goes out / into the system. Below is a simple example to set it up.

Setup standard Publish Channel to send XML message

• Create a new Publish Channel: 

Message Engine doesn't start after setting up cluster

This issue hit me a few times and always took me some time to figure out what happened. So I thought it's a good idea to note it down.

Symptom:

When setting up cluster environment for Maximo, I will need to setup an integration bus with a message engine for each cluster (IF, UI, Cron etc.)

Each message engine will require its own individual schema (and thus individual user if the Oracle DB is used)

After integration bus are setup and Maximo cluster started, we see a lot of errors in the log file, usually in the Cron or MIF cluster due to message engine is not available.

Consider Time Zone with automation script

It took me some time to get to this piece of code, but requirement changed and I needed to ditch it. But I'm sure I'll need to use it at some point in the future, so why not put a note here just in case.

Requirement: 

- A client has many hotels in Sydney and Brisbane which are in two different time zones. Sydney has daylight saving while Brisbane doesn't have. In other words, there is one hour difference for half of the year, and no time difference for the other half.

- Client wants to display a warning message to the user that when he/she raised a service request out of normal working hours (7AM to 7PM), if it's a high priority item, they should make a phone call instead.

Issues with Maximo attachment (DOCLINKS)

 Just a bunch of my own personal notes regarding Maximo attachment (DOCLINKS) function.

•  When a file is attached to a record in Maximo, it creates a record in the DOCLINKS and a record in the DOCINFO table to keep the details of the file. The file is copied to a location on disk, usually on a local folder (e.g. D:\DOCLINKS) or on a network shared folder. The path for Maximo to read the file is kept in the URLNAME field of the DOCINFO table.
• To setup this function, refer to this blog post by Bruno on MaximoDev blog (https://bportaluri.com/2014/06/attachments-doclinks-configuration.html)

• When a user uploads an attachment, the location where Maximo puts the file will depend on the Folder selected:

How to test SMTP with PowerShell for setting up Maximo email notification

In an enterprise IT environment, it is sometimes difficult to setup Maximo to talk with SMTP service due to networking and security restrictions. To troubleshoot SMTP configuration, in the past, we can use Telnet from CMD tool. However, in newer versions of Windows Server, Telnet is often not installed by default. In such cases, we can use PowerShell to test and confirm the SMTP and firewall setting is working before trying to configure SMTP in Maximo.

The first thing I would do is checking whether firewall has been opened so that the port used by SMTP service (e.g. port 25) is reachable from the Maximo server.

Common issues when setting up Maximo with a Load Balancer

Just a couple of my own notes setting up Maximo with a Load Balancer which I learnt through the hard way:

- Property mxe.system.useLoadBalancer – should be set to 1. If not enabled, Maximo thinks the IP address of the Load Balancer’s IP is the client’s and blocks it when the number of requests exceeds a certain threshold (by default is 50 per 3 seconds). 

For more details about the IP blocking function, read this previous post

Messaging engine cannot be started as there is no runtime initialized for it yet

I sometimes have issues with message engine not running. Usually I'll just try to restart the whole system and hope that it goes away.

If it doesn't work, in most cases, it is caused by a corrupted file store used by the message engine and the suggestion from the Internet is to delete these files, which seems to work fine.

Sometimes, with the message engine uses a database store, I had a very similar issue. I find it quite hard to find out the exact root cause. So I chose the easier path by simply deleting the whole message engine, create a new one, giving a new schema name for the data store. This ensures it creates new tables when message engine is initialized the first time. 

Creating a new message engine and re-assigning bus destinations usually take less than 5 minutes, and it seems a lot easier than troubleshooting and finding the root cause of the issue.

Redeploy a single web.xml file

Most Maximo settings or java code can be deployed by copy/pasting the file directly to the installed folder in Websphere without having to rebuild and redeploy the application. However, with web.xml, it doesn't work that way. Sometimes, we need to update this file to increase timeout setting or enable/disable integrated security mode

Sure, we can directly modify the file in Websphere without redeployment, but we will also have update the file in a few temporary folders for which, I find the process quite tedious.

Recently, my colleague told me we can just deploy the single web.xml file instead. Below is the process:

- Update the web.xml file with new settings

Post HTTP request using Automation script

I want to post a simple JSON message to an external system and do not want to add any external library to Maximo as it would require a restart. 

In the past, I used the java HTTPClient library that comes with Maximo, but it would require half a page of boilerplate Jython code. Recently, I found a simpler solution below.

• First I use WebHook as a mock service for testing. Go to webhook.site, it will give us a unique URL to send request to:

Enabling up HTTP Compression for Maximo

To enable HTTP compression for Maximo, follow the steps below:

• Stop HTTP server
• Make a backup copy of \IBM\HTTPServer\conf\httpd.conf
• Edit httpd.conf:
• Enable the following two lines:

The fun (and pain) of Kronos Integration

One of our clients undertook a massive IT transformation program which involved switching to a new financial management system, upgrading and rebuilding a plethora of interfaces among several systems, both internal and external to the business. Kronos was chosen to replace an old timesheet software and there was the need to integrate it with other systems such as Maximo and TechnologyOne. WebMethods was used as the integration tool for this IT ecosystem. This is my first experience with Kronos. The project took almost two years to finish. As always, when dealing with something new, I had quite a bit of fun (and pain) during this project. As it is approaching the final stage now, I think I should write down what I’ve learnt. Hopefully, it will be useful for people out there who’re doing a similar task.

REST API: Kronos provides a pretty good reference source for the REST API at this Link. REST API theoretically offers the advantage of supporting real-time integration and enables seamless workflow. However, we don’t have such a requirement in this project. On the other hand, this has two major limitations.

Troubleshoot integration issues when Maximo stops publishing data to external system.

I had to deal with this quite often. Most of the times, I got it right and able to identify the problem quickly. In a few cases, it took some time, and usually very stressful as it mostly occurs in Production. (It occurs in DEV and PRE-PROD all the time, it’s just that people usually don’t care, and it goes unnoticed)

Today I had to deal with it again and it took me some time. The cause was something I dealt with before, was told by a colleague on how to fix it (the easy way), but I forgot. This time around, under panic mode, I restarted a few JVMs before I remembered I should ask around and was reminded by my colleague again that it could be fixed with much less damage. I told myself I should write it down for the next time, so here is the sum of what I learned:

Setting up alarms for integration

 When writing a piece of software, we are in total control of the quality of the product. With integration, many elements are not under our control. Network and firewall are usually managed by IT. With external systems, we usually don’t know how they work, or many times, not given access. Yet, any changes to these elements can cause our interfaces to fail.

For synchronous interfaces, the user would receive instant feedback after each action is taken (e.g. Maximo - GIS integration), thus, we don’t usually need to setup alarms. For asynchronous interfaces, which usually run in the background, and don’t give instant feedback, when failure occurs, it usually goes unnoticed. In many cases, we only find out about failures after it has caused some major damage.

A good interface must provide adequate mechanism to handle failures, and in the case of async integration, proper alarms and reports should be setup so that failures are captured and handled proactively by IT and application administrators.

Avoiding recursion filter on Publish Channel

The standard way to send a message from Maximo to an external system is by setting up a Publish Channel and enabling Event Listener. By default, Integration Framework doesn’t re-publish a change if it comes from another inbound interface to prevent recursion on a bi-directional interface. Although I don’t agree with this logic as one-way integration is much more common, but anyway, IBM said it is easy to override that by extending the Event Filter javaclass.

The problem is, with the rise of automation script, no one wants java customization anymore. Of course, for massive systems where performance is critical, it is still the best choice. But, for most medium-sized clients I work with, they’re all moving away from java customization.

Anyway, an approach we can deal with this is not to use Event Listener at all. Instead, we can trigger a publish from an Object Save launch point from automation script using the example python code below:

from psdi.iface.mic import PublishChannelCache

PublishChannelCache.getInstance().getPublishChannel("PUBLISH_CHANNEL_NAME>").publish(mbo, True)

 

 Happy coding!

Check Network/Firewall Status using PowerShell

Working with Maximo, we have to deal with network/firewall all the time. I can spend a whole day telling you fun stories (or extremely frustrating experience) I had with IT/Network guys. But that's not the point. If you end up here, I guess you're having an issue with firewall. So below is a quick recap of my tricks:

- Ping: the first thing we try when we want to see if a server is accessible from our current machine. But don't jump to a conclusion too early if it shows the server is unreachable. Sometimes, the Network Admin has disabled ping response to enhance security. 

- Telnet: to check if a port is opened, we can use telnet from Windows Command console (e.g. telnet google.com 80). If it can be connected, means the target server is accessible on the specified port. But if it doesn't, first, make sure the server is listening on that port, before jumping to a conclusion that it's a Firewall issue. I made this mistake a few times, blaming the network, then it turned out it's Websphere/Maximo service is not running on the port that I assumed it should be running on

- PowerShell: in many cases, the server is not connected to the Internet, and Telnet is not installed, (and yes, you don't have permission to install new software either). We can use PowerShell to check network connectivity using the two commands below:

$connection = New-Object System.Net.Sockets.TcpClient("google.com", 80)

Write-Host $connection.Connected

The first line will return some errors if it cannot connect to the server, like the screenshot below: 

 If the server is accessible via the provided IP and port, the 2nd line will return the status = Connected

Common use cases for EAM mobile solutions

In some organisations, when starting a mobile project, the stakeholders may not have prior experience with mobility solutions for EAM, as such, we are sometimes asked to implement features that do not add value to the business. As consultants, it is satisfying to see something we implement being used and helps the end-users on the field. And nothing can be more frustrating than spending time building features that are not going to be used. In this post, I will list out based on my experience some commonly used and not used functions of a mobile app:

Work Execution: despite its name is Asset Management software, 80% of activities in Maximo happen around the work management process. It is not a surprise work execution is the number one use-case for a mobile app. However, work management is a big process with several major stages and many different activities. Only certain activities need to be recorded in the field with a mobile device. Below are some common ones: