LCM Import timeout for HPCM artifacts:EPMLCM-13000:Service currently not available

Hi Friends,

Topic: How to fix LCM timeout issue while importing HPCM (Hyperion Profitability and Cost Management) artifacts

In this post, we will see LCM (Lifecycle Management) timeout error (EPMLCM-13000: Service currently not available) and how to fix it while we import HPCM (Hyperion Profitability and Cost Management) artifacts into any Oracle Hyperion environment.

Important Note: 

• This post has been written and associated activities have been demonstrated on Oracle HPCM (Hyperion Profitability and Cost Management) version 11.1.2.4.127.
• Different paths and folders' names mentioned in this blog may slightly vary for different Oracle Hyperion setups but at large it should be the same. 
• It's very important to test any workaround/fix first in a non-Prod environment, and move/apply it to PROD only when you are fully sure about its workability. 
• Never forget to take a complete backup of EVERYTHING before deleting or changing anything.

Issue:

In our Production environment, when we were trying to import HPCM application models, POVs etc. components using LCM, it was getting aborted after 5 minutes throwing following error in migration details page:

EPMLCM-13000: Service currently not available.  

We tried to run the LCM import two more times, but every time it was getting failed after 5 mins.

Analysis:

When we checked SharedService_LCM.log (MIDDLEWARE_HOME/user_projects/domains/EPMSystem/servers/FoundationServices0/logs), we found following error message recorded over there:

2020-05-16T09:04:44.587-06:00] [FoundationServices2] [NOTIFICATION] [EPMLCM-13000] [oracle.EPMLCM] [tid: 815] [userId: <anonymous>] [ecid: 00j15bZ53w2EoItDwfV4CW0003po001Y3p,0:1:4:3:4:3:3:3:3:4:3:3:3:3:4:3:4:4:4:3:3:3:3:3:4:3:4:3:3:3:3:3:3:3:3:3:3:3:3:3:3:4:4:4:3:4:3:3:3:3:3:4:3:3:3:3:3:3:4:3:3:3:4:3:4:4:4:3:3:4:4:4:3:4:3:3:3:4:3:3:3:4:3:3:3:3:3:3:3:4:4:4:3:3:3:3:3:4:3:3:4:4:4:4:3:3:4:4:3:4:4:3:4:4:3:4:4:3:3:3:4:3:3:4:3:4:4:3:4:4:4:4:4:4:3:4:4:3:3:3:3:3:3:3:3:4:4:3:4:3:3:3:3:3:3:3:3:3:3:4:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:4:4:3:3:3:4:3:3:3:3:4:3:3:3:3:3:3:3:4:4:4:4:4:4:4:4:4:4:3:4:4:4:4:3:4:3:3:3:3:4:4:3:4:3:4:3:3:3:4:3:4:3:3:3:3:3:3:3:3:4:3:1:1:1] [APP: SHAREDSERVICES#11.1.2.0] Service currently not available.

[2020-05-16T09:04:44.587-06:00] [FoundationServices2] [ERROR] [EPMLCM-30052] [oracle.EPMLCM] [tid: 815] [userId: <anonymous>] [ecid: 00j15bZ53w2EoItDwfV4CW0003po001Y3p,0:1:4:3:4:3:3:3:3:4:3:3:3:3:4:3:4:4:4:3:3:3:3:3:4:3:4:3:3:3:3:3:3:3:3:3:3:3:3:3:3:4:4:4:3:4:3:3:3:3:3:4:3:3:3:3:3:3:4:3:3:3:4:3:4:4:4:3:3:4:4:4:3:4:3:3:3:4:3:3:3:4:3:3:3:3:3:3:3:4:4:4:3:3:3:3:3:4:3:3:4:4:4:4:3:3:4:4:3:4:4:3:4:4:3:4:4:3:3:3:4:3:3:4:3:4:4:3:4:4:4:4:4:4:3:4:4:3:3:3:3:3:3:3:3:4:4:3:4:3:3:3:3:3:3:3:3:3:3:4:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:3:4:4:3:3:3:4:3:3:3:3:4:3:3:3:3:3:3:3:4:4:4:4:4:4:4:4:4:4:3:4:4:4:4:3:4:3:3:3:3:4:4:3:4:3:4:3:3:3:4:3:4:3:3:3:3:3:3:3:3:4:3:1:1:1] [APP: SHAREDSERVICES#11.1.2.0] [SRC_CLASS: ?] [SRC_METHOD: ?:?] Failed to connect to "HPCMPRD" while performing import for application. Received status code - "503" with error message - "Weblogic Bridge Message Failure of server APACHE bridge: No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent. ". Possible cause of error no backend server available for connection.

We first tried to restart Hyperion Foundation and HPCM related services but no luck. As per the SharedService_LCM.log, there seems to be an issue while connecting to the HPCM web application server during the LCM import process. As we noticed the LCM import process is getting aborted every time exactly after five mins. So we thought it to be a time out issue probably at the OHS layer. 

Solution:

As LCM import was getting aborted after 5 mins so we thought to check and modify (increase or add) the timeout setting for profitability application in the OHS (Oracle HTTP Server) configuration file i.e. mod_wl_ohs.conf. The Oracle HTTP Web server can be configured to time out if a job takes longer than a predefined period. 

Note: If your OHS server has not been configured to a specific shared location and has been kept as default then your mod_wl_ohs.conf file will be stored at <EPM_ORACLE_INSTANCE\httpConfig\ohs\config\OHS\ohs_component\mod_wl_ohs.conf. But If your OHS server is load balanced and has been configured to a shared location and you don't know that location then you can run a registry report and search for that shared location. 

How to modify LCM timeout default settings for HPCM (Profitability) application in Oracle HTTP Server server configuration file (mod_wl_ohs.conf ):

If you are using Hyperion LCM to import large HPCM models or POVs, the LCM import process may take longer to process than the time specified in the default timeout settings on the Oracle WebLogic Server. To fix this issue, you need to increase or add a timeout setting for Profitability application in OHS configuration file i.e. mod_wl_ohs.conf. 

Steps:

1- Stop your Oracle Hyperion services (Foundation and OHS service must be stopped).

2- On your OHS Web server, navigate to the path:

%Middleware_HOME%\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component\mod_wl_ohs.conf.

3- First take a backup of mod_wl_ohs.conf file. Then open it and locate the below section: 

<LocationMatch ^/profitability>

4- Add the following lines within the <LocationMatch ^/profitability> section (modify or add the WLIOTimeoutSecs property with a value that will encompass the duration of typical HPCM migration tasks):

WLIOTimeoutSecs 3000

Idempotent OFF

5- Now on OHS web server, navigate to the path:

%Middleware_HOME%\user_projects\epmsystem1\httpConfig\ohs\config\OHS\ohs_component\htppd.conf

6- First take a backup of htppd.conf file. Then open it and locate the below section:

# Timeout: The number of seconds before receives and sends time out. 

7- Add or modify the Timeout value to 3000, as shown below:

# Timeout: The number of seconds before receives and sends time out. 

Timeout 3000

Note: The server timeout shown above is a suggested limit, and may be modified to suit the specific timeout settings for your application server.

8- Start your Oracle Hyperion services and try to re-import your HPCM application artifacts. It should work fine now without any timeout issue.

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!