EPM upgrade from version 11.2.7 to 11.2.8 to 11.2.10

Recently we upgraded our Oracle EPM environment, first from version 11.2.7 to 11.2.8 and then from 11.2.8 to 11.2.10. To give you an idea of the EPM setup, we have Essbase, HFM, HFR, HPCM, FDMEE, DRM applications in a distributed and load balanced environment.

In this upgrade journey, below are some points I found worth sharing for all who are planning to upgrade their on-prem EPM 11.2.x environment:

1- EPM 11.2.7 to 11.2.8 is upgraded through 'Apply Update' feature while 11.2.8 to 11.2.10 upgrade happens by installing OPatches (Beginning with EPM Release 11.2.9, release updates are distributed as a set of OPatches).

2- Prior to upgrade, you must take a backup of following 2 folders without fail. You can delete these folders after you apply the update and validate your applications.

EPM Oracle Home: MIDDLEWARE_HOME/EPMSystem11R1

EPM Oracle Instance: MIDDLEWARE_HOME/user_projects/epmsystem1

This will ensure quick restoration of your environment to pre-upgrade running state.

3- I am sure your IT team would already be taking backup of your EPM servers/VMs and the relational databases/schemas. Make sure you check with them to confirm.

4- You should always run installTool.cmd OR ./installTool.sh (for 11.2.7 to 11.2.8 upgrade) and .\ApplyUpdate.ps1 OR ./ApplyUpdate.sh (for 11.2.7 to 11.2.8 upgrade), as an Administrator.

5- If you have a server where only one product is installed for example, Essbase, HFM etc. (In our case it was Essbase on Linux server), then before running installTool.cmd OR ./installTool.sh (for 11.2.7 to 11.2.8 upgrade), manually create the domains folder, if its not there: EPM_ORACLE_HOME\user_projects\domains. Otherwise your 11.2.8 Installer will hang at 80% "creating Oracle Inventory". For more details, check this Oracle document- Doc ID 2853657.1 .

6- If you are upgrading from Release 11.2.5 or later to Release 11.2.8 and you are already on the latest version of Oracle WebLogic server, Oracle HTTP Server (OHS) etc. (most probably as part of your security Vulnerability patching), in that case these components will not be updated during the upgrade process.

7- Post 11.2.8 upgrade, we faced error: java.lang.NoClassDefFoundError-org-apache-log4j-FileAppender, while running FDMEE DLR (Data Load Rule). 

Solution is to apply Patch 34812016. But the tricky part here is that you should apply Patch 34812016 post upgrade to EPM 11.2.10 (if your target is to eventually upgrade to 11.2.10) otherwise during FDMEE 11.2.10 upgrade, patch conflict error will occur on FDMEE server .

8- Post 11.2.8 upgrade, your EPM JDK version is not changed. It will remain the same what you had in 11.2.7 before upgrading to 11.2.8. So technically no JDK update is required.

9- Post 11.2.8 upgrade, external user directories like MSAD remains configured/connected to your Shared Services.

10- Post 11.2.10 upgrade, your EPM JDK version will be changed to 1.8.0_331 on all the servers. It happens because Release 11.2.10 also includes Java Updates. So make sure you upgrade your JDK, If you were on some higher JDK version prior to 11.2.10 upgrade (normally we do this as part of Java security Vulnerability fixes on EPM servers).

11- As your JDK version will be updated post 11.2.10 upgrade, you must take back up of following keystores on all servers, prior to 11.2.10 upgrade:

MIDDLEWARE_HOME\jdk\jre\lib\security\cacerts

MIDDLEWARE_HOME\EPMSystem11R1\common\JRE\Sun\1.8.0\lib\security\cacerts

Ensure that you restore the backed up keystores (cacerts) after the upgrade.

12- Post EPM 11.2.10 upgrade, make sure to install the latest version of Clients like EASConsole.exe, EssbaseClient.exe, supplied with EPM 11.2.10 release.

13- Post 11.2.10 upgrade we noticed, users are unable to connect to Essbase through Smartview and also unable to see Essbase calc rules in Calc Manager. On further analysis it was identified that APS patch that comes with 11.2.10 OPatches is not fully applied using ApplyUpdate script (some files might be missing). So to fix the problem, we rolled back APS patch 33485394, downloaded it from Oracle support portal separately and reinstalled it on APS server (as suggested by Oracle).

14- Oracle Data Relationship Management (DRM) is installed separately and will continue as a full install release (for both 11.2.8 and 11.2.10 upgrade and so on).

15- Your configuration settings for DRM integration with HSS for users authentication will remain intact after DRM 11.2.8 or DRM 11.2.10 installation, so no action required on that front.

16- Before applying the update for EPM 11.2.10,  you can identify any patch conflicts on your EPM servers using following command:

On Windows: .\ApplyUpdate.ps1 <MIDDLEWARE_HOME> -verify

On Linux: ./ApplyUpdate.sh <ORACLE_HOME> -verify

If you see any patch conflict, you should resolve it before applying the 11.2.10 upgrade.

That's all for this post. I hope this article has helped you.
If you have any further queries, feel free to contact me at epmzones@gmail.com.

Share this post.

__PRESENT__PRESENT

Read More

Use of Microsoft .NET Framework in Oracle EPM 11.2.x

Question: 

What is the use of Microsoft .NET Framework in Oracle EPM 11.2.x setup?

Answer: 

In Oracle EPM 11.2.x, Microsoft .NET 4.7.2 Framework as provided by Windows Operating System, is REQUIRED ONLY FOR Data Relationship Management (DRM) and SmartView. No other EPM 11.2.x application uses .NET Framework.

For example, if you are using Windows 2019 Server for your EPM 11.2.10 setup, Windows 2019 Server by default comes with IIS 10 and .NET Framework 4.7 installed so you don't even need to install any .NET Framework version manually. But yes, as part of your security vulnerability fixes on the EPM servers, you need to install KBs or Windows updates related to .NET Framework 4.7/4.7.x occasionally.

__PRESENT

Read More

How to stop/start Oracle HTTP Server (OHS) 12c in EPM 11.2.x

Unlike prior versions, EPM 11.2.x does not include Oracle HTTP server (OHS) startup service. So you need to either manually start ohs component or create a script for the same.

EPM services are generally started in below order in EPM 11.2.x:

1- Start Oracle WebLogic Admin Server

2- Start all EPM services including Node Manager service from the Windows Services panel. Node Manager service name will be like Oracle Weblogic ohs NodeManager (E_Oracle_MIDDLE~1_ohs_wlserver). 

Note: Oracle HTTP Server (OHS) is managed and monitored with Node Manager. So OHS needs Oracle Node manager service running to start. 

3- Start Oracle HTTP Server (OHS) from command line.

Starting Oracle HTTP Server from the command line:

1- Open Command prompt as an Administrator.

2- Type below command to start OHS server:

E:\Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs_component

3- It will prompt for a password, Enter the WebLogic Admin Server password.

4- OHS server is successfully started. 

# To stop OHS server:

• Stop the Node Manager service from Windows Services panel: Oracle Weblogic ohs NodeManager (E_Oracle_MIDDLE~1_ohs_wlserver)
• Stop Oracle HTTP Server (OHS) from command line using below command:

 E:\Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\stopComponent.cmd ohs_component

__PRESENT__PRESENT__PRESENT

__PRESENT

Read More

EPM: Log4j vulnerability/security-threat in EPM 11.1.2.4

We all have heard about the Log4j vulnerabilities (CVE-2021-44228 and CVE-2021-45046) reported recently in Dec 2021. 

The Apache Software Foundation has released a security advisory to address a remote code execution vulnerability (CVE-2021-44228) and a denial of service vulnerability (CVE-2021-45046) affecting Log4j versions 2.0-beta9 to 2.15 ( a system running Apache Log4j version 2.15 or below i.e. Affected Versions are Apache Log4j versions 2.0--2.15.0). A remote attacker could exploit these vulnerabilities to take control of an affected system by executing arbitrary code. The recommendation is to upgrade to the latest Log4j 2.16.0 or applying recommended mitigations immediately. 

It has been determined that Log4j vulnerability impacts EPM (Enterprise Performance Management) application too via the Apache Log4j open source component it ships (EPM ships the log4j Java library as a jar file to be used by the applications like HFM, FCM etc.).

So its imperative to take mitigation steps to alleviate the impact associated with Log4j vulnerability for Oracle Enterprise Performance Management (EPM).

Currently we are in the process of upgrading our existing EPM 11.1.2.4 environment to EPM 11.2. So we though to know the impact of Log4j vulnerability/security-threat first on EPM 11.1.2.4 as we are sure that EPM 11.2.x is impacted by it.

What could be better option than checking with vendor Oracle itself about the impact and mitigation plan of Log4j vulnerability in EPM 11.1.2.4?

So those who still have EPM 11.1.2.4 up and running (as of 28th Dec 2021), you should know that:

EPM 11.1.2.4 is NOT AFFECTED by Log4j vulnerability/security-threat, as confirmed by Oracle support and shown below. EPM 11.1.2.4 uses log4j 1.x library which is not impacted by Log4j vulnerability (CVE-2021-44228 and CVE-2021-45046) reported for Apache Log4j version 2.x (i.e. Affected Versions are Apache Log4j versions 2.0--2.15.0).

That's all for this post.

I hope this article has helped you. Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Share this post.

Read More

EPM: Batch Script to backup essbase.sec file on Oracle Essbase server

As part of your EPM backup strategy, its important to backup your Essbase security file (essbase.sec) on daily basis to handle unforeseen issues in your Oracle EPM Essbase server/application.

Below is one batch script that you can use to backup/copy the latest essbase.sec file from your source (Essbase server) and paste it into a destination (Essbase server itself or any network share) renaming it with today's date for easy identification during restoration.

@echo off

:: Format today's date in YYYYMMDD format
for /f "delims=" %%a in ('wmic OS Get localdatetime ^| find "."') do set "dt=%%a"
set "YYYY=%dt:~0,4%"
set "MM=%dt:~4,2%"
set "DD=%dt:~6,2%"
set "today_date=%YYYY%%MM%%DD%"

:: Source path
set sourcepath=E:\apps\oracle\epm\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin

:: Destination path
set destinationpath=E:\EPMBackup\Essbase_Backup\BKPFiles

:: Log path
set logfile=E:\EPMBackup\Essbase_Backup\Log\Essbase-Sec-File-Copy_%today_date%.log

:: Copy the latest essbase.sec file
for /f %%i in ('dir "%sourcepath%\essbase.sec" /b/a-d/od/t:c') do set NewestFile=%%i
echo %today_date% >>%logfile%
echo ---------------->>%logfile%
echo %NewestFile% >>%logfile%
copy "%sourcepath%\%NewestFile%" "%destinationpath%\%NewestFile%_%today_date%" >>%logfile%

Notes:

• I assumed that you have your Essbase component installed and configured on a Windows server.
• If your Essbase is installed on Linux/Unix server, you can create a bash/shell script accordingly based on this same logic.
• You can change source path, destination path and log file path as per your Essbase server and requirement.
• Schedule this script in Windows Task scheduler on Essbase server to run everyday at a fixed time.

On successful run, everyday following two files will be created i.e. essbase.sec backup file and a corresponding log file.

That's all for this post.
I hope this article has helped you. Your suggestions/feedback are most welcome.
Keep learning and Have a great day!!!

Share this post.

Read More

Unable to add/connect or expand Essbase server in EAS (Essbase Administration Services) console

Issue:

When we try to add/connect a new Essbase server or expand an existing Essbase server in EAS (Essbase Administration Services) console to see applications list etc., it keeps loading/rolling for long time and eventually Essbase server doesn’t get added or expanded being hanged, as shown below:

Adding new Essbase server:

Expanding existing Essbase server:

Investigation:

We check the easserver.log for the timestamp and noticed following error:

Log file path: E:\apps\OracleEPM\Middleware\user_projects\domains\EPMSystem\servers\EssbaseAdminServices0\logs\easserver.log

Error:

[2021-10-18T09:17:26.196-06:00] [EssbaseAdminServices0] [ERROR] [ESSEAS-24206] [oracle.epm.essbase.eas] [tid: 18] [userId: <anonymous>] [ecid: 00jBJTZ37vAB1FwDwFj8CW0002es0000Zt,0:1:4] [APP: EAS#11.1.2.0] Failed to handle request for com.essbase.eas.essbase.defs.ServerCommands.Connect. See below stack trace for more information. Possible cause for this exception is missing java archive for this request to handle. Check application server WEB-INF/lib folder or CLASSPATH, if the required java archive file is available.[[
com.essbase.eas.framework.server.defs.ApplicationException: java.net.SocketException: socket write error: Connection aborted by peer
.....
.....
Caused by: java.net.SocketException: socket write error: Connection aborted by peer

Solution that worked for us:

Note: We have Essbase component installed on Linux server and EAS service installed on Windows 20212 R2 server.

Error "socket write error: Connection aborted by peer" is usually caused by writing to a connection that had been aborted by the peer before getting the full response. It means that the other side aborted the connection. Since we are facing this error on client side (EAS console), then the Essbase server side must be the one aborting the connection. In most cases this can be caused either by the timeout issue (e.g. the response takes too much time or server is overloaded with the requests), or the client sent the SYN, but it didn't receive ACK (acknowledgment of the connection termination) from the other side.

We first checked with Network team to see if there is any network issue between the source (EAS server) and the target (Essbase server). But they confirmed it was all ok at Network end.

Then we decided to reboot our Essbase Linux server as the connection was getting aborted from that side.

Steps:

• Stop Essbase and EAS services.
• Reboot Essbase (Linux) server.
• Start Essbase service and then EAS service.
• Login to EAS console and try to add a new Essbase server or expand an existing Essbase server. It worked fine.

That's all for this post.

I hope this article has helped you. Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Share this post:

Read More

Oracle Middleware Home, EPM Oracle Home and EPM Oracle Instance

In this post, we will see three important directories related to Oracle EPM installation and configuration:

1. Oracle Middleware Home
2. EPM Oracle Home
3. EPM Oracle Instance

Oracle Middleware Home

In an Oracle EPM system, Oracle Middleware home consists of Oracle WebLogic Server home and EPM Oracle home. You can setup your Oracle Middleware home either on a local file system on the server or on a remote network shared location. 

When you do EPM installation and configuration first-time, you need to define Oracle Middleware Home, which will be then used for all further EPM product installations on that system. 

Oracle Middleware Home installation directory is referred as MIDDLEWARE_HOME with default location as Oracle/Middleware on your EPM system.  For example it could be:

MIDDLEWARE_HOME : E:\apps\OracleEPM\Middleware

Below is an example of MIDDLEWARE_HOME directory structure:

EPM Oracle Home

EPM Oracle Home resides within Oracle Middleware Home and contains following things:

• Files/folders related to EPM applications/products installed on that EPM system
• Files/folders related to common internal components used by these EPM applications/products .

You cannot change the EPM Oracle Home location, so before starting the installation and configuration of EPM applications on a system make sure you have allocated sufficient disk space to your EPM Oracle Home drive to accommodate all the EPM applications/products that you are going to install on that system. 

The default EPM Oracle home location is MIDDLEWARE_HOME/EPMSystem11R1. EPM Oracle home location is defined in the system environment variable called EPM_ORACLE_HOME. For example it could be:

EPM_ORACLE_HOME : E:\apps\OracleEPM\Middleware\EPMSystem11R1

In a distributed EPM environment setup, the EPM Oracle home directory structure must be the same on each machine of that EPM environment. For example, if the path for EPM Oracle home is E:\apps\OracleEPM\Middleware\EPMSystem11R1 on the first machine you configure, it must be E:\apps\OracleEPM\Middleware\EPMSystem11R1 on all the other machines in the deployment.

Below is an example of EPM_ORACLE_HOME directory structure:

EPM Oracle Instance

EPM Oracle Instance contains one or more EPM components like Oracle HTTP Server (OHS), Oracle Essbase Server, one or more Java web applications in one or more domains. 

Unlike EPM Oracle Home, EPM Oracle instance can reside anywhere; it need not to be within the Oracle Middleware home directory.

The default location for the EPM Oracle instance referred to as EPM_ORACLE_INSTANCE is MIDDLEWARE_HOME/user_projects/epmsystem1. For example it could be:

EPM_ORACLE_INSTANCE : E:\apps\OracleEPM\Middleware\user_projects\epmsystem_web

Java web applications are deployed to MIDDLEWARE_HOME/user_projects/domains/domainName for example it could be: E:\apps\OracleEPM\Middleware\user_projects\domains\EPMSystem.

For a distributed EPM environment, you need to create a new EPM Oracle instance on each machine of the environment. But if you are installing all EPM products on a single machine, then all EPM products are configured under a single EPM Oracle instance what you create for the first product configuration.

Below is an example of EPM_ORACLE_INSTANCE directory structure:

That's all for this post.

I hope this article has helped you. Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Share this post:

Read More