’m over in Singapore at the moment, and the client I’m working with is looking for some help with troubleshooting Oracle BI EE installations. This is an interesting area because as developers, we generally focus on building new applications rather than supporting existing ones, but if you’re then handing these systems off for other people to support you need to give them some guidance on how to solve problems. So if you’re going to support an Oracle BI EE system, how do you go about troubleshooting them?
If we are considering Oracle BI EE troubleshooting, to me there are three areas we need to cover:
- What indicators are there of current and potential problems?
- What are some of the common problems we might encounter when supporting Oracle BI EE?
- Are there things we can do to minimize the chance of problems occuring?
So to start with the first point, what indicators are there of potential problems? Well the first place I would look would be the various logs that the Oracle BI EE server components generate. The most obvious logs are the BI Server, BI Scheduler and BI Cluster Controller ones:
- $ORACLEBI/server/log/NQServer.log (the BI Server “alert log” file)
- $ORACLEBI/server/log/NSQuery.log (the BI Server query log file)
- $ORACLEBI/server/log/NQSheduler.log (the Delivers/scheduler log file)
- $ORACLEBI/server/log/NQCluster.log (the Cluster Controller log file)
with the NQServer.log one being the most useful when diagnosing startup and other issues with the BI Server. In the example below, the BI Server can’t start up because there are no available subject areas in the RPD file (in this case, due to developers saving the RPD with consistency errors).
If anything goes wrong with the BI Server, or causes it to do something unusual (like not start usage tracking, or not be able to connect to a required external authentication mechanism) it’ll appear in this log. If a user reports that Oracle BI EE is unavailable, particularly if the dashboard interface comes up but it can’t connect to the BI Server, this is the place to look. It also makes sense to check this log file anyway, on a regular basis, in case it’s reporting an issue that users haven’t spotted yet but that might cause problems for you later on.
The other log file that’s worth keeping an eye on is the Presentation Services log file, found at
- $ORACLEBIDATA/web/log/sawlog0.log
This log file records failed login attempt (useful to see if someone is trying to hack into your system, or you’re having problems authenticating against LDAP or Shared Services, for example), and it also shows issues you might be having with dashboard elements and reports.
An couple of examples of where this particular log file were useful include:
- On an Oracle BI Applicaitons implementation on AIX, the Presentation Server would crash and segfault on particular dashboard pages, and when we looked in the Presentation Services log we could see that certain pivot table reports had a high level of cells which should normally have been caught by a setting in the instanceconfig.xml file but instead was causing the Presentation Server to crash.
- When setting up EPM and OBIEE integration, we could see that incoming login requests from the EPM Dashboard were being rejected as the Presentation Server wasn’t properly handling the incoming SSO token.
Together with the NQServer.log file, these two files should be the first place you look if the BI Server or Presentation Server are either failing to start, regularly crashing or exhibiting strange behavior. In addition, if you’re lucky enough to have the Enterprise Manager BI Management Pack, there are various screens within the pack for remotely monitoring the contents of these logs.
Other logs that you will want to keep an eye on include:
- $ORACLEBIDATA/web/log/javahost/jhost0.log (Java host log)
This is the log file for the Java Host process, and can be useful in diagnosing problems when setting up the Hyperion Custom Authenticator amongst other things.
The logs generated by the J2EE application server hosting the Presentation Services plug-in can be useful as well. The log files for OC4J and Weblogic 9.2 can be found here:
- $ORACLEBI/oc4j_bi/j2ee/home/log/server.log
- $BEA_HOME/user_projects/domains/bi/servers/AdminServer/logs/AdminServer/logs/AdminServer.log
In addition, if you are running the Oracle BI components on Windows, the Application log in Event Viewer shows you when the server components fail to start or shut down abnormally, from what I can see though this is more or less the same information that’s in the NQServer.log file albeit in a centralized place that systems management tools can make use of.
Under Red Hat Linux / Oracle Enterprise Linux, the equivalent utility is the Systems Logs application, and I guess there are similar utilities available for the various Unix equivalents.
Outside of OBIEE, problems you might encounter might be down to the underlying database or OLAP server either not being available or failing to authenticate the user credentials being passed across by the connection pool, so it makes sense to keep an eye on the Oracle Alert Log, for example, to see if the problem lies with the database or just to spot potential database problems before they hit the user.
So, that’s it for log files and places to diagnose potential problems. Do you know of any others that you find useful on a regular basis? I’m off to work now, but later on I’ll continue this series with some tips on what problems you are likely to encounter on an Oracle BI EE system, how you can do some preventative maintenance to avoid them, and also a high-level view of how to troubleshoot an attached Essbase OLAP server and Shared Services.
Posts
