DOYENSYS Knowledge Portal




We Welcome you to visit our DOYENSYS KNOWLEDGE PORTAL : Doyensys Knowledge Portal




Tuesday, December 27, 2016

           OIDMOS status stop and OID not starting up after crash


Our system has a system shutdown in which OID 11.1.1.9 was running and not starting up using opmnctl after server is brought up. A few things I went through to fix it up is posted below.

Check the status from the Oracle Instance home as below.

[apploid@oam01 bin]$ ./opmnctl status

Processes in Instance: asinst_1
———————————+——————–+———+———
ias-component                    | process-type       |     pid | status
———————————+——————–+———+———
ohs1                             | OHS                |   19283 | Alive
ovd1                             | OVD                |   19282 | Alive
oid1                             | oidldapd           |     N/A | Down
oid1                             | oidldapd           |     N/A | Down
oid1                             | oidmon             |    5562 | Stop
EMAGENT                          | EMAGENT            |   19281 | Alive

You can see oidmon is stopped and oidldap is down. Normally opmnctl startall can bring up in almost all the cases, but I was getting time outs and stuck at below in the oidmon****.log file.

[2015-09-04T13:23:59.800201+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0] OIDMON_STOP: Thread started

[2015-09-04T13:23:59.807343+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0] OIDMON_STOP: Connecting to database, connect string is oiddb

I did check the database connectivity from the OID application server, checked the ODS schema status and almost everything including the status in ODS_PROCESS_STATUS (I will post about this table soon).

Since we had a abrupt shutdown, I guessed we have some uncleaned files in the directory below. Removed the files after taking the backup after shutting down everything, then try starting up.

$ORACLE_INSTANCE/bin/opmnctl stopall

cd $ORACLE_INSTANCE/config/OPMN/opmn/states

[apploid@oam01 states]$ ls
p1515733570

[apploid@oam01 states]$ cd ..

[apploid@oam01 opmn]$ mv states states_bak

[apploid@oam01 opmn]$ mkdir states

$ORACLE_INSTANCE/bin/opmnctl startall

Check the log file

[apploid@oam01 opmn]$ tail -f /u01/oid/Oracle/Middleware/asinst_1/diagnostics/logs/OID/oid1/oidmon-0000.log
[2015-09-04T13:33:43.438248+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Starting OIDLDAPD  cmdLine args = “instance=1 inst=1  “

[2015-09-04T13:33:43.446457+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Starting OIDLDAP Server,PID=19818

[2015-09-04T13:33:43.449019+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Updating Process Table…

[2015-09-04T13:33:43.452984+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gsldda_DeleteEntry:gsldfgGetEntryID Error..1010

[2015-09-04T13:33:43.707887+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: CreateInstance: Successfuly created entry cn=asinst_1_oid1_1_oam01.saroj.com.au,cn=osdldapd,cn=subregistrysubentry

[2015-09-04T13:33:53.403166+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gslmmonOpmnProcStatus: Sending OIDLDAPD PID=,19956 status to OPMN

[2015-09-04T13:33:55.405243+10:00] [OID] [NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: gslmmonOpmnProcStatus:Sending OID Host and ports info to OPMN
          HOST=oam01.saroj.com.au LDAP NONSSL PORT = 3060 LDAP SSL PORT = 3131

[apploid@oam01 bin]$ ./opmnctl status

Processes in Instance: asinst_1
———————————+——————–+———+———
ias-component                    | process-type       |     pid | status
———————————+——————–+———+———
ohs1                             | OHS                |   19773 | Alive
ovd1                             | OVD                |   19772 | Alive
oid1                             | oidldapd           |   19956 | Alive
oid1                             | oidldapd           |   19818 | Alive
oid1                             | oidmon             |   19771 | Alive
EMAGENT                          | EMAGENT            |   19770 | Alive


No comments: