ORACLE initialization or shutdown in progress

Our database guy was trying to perform some operations on the oracle 11g database today and got the following error:

ORA-011033: ORACLE initialization or shutdown in progress

This could have a lot of underlying errors, but in our situation this occurs when the power is unexpectedly turned off to the server, causing the database transaction logs to not be closed properly.

Provided that you configured RMAN backups when you installed the instance,  (Which should have been rather apparent during the installation of the software and the creation of the database) recovery from this situation should be quite smooth.

I will highlight the commands manually entered.  Anything else is a response from command(s).

First, we log into the server using the oracle account you created before installing the database software.  Once you are logged into the operating system, on the console you will want to run sqlplus as the SYSDBA account, and properly shutdown the database. (It's probably stuck trying to initialize but cannot, since it will attempt to start on boot when the power came back on)  Then you will want to start the instance, manually mount the control files, but NOT open the database yet, just quit.

ORACLEBIR:/export/home/oracle$ sqlplus '/as sysdba'

SQL> shutdown abort

ORACLE Instance shut down.

SQL> startup nomount

ORACLE Instance started

SQL> alter database mount;

SQL> quit

At this point, your instance has been started and the database files have been mounted.  Now we run RMAN or the Recovery Manager.  Here we will request that RMAN performs a recovery.  If you get more errors or this does not work for you, it would seem your backup and recovery settings are not configured properly, and you are really in trouble.  Otherwise, it should look something like this:

ORACLEBOX:/export/home/oracle$ rman

RMAN> connect target

connected to target database: DBV2 (DBID=2494479496, not open)

using target database control file instead of recovery catalog

RMAN> recover database;

Starting recover at 09-APR-14

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=161 device type=DISK

starting media recovery

archived log for thread 1 with sequence 871 is already on disk as file /u1/app/oracle/oradata/dbv2/redo01.log

archived log file name=/u1/app/oracle/oradata/dbv2/redo01.log thread=1 sequence=871

media recovery complete, elapsed time: 00:00:07

Finished recover at 09-APR-14

RMAN> quit

If you see something like above, congratulations, your database has recovered.  Now, let's go back and actually open the database and reset the logs.  Also, make sure to start enterprise manager, and that your backup admin knows how to connect and use this tool to some degree.  Although some do not like the Enterprise Manager, it is essential to automate tasks for anyone who is not familiar with some of the basic operations.  And you might get hit by a bus, leaving all the work in your colleague's lap.

ORACLEBOX:/export/home/oracle$ sqlplus '/as sysdba'

SQL> alter database open resetlogs;

SQL> quit

ORACLEBOX:/export/home/oracle$ emctl start dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.0

Copyright (c) 1996, 2011 Oracle Corporation.  All rights reserved.

https://ORACLEBOX:1158/em/console/aboutApplication

Starting Oracle Enterprise Manager 11g Database Control .................... started.

------------------------------------------------------------------

Logs are generated in directory /u1/app/oracle/product/11.2.0.3/dbhome_1/ORACLEBOX_dbv2/sysman/log

Your EM Login:

https://ORACLEBOX:1158/em/console/logon/logon

Backup and Recovery Guide for Oracle RDBMS

Administration Guide for Oracle 11g