Shell Scripting(sh&ksh)

USEFULL LINK FOR LINUX
http://linuxcommand.org/lc3_writing_shell_scripts.php


$# Stores the number of command-line arguments that were passed to the shell program.
$? Stores the exit value of the last command that was executed.
$* is a string containing all arguments.
$1 is the first argument.


SSH FUNCTION

{
if [ ${DB_OS_TYPE} == "SunOS" ]; then
        COMMAND='cat /etc/release | head -1'
    DB_CPU_TEMP=`exec_over_ssh ${ORA_USER} ${ORA_HOST} ". /export/home/$ORA_USER/.profile > /dev/null;opatch lsinventory| grep -i 'DATABASE PATCH'| head -1"`
else
    DB_CPU_TEMP=`exec_over_ssh ${ORA_USER} ${ORA_HOST} ". /home/$ORA_USER/.bash_profile > /dev/null;opatch lsinventory| grep -i 'DATABASE PATCH'| head -1"`
fi



exec_over_ssh()
{
        USER=$1
        HOST=$2
        COMMAND=$3
        OUTPUT=`ssh ${USER}@${HOST} ${COMMAND}`
        echo ${OUTPUT}
}

###################created script by me#
#!/bin/bash
###set -x
user=$1
HOST=$2
COMMAND=$3
exec_ssh()
{
OUTPUT=`ssh ${user}@${HOST} ${COMMAND}`
echo ${OUTPUT}
}
if [ $# == 3 ]
then
exec_ssh
else
echo "enter correect parameter"
exit
fi

REDOLOG VS ARCHIVE LOG

SQL> SELECT distinct member LOGFILENAME FROM V$LOGFILE;

LOGFILENAME
--------------------------------------------------------------------------------
/u01/install/APPS/data/ebsdb/log1.dbf
/u01/install/APPS/data/ebsdb/log2.dbf
/u01/install/APPS/data/ebsdb/log3.dbf

ALTER SYSTEM SWITCH LOGFILE vs
ALTER SYSTEM ARCHIVE LOG CURRENT

hat is the difference between ALTER SYSTEM SWITCH LOGFILE and ALTER SYSTEM ARCHIVE LOG CURRENT, and when do I use each?
Answer:  Yes, both ALTER SYSTEM SWITCH LOGFILE and ALTER SYSTEM ARCHIVE LOG CURRENT will force a log switch, but they do it in different ways! 
Both the SWITCH LOGFILE and ARCHIVE LOG CURRENT write a quiesce checkpoint, a firm place whereby that last redo log is a part of the hot backup, but ARCHIVE LOG CURRENT waits for the writing to complete.  This can take several minutes for multi-gigabyte redo logs.
Conversely, the ALTER SYSTEM SWITCH LOGFILE command is very fast and returns control to the caller in less than a second while ALTER SYSTEM ARCHIVE LOG CURRENT pauses.
As we see below, the ALTER SYSTEM SWITCH LOGFILE is fast because it does not wait for the archiver process (ARCH) to complete writing the online redo log to the archivelog log filesystem:

1. It issues database checkpoint
2.  It immediately starts writing to the next redo log
3.  In the background, the "switch logfile" command tells the ARCH background process to copy the "old" redo log file to the redo log filesystem.  

Here are the important differences between ALTER SYSTEM SWITCH LOGFILE and ALTER SYSTEM ARCHIVE LOG CURRENT:

•  RAC:  If you are running RAC, the ALTER SYSTEM ARCHIVE LOG CURRENTwill switch the logs on all RAC nodes (instances), whereasALTER SYSTEM SWITCH LOGFILE will only switch he logfile on the instance where you issue the switch command.  Hence, ALTER SYSTEM ARCHIVE LOG CURRENT is a best practice for RAC systems.

ENABLING ARCHIVELOG and SWITCHING LOG

SQL> archive log list

Database log mode              No Archive Mode

Automatic archival             Disabled

Archive destination            /u01/install/APPS/data/ebsdb/archive

Oldest online log sequence     7

Current log sequence           9

SQL> shut immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance

ORACLE instance started.

Total System Global Area 2147483648 bytes

Fixed Size                  2926472 bytes

Variable Size             553650296 bytes

Database Buffers         1577058304 bytes

Redo Buffers               13848576 bytes

Database mounted.

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /u01/install/APPS/data/ebsdb/archive

Oldest online log sequence     7

Next log sequence to archive   9

Current log sequence           9

SQL> alter switch logfile;

alter switch logfile

      *

ERROR at line 1:

ORA-00940: invalid ALTER command

SQL> ALTER SYSTEM SWITCH LOGFILE ;

System altered.

SQL> archive log list;

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            /u01/install/APPS/data/ebsdb/archive

Oldest online log sequence     8

Next log sequence to archive   10

Current log sequence           10

SQL> exit

REALTIME ISSUES - ORACLECLOUD

****DCS-10045:Validation error encountered: Backup type is invalid.****
CONSOLE ERROR:

Status: The database restore operation failed. Run 'dbcli list-jobs' on all hosts in the DB system to check for problems and fix them before rerunning the operation.

[root@testbackupdb ~]# dbcli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
7182911b-1733-4805-b351-081f6be6605b     Authentication key update for DCS_ADMIN                                     May 5, 2019 4:09:58 PM UTC          Success
ce8f602b-84c7-4295-982a-49098dca465d     Provisioning service creation                                               May 5, 2019 4:11:41 PM UTC          Success
47876bcb-50a5-42ac-9648-52221a12a974     SSH keys update                                                             May 5, 2019 5:03:22 PM UTC          Success
9d01bc5f-0831-48ed-a1fc-f0982e54feaa     SSH key delete                                                              May 5, 2019 5:05:14 PM UTC          Success
80f9b24e-1061-4906-88d1-78f6b824b454     create object store:b9a7iawWuBfGYFaNH7MC                                    May 6, 2019 3:14:45 AM UTC          Success
25845fda-ea88-40b0-bb54-3be86ec265f6     create backup config:b9a7iawWuBfGYFaNH7MC_BC                                May 6, 2019 3:15:31 AM UTC          Success
64903937-7519-415b-b658-186cc6bebbdd     update database : BKUPDB                                                    May 6, 2019 3:16:23 AM UTC          Success
f3d0309f-3bf0-4ec2-b3d7-2d2fcde8599f     Server Patching                                                             May 6, 2019 3:19:53 AM UTC          Success
4fa3b016-8448-4535-b901-a5edd6dc2c1c     Create Regular-L0 Backup with TAG-DBTRegular-L01557111640379VSL for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 3:19:56 AM UTC          Success
5e902ca8-f332-4c7f-86ca-afca1590e4b9     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 3:26:48 AM UTC          Success
251ba542-234e-40f6-a7a9-f3a1ef281cb8     DB Home Prechecks                                                           May 6, 2019 5:05:43 AM UTC          Success
bc5eb7cd-b320-453f-8dea-e296e645df58     Create Longterm Backup with TAG-DBTLongterm1557125463725Zmf for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 6:52:12 AM UTC          Success
43a924a2-6e16-43e0-a2e9-cc4cd1d6b53f     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 7:05:18 AM UTC          Success
eefab9b7-2b32-4327-9eff-bf55d99ca7eb     Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB           May 6, 2019 7:10:57 AM UTC          Failure

[root@testbackupdb ~]# dbcli describe-job  --jobid eefab9b7-2b32-4327-9eff-bf55d99ca7eb

Job details
----------------------------------------------------------------
                     ID:  eefab9b7-2b32-4327-9eff-bf55d99ca7eb
            Description:  Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB
                 Status:  Failure
                Created:  May 6, 2019 7:10:57 AM UTC
                Message:  DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : testbackupdb: /opt/oracle/dcs/log/testbackupdb/rman/bkup/BKUPDB_fra1k8/rman_restore_2019-05-06_07-12-23-4741329873884168661.log.Failed to do restore validati

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_7173               May 6, 2019 7:11:07 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure
task:TaskSequential_7174                 May 6, 2019 7:11:07 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure
Database recovery validation             May 6, 2019 7:11:08 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure

[
ISSUE is #####rman log error#showing inorder recovery of scn#####

validation succeeded for archived log
recovery will be done up to SCN 1761481
Media recovery start SCN is 1760987
Recovery must be done beyond SCN 1761012 to clear datafile fuzziness
could not locate pieces of backup set key 20
validation succeeded for backup piece
Finished restore at 2019/05/06 07:12:36

Recovery Manager complete.

we have used until time "05/06/2019 07:10:08"


SQL> select sequence#,first_change#,next_change# from v$log order by 1;

 SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
---------- ------------- ------------
        18       1761147      1761155
        19       1761155      1761444
        20       1761444   2.8147E+14


FIX :so i restored using until scn 1761444 

[root@testbackupdb ~]# dbcli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
7182911b-1733-4805-b351-081f6be6605b     Authentication key update for DCS_ADMIN                                     May 5, 2019 4:09:58 PM UTC          Success
ce8f602b-84c7-4295-982a-49098dca465d     Provisioning service creation                                               May 5, 2019 4:11:41 PM UTC          Success
47876bcb-50a5-42ac-9648-52221a12a974     SSH keys update                                                             May 5, 2019 5:03:22 PM UTC          Success
9d01bc5f-0831-48ed-a1fc-f0982e54feaa     SSH key delete                                                              May 5, 2019 5:05:14 PM UTC          Success
80f9b24e-1061-4906-88d1-78f6b824b454     create object store:b9a7iawWuBfGYFaNH7MC                                    May 6, 2019 3:14:45 AM UTC          Success
25845fda-ea88-40b0-bb54-3be86ec265f6     create backup config:b9a7iawWuBfGYFaNH7MC_BC                                May 6, 2019 3:15:31 AM UTC          Success
64903937-7519-415b-b658-186cc6bebbdd     update database : BKUPDB                                                    May 6, 2019 3:16:23 AM UTC          Success
f3d0309f-3bf0-4ec2-b3d7-2d2fcde8599f     Server Patching                                                             May 6, 2019 3:19:53 AM UTC          Success
4fa3b016-8448-4535-b901-a5edd6dc2c1c     Create Regular-L0 Backup with TAG-DBTRegular-L01557111640379VSL for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 3:19:56 AM UTC          Success
5e902ca8-f332-4c7f-86ca-afca1590e4b9     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 3:26:48 AM UTC          Success
251ba542-234e-40f6-a7a9-f3a1ef281cb8     DB Home Prechecks                                                           May 6, 2019 5:05:43 AM UTC          Success
bc5eb7cd-b320-453f-8dea-e296e645df58     Create Longterm Backup with TAG-DBTLongterm1557125463725Zmf for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 6:52:12 AM UTC          Success
43a924a2-6e16-43e0-a2e9-cc4cd1d6b53f     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 7:05:18 AM UTC          Success
eefab9b7-2b32-4327-9eff-bf55d99ca7eb     Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB           May 6, 2019 7:10:57 AM UTC          Failure
eab18e5c-ea75-41dd-a4ed-98aa209f2132     Create detailed Backup Report                                               May 7, 2019 2:16:37 AM UTC          Success
f22341d1-d835-4828-8bf3-b1d8f0158c9b     Create recovery-latest for db : BKUPDB                                      May 7, 2019 2:45:07 AM UTC          Failure
65f2fcf5-776c-4630-ab8f-76975ab0b935     Create recovery-scn : scn 1761444 for db : BKUPDB                           May 7, 2019 2:57:46 AM UTC          Success




[root@testbackupdb ~]# dbcli describe-job --jobid 65f2fcf5-776c-4630-ab8f-76975ab0b935

Job details
----------------------------------------------------------------
                     ID:  65f2fcf5-776c-4630-ab8f-76975ab0b935
            Description:  Create recovery-scn : scn 1761444 for db : BKUPDB
                 Status:  Running
                Created:  May 7, 2019 2:57:46 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             May 7, 2019 2:57:58 AM UTC          May 7, 2019 2:59:37 AM UTC          Success
Database recovery                        May 7, 2019 2:59:38 AM UTC          May 7, 2019 2:59:38 AM UTC          Running

[root@testbackupdb ~]# dbcli describe-job --jobid 65f2fcf5-776c-4630-ab8f-76975ab0b935

Job details
----------------------------------------------------------------
                     ID:  65f2fcf5-776c-4630-ab8f-76975ab0b935
            Description:  Create recovery-scn : scn 1761444 for db : BKUPDB
                 Status:  Success
                Created:  May 7, 2019 2:57:46 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             May 7, 2019 2:57:58 AM UTC          May 7, 2019 2:59:37 AM UTC          Success
Database recovery                        May 7, 2019 2:59:38 AM UTC          May 7, 2019 3:02:47 AM UTC          Success
Enable block change tracking             May 7, 2019 3:02:47 AM UTC          May 7, 2019 3:02:50 AM UTC          Success
Database opening                         May 7, 2019 3:02:50 AM UTC          May 7, 2019 3:03:44 AM UTC          Success
Database restart                         May 7, 2019 3:03:44 AM UTC          May 7, 2019 3:05:06 AM UTC          Success
Recovery metadata persistance            May 7, 2019 3:05:06 AM UTC          May 7, 2019 3:05:06 AM UTC          Success

[root@testbackupdb ~]#

REALTIME ISSUES - WEBLOGIC

Weblogic :

1.MACHINE IS INCOMPATABLE STATUS (2020154.1)

managedserver.log  ----OUTPUT

<May 2, 2019 10:04:00 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_PRM1" using the node manager. Reason: Server 'P6_PRM1' is not currently running.>

<May 2, 2019 10:04:12 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_PRM1 : Exception while starting server 'P6_PRM1'>

<May 2, 2019 10:06:57 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_PR_REP1_DS" using the node manager. Reason: Server 'P6_PR_REP1_DS' is not currently running.>

<May 2, 2019 10:07:07 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_PR_REP1_DS : Exception while starting server 'P6_PR_REP1_DS'>

<May 2, 2019 10:11:40 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_WEB_SRV1" using the node manager. Reason: Server 'P6_WEB_SRV1' is not currently running.>

<May 2, 2019 10:11:50 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_WEB_SRV1 : Exception while starting server 'P6_WEB_SRV1'>

<May 3, 2019 6:52:12 AM BST> <Error> <HTTP> <BEA-101215> <Malformed Request "/<script>alert(53416)</script>". Request parsing failed, Code: -1>

<May 3, 2019 7:02:07 AM BST> <Error> <RJVM> <BEA-000503> <Incoming message header or abbreviation processing failed

 java.io.InvalidClassException: sun.rmi.server.UnicastRef; Unauthorized deserialization attempt

java.io.InvalidClassException: sun.rmi.server.UnicastRef; Unauthorized deserialization attempt

        at weblogic.utils.io.oif.WebLogicObjectInputFilter.checkLegacyBlacklistIfNeeded(WebLogicObjectInputFilter.java:236)

        at weblogic.utils.io.FilteringObjectInputStream.checkLegacyBlacklistIfNeeded(FilteringObjectInputStream.java:54)

        at weblogic.rjvm.InboundMsgAbbrev$ServerChannelInputStream.resolveClass(InboundMsgAbbrev.java:122)

        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)

        Truncated. see log file for complete stacktrace

>

<May 3, 2019 10:04:00 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_PRM1" using the node manager. Reason: Server 'P6_PRM1' is not currently running.>

<May 3, 2019 10:04:11 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_PRM1 : Exception while starting server 'P6_PRM1'>

<May 3, 2019 10:06:53 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_PR_REP1_DS" using the node manager. Reason: Server 'P6_PR_REP1_DS' is not currently running.>

<May 3, 2019 10:07:03 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_PR_REP1_DS : Exception while starting server 'P6_PR_REP1_DS'>

<May 3, 2019 10:11:30 PM BST> <Error> <NodeManager> <BEA-300034> <Could not execute command "kill" for server "P6_WEB_SRV1" using the node manager. Reason: Server 'P6_WEB_SRV1' is not currently running.>

<May 3, 2019 10:11:40 PM BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server P6_WEB_SRV1 : Exception while starting server 'P6_WEB_SRV1'>

************

<May 3, 2019 10:04:10 PM BST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)

        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        Truncated. see log file for complete stacktrace

Caused By: javax.security.auth.login.FailedLoginException: [Security:090303]Authentication

FIX is either boot.properties file or ldap under domain_name/servers/managedserver/data

moved ldap to ldapbkp and created ldap direcotry again by using mkdir command

[XXXXXXXd1@XXXXXXX6 data]$ ls

ldap  ldapbkp  nodemanager  store

[XXXXXXXd1@XXXXXXX6 data]$ pwd

/XXXXXXXd1/primapp/XXXXXXXppm/domain/XXXXXXXppm83/servers/P6_PRM1/data

copied the correct boot.properties file from another running managed server

[XXXXXXXd1@XXXXXXX6 nodemanager]$ ls

boot.properties  boot.propertiesbkp  P6_PRM1.lck  P6_PRM1.pid  P6_PRM1.state  P6_PRM1.statebkp  P6_PRM1.url  startup.properties

[XXXXXXXd1@XXXXXXX6 nodemanager]$

NOTE:Also you can try managedserver.state file remove and strat from console



ORACLE CLOUD RECOVERY ISSUE


****DCS-10045:Validation error encountered: Backup type is invalid.****


[root@testbackupdb ~]# dbcli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
7182911b-1733-4805-b351-081f6be6605b     Authentication key update for DCS_ADMIN                                     May 5, 2019 4:09:58 PM UTC          Success
ce8f602b-84c7-4295-982a-49098dca465d     Provisioning service creation                                               May 5, 2019 4:11:41 PM UTC          Success
47876bcb-50a5-42ac-9648-52221a12a974     SSH keys update                                                             May 5, 2019 5:03:22 PM UTC          Success
9d01bc5f-0831-48ed-a1fc-f0982e54feaa     SSH key delete                                                              May 5, 2019 5:05:14 PM UTC          Success
80f9b24e-1061-4906-88d1-78f6b824b454     create object store:b9a7iawWuBfGYFaNH7MC                                    May 6, 2019 3:14:45 AM UTC          Success
25845fda-ea88-40b0-bb54-3be86ec265f6     create backup config:b9a7iawWuBfGYFaNH7MC_BC                                May 6, 2019 3:15:31 AM UTC          Success
64903937-7519-415b-b658-186cc6bebbdd     update database : BKUPDB                                                    May 6, 2019 3:16:23 AM UTC          Success
f3d0309f-3bf0-4ec2-b3d7-2d2fcde8599f     Server Patching                                                             May 6, 2019 3:19:53 AM UTC          Success
4fa3b016-8448-4535-b901-a5edd6dc2c1c     Create Regular-L0 Backup with TAG-DBTRegular-L01557111640379VSL for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 3:19:56 AM UTC          Success
5e902ca8-f332-4c7f-86ca-afca1590e4b9     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 3:26:48 AM UTC          Success
251ba542-234e-40f6-a7a9-f3a1ef281cb8     DB Home Prechecks                                                           May 6, 2019 5:05:43 AM UTC          Success
bc5eb7cd-b320-453f-8dea-e296e645df58     Create Longterm Backup with TAG-DBTLongterm1557125463725Zmf for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 6:52:12 AM UTC          Success
43a924a2-6e16-43e0-a2e9-cc4cd1d6b53f     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 7:05:18 AM UTC          Success
eefab9b7-2b32-4327-9eff-bf55d99ca7eb     Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB           May 6, 2019 7:10:57 AM UTC          Failure

[root@testbackupdb ~]# dbcli describe-job  --jobid eefab9b7-2b32-4327-9eff-bf55d99ca7eb

Job details
----------------------------------------------------------------
                     ID:  eefab9b7-2b32-4327-9eff-bf55d99ca7eb
            Description:  Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB
                 Status:  Failure
                Created:  May 6, 2019 7:10:57 AM UTC
                Message:  DCS-10001:Internal error encountered: Failed to run RMAN command. Please refer log at location : testbackupdb: /opt/oracle/dcs/log/testbackupdb/rman/bkup/BKUPDB_fra1k8/rman_restore_2019-05-06_07-12-23-4741329873884168661.log.Failed to do restore validati

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_7173               May 6, 2019 7:11:07 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure
task:TaskSequential_7174                 May 6, 2019 7:11:07 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure
Database recovery validation             May 6, 2019 7:11:08 AM UTC          May 6, 2019 7:12:37 AM UTC          Failure

[
ISSUE is #####rman log error#showing inorder recovery of scn#####

validation succeeded for archived log
recovery will be done up to SCN 1761481
Media recovery start SCN is 1760987
Recovery must be done beyond SCN 1761012 to clear datafile fuzziness
could not locate pieces of backup set key 20
validation succeeded for backup piece
Finished restore at 2019/05/06 07:12:36

Recovery Manager complete.

we have used until time "05/06/2019 07:10:08"


SQL> select sequence#,first_change#,next_change# from v$log order by 1;

 SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
---------- ------------- ------------
        18       1761147      1761155
        19       1761155      1761444
        20       1761444   2.8147E+14


FIX : so i restored using until scn 1761444 

[root@testbackupdb ~]# dbcli list-jobs

ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
7182911b-1733-4805-b351-081f6be6605b     Authentication key update for DCS_ADMIN                                     May 5, 2019 4:09:58 PM UTC          Success
ce8f602b-84c7-4295-982a-49098dca465d     Provisioning service creation                                               May 5, 2019 4:11:41 PM UTC          Success
47876bcb-50a5-42ac-9648-52221a12a974     SSH keys update                                                             May 5, 2019 5:03:22 PM UTC          Success
9d01bc5f-0831-48ed-a1fc-f0982e54feaa     SSH key delete                                                              May 5, 2019 5:05:14 PM UTC          Success
80f9b24e-1061-4906-88d1-78f6b824b454     create object store:b9a7iawWuBfGYFaNH7MC                                    May 6, 2019 3:14:45 AM UTC          Success
25845fda-ea88-40b0-bb54-3be86ec265f6     create backup config:b9a7iawWuBfGYFaNH7MC_BC                                May 6, 2019 3:15:31 AM UTC          Success
64903937-7519-415b-b658-186cc6bebbdd     update database : BKUPDB                                                    May 6, 2019 3:16:23 AM UTC          Success
f3d0309f-3bf0-4ec2-b3d7-2d2fcde8599f     Server Patching                                                             May 6, 2019 3:19:53 AM UTC          Success
4fa3b016-8448-4535-b901-a5edd6dc2c1c     Create Regular-L0 Backup with TAG-DBTRegular-L01557111640379VSL for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 3:19:56 AM UTC          Success
5e902ca8-f332-4c7f-86ca-afca1590e4b9     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 3:26:48 AM UTC          Success
251ba542-234e-40f6-a7a9-f3a1ef281cb8     DB Home Prechecks                                                           May 6, 2019 5:05:43 AM UTC          Success
bc5eb7cd-b320-453f-8dea-e296e645df58     Create Longterm Backup with TAG-DBTLongterm1557125463725Zmf for Db:BKUPDB in OSS:b9a7iawWuBfGYFaNH7MC May 6, 2019 6:52:12 AM UTC          Success
43a924a2-6e16-43e0-a2e9-cc4cd1d6b53f     Delete Backup for Database name: BKUPDB_fra1k8                              May 6, 2019 7:05:18 AM UTC          Success
eefab9b7-2b32-4327-9eff-bf55d99ca7eb     Create recovery-pitr : time '05/06/2019 07:10:08' for db : BKUPDB           May 6, 2019 7:10:57 AM UTC          Failure
eab18e5c-ea75-41dd-a4ed-98aa209f2132     Create detailed Backup Report                                               May 7, 2019 2:16:37 AM UTC          Success
f22341d1-d835-4828-8bf3-b1d8f0158c9b     Create recovery-latest for db : BKUPDB                                      May 7, 2019 2:45:07 AM UTC          Failure
65f2fcf5-776c-4630-ab8f-76975ab0b935     Create recovery-scn : scn 1761444 for db : BKUPDB                           May 7, 2019 2:57:46 AM UTC          Success




[root@testbackupdb ~]# dbcli describe-job --jobid 65f2fcf5-776c-4630-ab8f-76975ab0b935

Job details
----------------------------------------------------------------
                     ID:  65f2fcf5-776c-4630-ab8f-76975ab0b935
            Description:  Create recovery-scn : scn 1761444 for db : BKUPDB
                 Status:  Running
                Created:  May 7, 2019 2:57:46 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             May 7, 2019 2:57:58 AM UTC          May 7, 2019 2:59:37 AM UTC          Success
Database recovery                        May 7, 2019 2:59:38 AM UTC          May 7, 2019 2:59:38 AM UTC          Running

[root@testbackupdb ~]# dbcli describe-job --jobid 65f2fcf5-776c-4630-ab8f-76975ab0b935

Job details
----------------------------------------------------------------
                     ID:  65f2fcf5-776c-4630-ab8f-76975ab0b935
            Description:  Create recovery-scn : scn 1761444 for db : BKUPDB
                 Status:  Success
                Created:  May 7, 2019 2:57:46 AM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database recovery validation             May 7, 2019 2:57:58 AM UTC          May 7, 2019 2:59:37 AM UTC          Success
Database recovery                        May 7, 2019 2:59:38 AM UTC          May 7, 2019 3:02:47 AM UTC          Success
Enable block change tracking             May 7, 2019 3:02:47 AM UTC          May 7, 2019 3:02:50 AM UTC          Success
Database opening                         May 7, 2019 3:02:50 AM UTC          May 7, 2019 3:03:44 AM UTC          Success
Database restart                         May 7, 2019 3:03:44 AM UTC          May 7, 2019 3:05:06 AM UTC          Success
Recovery metadata persistance            May 7, 2019 3:05:06 AM UTC          May 7, 2019 3:05:06 AM UTC          Success

[root@testbackupdb ~]#