Chinaunix首页 | 论坛 | 博客
  • 博客访问: 1746002
  • 博文数量: 370
  • 博客积分: 1860
  • 博客等级: 上尉
  • 技术积分: 4113
  • 用 户 组: 普通用户
  • 注册时间: 2008-04-07 08:51
个人简介

偶尔有空上来看看

文章分类

全部博文(370)

文章存档

2021年(101)

2020年(115)

2019年(11)

2018年(9)

2017年(9)

2016年(17)

2015年(7)

2014年(4)

2013年(1)

2012年(11)

2011年(27)

2010年(35)

2009年(11)

2008年(11)

分类: Oracle

2021-04-12 16:14:17


aix平台 单机 10.2.0.4 突然实例宕机

alert.log 中记录

  1. Mon Apr 12 13:25:58 2021
  2. Thread 1 advanced to log sequence 140477 (LGWR switch)
  3.   Current log# 9 seq# 140477 mem# 0: /oradata/orcl/redo09.log
  4. Mon Apr 12 13:26:38 2021
  5. KCF: write/open error block=0x6706 online=1
  6.      file=38 /oradata/orcl/TSDAT02_08.dbf
  7.      error=27063 txt: 'IBM AIX RISC System/6000 Error: 16: Device busy
  8. Additional information: -1
  9. Additional information: 8192'
  10. Automatic datafile offline due to write error on
  11. file 38: /oradata/orcl/TSDAT02_08.dbf
  12. Mon Apr 12 13:26:43 2021
  13. Errors in file /home/oracle/admin/orcl/udump/orcl_ora_14155964.trc:
  14. ORA-00376: file 38 cannot be read at this time
  15. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
  16. ORA-00376: file 38 cannot be read at this time
  17. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
  18. Mon Apr 12 13:26:49 2021
  19. Errors in file /home/oracle/admin/orcl/bdump/orcl_pmon_34537506.trc:
  20. ORA-00376: file 38 cannot be read at this time
  21. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
  22. Mon Apr 12 13:42:08 2021
  23. Errors in file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc:
  24. ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 36307002'
  25. Mon Apr 12 13:42:09 2021
  26. System State dumped to trace file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc
  27. Killing enqueue blocker (pid=36307002) on resource CF-00000000-00000000
  28.  by killing session 552.1
  29. Mon Apr 12 13:47:11 2021
  30. Errors in file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc:
  31. ORA-00494: enqueue [CF] held for too long (more than 900 seconds) by 'inst 1, osid 36307002'
  32. Mon Apr 12 13:47:12 2021
  33. System State dumped to trace file /home/oracle/admin/orcl/bdump/orcl_lgwr_53149820.trc
  34. Killing enqueue blocker (pid=36307002) on resource CF-00000000-00000000
  35.  by terminating the process
  36. LGWR: terminating instance due to error 2103
  37. Instance terminated by LGWR, pid = 53149820
看到读写38号文件时 提示error=27063 txt: 'IBM AIX RISC System/6000 Error: 16: Device busy

查看os日志

  1. [ oracle ]host1:/home/oracle/admin/orcl/bdump:errpt|head
  2. IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
  3. DCB47997 0412142821 T H hdisk112 DISK OPERATION ERROR
  4. DCB47997 0412142821 T H hdisk119 DISK OPERATION ERROR
  5. B6267342 0412142821 P H hdisk119 DISK OPERATION ERROR
  6. B6267342 0412142821 P H hdisk140 DISK OPERATION ERROR
  7. B6267342 0412142821 P H hdisk119 DISK OPERATION ERROR
  8. B6267342 0412142821 P H hdisk112 DISK OPERATION ERROR
  9. DCB47997 0412142721 T H hdisk112 DISK OPERATION ERROR
  10. DCB47997 0412142721 T H hdisk112 DISK OPERATION ERROR
  11. B6267342 0412142721 P H hdisk119 DISK OPERATION ERROR

[ oracle ]host1:/home/oracle/admin/orcl/bdump:errpt -aj DCB47997|more
---------------------------------------------------------------------------
LABEL:          SC_DISK_ERR4
IDENTIFIER:     DCB47997


Date/Time:       Mon Apr 12 14:28:34 GMT+08:00 2021
Sequence Number: 847721
Machine Id:      00F710984C00
Node Id:         host1
Class:           H
Type:            TEMP
WPAR:            Global
Resource Name:   hdisk112        
Resource Class:  disk
Resource Type:   CLAR_FC_VRAID
Location:        U5802.001.9K8N405-P1-C2-T2-W500601690960398D-L3000000000000


VPD:             
        Manufacturer................DGC     
        Machine Type and Model......VRAID           
        ROS Level and ID............0533
        Serial Number...............CETV2173400018
        Subsystem Vendor/Device ID..VNX5600        
        Device Specific.(PQ)........00
        Device Specific.(VS)........255F081CECCL
        Device Specific.(UI)........600601600A804500571D3D8381C2EA11
        FRU Label...................0025
        Device Specific.(Z0)........10
        Device Specific.(Z1)........10


Description
DISK OPERATION ERROR


Probable Causes
MEDIA
DASD DEVICE


User Causes
MEDIA DEFECTIVE


        Recommended Actions
        FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
        PERFORM PROBLEM DETERMINATION PROCEDURES


Failure Causes
MEDIA
DISK DRIVE


        Recommended Actions
        FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
        PERFORM PROBLEM DETERMINATION PROCEDURES


Detail Data
PATH ID
           0
SENSE DATA
0A00 2800 2117 0920 0004 0004 0000 0000 0000 0000 0000 0000 0118 0000 0000 0000 
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
0000 0000 0000 034A 045C 0007 8000 0000 0000 0000 0000 0000 0000 0000 0083 0000 
0000 003D 001D 
---------------------------------------------------------------------------


据说是磁盘坏道或者链路闪断导致。

先处理故障,直接启动实例
startup 耗时20分钟终于起来了
实例启动后,38号文件一直报错

  1. Mon Apr 12 14:22:33 2021
  2. Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
  3. ORA-00376: file 38 cannot be read at this time
  4. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
  5. ORACLE Instance orcl (pid = 9) - Error 376 encountered while recovering transaction (30, 43) on object 1192947.
  6. Mon Apr 12 14:22:34 2021
  7. Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
  8. ORA-00376: file 38 cannot be read at this time
  9. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
  10. ORACLE Instance orcl (pid = 9) - Error 376 encountered while recovering transaction (30, 43) on object 1192947.
  11. Mon Apr 12 14:22:35 2021
  12. Errors in file /home/oracle/admin/orcl/bdump/orcl_smon_34537570.trc:
  13. ORA-00376: file 38 cannot be read at this time
  14. ORA-01110: data file 38: '/oradata/orcl/TSDAT02_08.dbf'
检查38号数据文件
 select file#,status,bytes/1024/1024,name from v$datafile;

  1. ...
  2. 37 ONLINE 30720 /oradata/orcl/TSDAT03_12.dbf
  3. 38 RECOVER 30720 /oradata/orcl/TSDAT02_08.dbf
  4. 39 ONLINE 30720/oradata/orcl/TSDAT03_13.dbf
  5. ...

归档模式,问题发现的比较及时,赶紧修复
recover datafile 38;
alter database datafile 38 online;
很快顺利修复,又躲过一劫。

参考:
HOW TO RECOVER OFFLINE DROPPED DATAFILE IN ARCHIVELOG MODE (Doc ID 286355.1)
阅读(124) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~