From: Alan Douglas (alan.douglas@mentec.ie)
Date: Tue Nov 18 2003 - 05:09:21 EST
Thanks for you help, apologies for the delay in summarising the results. Got
some good ideas from the usual suspects - Alan Rollow, Ryan Frantz, David
Stack and others, thanks to all.
We were unable to find out exactly what happened and stopped short of
getting HP to have a look. We are keeping a close eye on the system, but at
this point it is being put into the black hole of problems.
Below are a couple of the ideas forwarded.
Thanks,
Alan
- You can check the .sh_history file under root to see if the shutdown
command
was given. Also if you have decevent installed, it will also log a shutdown
command.
- You can always check the '.history' file in / to see if whoever was su'd
in
issued the 'shutdown' command or one of it's cousins
- Crash dumps occur when the system decides to take itself down, either in
response to a software inconsistency or
machine check. If, for example, the system's power supplies or internal
power/temperature sensors detected a problem,
it could shutdown itself down and the operating system would never be the
wiser. Some systems are pretty smart and
might keep track of those things, but you need to shutdown the system to get
to the console (except for the really
big systems that have an independent console).
A typical user initiiated shutdown or halt, should leave a trace somewhere;
either in the binary error log or one
of the other log files.
Original Question
> We had a system down when we arrived in this morning. We had a DBA logged
> in with SU access. We diagnosed the problem as below. Is there anything
> elso to check. I cant get into a blame game with no proof of wrong doing.
> As it stands, the system may have shutdown, but I cant see how.
> An help would be appreciated.
>
> Alan
>
>
> 1. We looked for a crash dump file. There is none present.
>
> 2. We looked at all the log files. The log files showed no evidence of any
> unusual occurrences. The boot sequence gave no indication of an unclean
> system shutdown.
> checked messages, biblog, syslogs etc
>
> 3. We ran a sys_check on the system, to make sure we hadnt missed anything
> silly. Again we found nothing to suggest the system had crashed.
>
> 4. The system is monitored by the HP Collect utility. From this utility we
> were able to pinpoint the system going down to within 10/15 seconds.
>
> 5. Once we had the exact downtime of the system, we were able to use the
> collect utility to see what was running and who was logged in at that
time.
> No unusual processes were running.
>
> 6. Oracle was not shut down cleanly on the system.
>
>
>
>
> CONFIDENTIALITY NOTICE
> This email is private and confidential and may contain legally privileged
> information.If you are not named above as an addressee it may be unlawful
> for you to read, copy, distribute, disclose or otherwise use the
> information in this email. If you are not the intended recipient of this
> email please contact our systems manager.
> Thank you.
> Any views or opinions presented are solely those of the author and do not
> necessarily represent those of the company or its associated companies
> unless otherwise specifically stated.
CONFIDENTIALITY NOTICE
This email is private and confidential and may contain legally privileged
information.If you are not named above as an addressee it may be unlawful
for you to read, copy, distribute, disclose or otherwise use the information
in this email. If you are not the intended recipient of this email please
contact our systems manager.
Thank you.
Any views or opinions presented are solely those of the author and do not
necessarily represent those of the company or its associated companies
unless otherwise specifically stated.
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:49:44 EDT