From: Tarasyuk Nik (NTarasyuk@snowyhydro.com.au)
Date: Sun May 05 2002 - 19:13:30 EDT
Hi Managers
A lot of thanks to Denise Dumas, John Ferlan, Oisin McGuinness, Ann Majeske, and Jim Belonis.
These are the suggestions that were given
1. set up audit subsystem
2. remove sialog - it's used only for debugging
3. look at cron jobs and rc3.d scripts, something might have been failing and retrying.
4. check auth.log
The problem was fixed thanks to Dr. Watson who has noticed that
one of very usefull services on remote server 100 miles away doesn't work.
That service was using little server program running on our ill server here.
So we have checked this server program and discovered that
its start-up script has a plain error in it.
Instead of calling the actual executable, it was calling itself over and over again.
The problem come up because we have rebooted server first time in a year.
So good old good luck helped us this time,
I wonder if audit subsystem could have tracked down the guilty service
if having been set up.
If yes, it might be something worth doing.
Good luck to everyone.
Nik Tarasyuk
Software Engineer
Snowy Hydro
Australia
-----Original Message-----
/var on one our servers got filled up.
The culprit was sialog, which was full of "Successful authentication for su from root to root" messages.
We cleaned the log, it started to grow again fast.
We've done reboot, it did NOT help.
CPU's idle time is zero, top shows that no specific process takes CPU time, but
CPU system time is high.
iowait is exremely high, network utilization is low, disk utilization is high.
So, some process does hundreds of su's per second, and it's logged by sialogd.
How to find out which one?
We are running 4.0f kit 4 on ES40.
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 10:48:40 EDT