[HPADM] SUMMARIZE: HP-UX: syslog.log: strange entries

From: Rossi.ettore (rossi.ettore@email.it)
Date: Fri May 27 2005 - 08:24:56 EDT


Hi Jee,

this morning customer said me that he solved the problem by monitoring
\"pipe of /dev/log\". If you have an idea please give me it. Sincerely I
don\'t understand it, but he said that found the process was writing in
syslog.log. Stopped out it the problem was solved.

Thanks anyway for you useful suggestions!

Have a nice weekend you and all hp-ux admins!

Regards,
Fabio

--------- Original Message --------
Da: \"jee\" <jee@eresmas.net>
To:
Oggetto: Re: [HPADM] HP-UX: syslog.log: strange entries
Data: 22/05/05 16:54

Hi Fabio,

and sorry about the rossi thing. I didn\'t pay much attention to it ;)

I don\'t know of any other ways of tracing it than back-tracing it. You have

to guess where the messages come from. But knowing how the thing works
should
suffice.

You have to look for the files using logger and then see which ones match
the
rules on syslog.conf. It is not possible to tell which script is doing it
from the ouput in the syslog log file. The program sending it is not telling

that. What you need to do is to add a tag to each script using logger.

I am pretty sure that the rule that is creating the messages is
\"*.info;mail.none /var/adm/syslog/syslog.log\"
this matches all facilities with levels [0-6] (i.e. [emerg-info])

I have just made a little test on my FreeBSD box:
1) edit syslog and include:
*.info /tmp/syslog.test
2) restart syslog (killall -1 syslogd)
3) start tail -f /tmp/syslog.test on a different session
4) run the following commands:
$ logger -p user.info -t USER_INFO LOGGERTEST
$ logger -p user.warn -t USER_WARN LOGGERTEST
$ logger -p user.notice -t USER_NOTICE LOGGERTEST

5) output of tail -f:

May 22 13:42:24 qemu1 syslogd: restart
May 22 13:43:55 qemu1 USER_INFO: LOGGERTEST
May 22 13:44:00 qemu1 /usr/sbin/cron[7458]: (operator) CMD
(/usr/libexec/save-entropy)
May 22 13:44:24 qemu1 USER_WARN: LOGGERTEST
May 22 13:44:49 qemu1 USER_NOTICE: LOGGERTEST
May 22 13:45:00 qemu1 /usr/sbin/cron[7475]: (root) CMD (/usr/libexec/atrun)

As you can see the rule *.info catches user.info, user.warn and user.notice
messages from logger. It matches user at least, but also other facilities
(because of the \"*\") as long as the priority is info or higher (lower
number).

If you were to use level debug it wouldn\'t produce any output on the tail
-f
session (debug is less critical -- level 7)

So, I would search for files with the pattern logger.
Make a list of those files containing logger and put a unique tag on them
(make a backup copy first). I would use ${0##*/} as the tag, this way you
know what script sent the message (even if you rename or reuse the script in

the future).

Try logger with different levels (modify a tag accordingly, i.e. add a
suffix with the facility and level to the tag). It can be scripted

If you change or add test rules in syslog.conf it is always useful to write
a
quick and dirty script to check just the rules first:

#!/usr/bin/ksh
# script: syslog_test.ksh
TAG=${0##*/}
FACILITY=user # or do a for-in-do-done with more values
MSG=\"MY_TEST_MESSAGE\"

for LEVEL in notice warning info debug
do
logger -p ${FACILITY}.${LEVEL} -t ${TAG}_${FACILITY}_${LEVEL} \"$MSG\"
sleep 10
done

I hope this helps,
Javier

On Sunday 22 May 2005 2:08, Rossi.ettore wrote:
> Hi Jee,
>
> even if my mail address is Rossi Ettore, I am Fabio :-)
> Thanks for your reply.
>
> I already saw syslog.conf and it doesn\\\'t contain no extra configuration
> than usual:
>
> # @(#)B11.23_LR
> #
> # syslogd configuration file.
> #
> # See syslogd(1M) for information about the format of this file.
> #
> mail.debug /var/adm/syslog/mail.log
> *.info;mail.none /var/adm/syslog/syslog.log
> *.alert /dev/console
> *.alert root
> *.emerg *
>
> Anyway I will try to look for \\\"logger\\\"string.
>
> Do you think there are other ways to trace where they come from? I
don\\\'t
> think and I don\\\'t know other ways.
>
> Thanks again!
>
> Regards,
> Fabio
>
>
>
> --------- Original Message --------
> Da: \\\"jee\\\" <jee@eresmas.net>
> To:
> Oggetto: Re: [HPADM] HP-UX: syslog.log: strange entries
> Data: 22/05/05 14:50
>
>
>
> Hi Rossi,
>
> as per your description of the problem I see that an application is
sending
> messages to syslogd with facility.level pairs:
> 1.4 == user.warning
> 1.5 == user.notice
> 1.6 == user.info
>
> you mat have a rule on your syslog.conf that matches them all.
> This could be:
> a) user.warning
> everything sent as \\\"user\\\" with level > warning (ie, warning, notice,
> info)
>
> will match
> b) *.warning
> everything sent by any facility with level > warning will match
> It may happen that only the user facility (number \\\"1\\\") is doing it.
>
> If there is a script sending the messages to syslog you won\\\'t
necessarily
> see
> the error messages unless you send one explicitly using the logger
command.
>
> The pattern you have to look for is \\\"logger\\\". This is the command
sending
> the
> messages to syslogd. Once you have identified those scripts, then use the
> -t
>
> (tag) option for logger. This way you can trace which messages are sent by
> whom.
>
> hint: You can play with logger from the command line using for instance:
>
> logger -p user.warning -t TESTTAG_WARN \\\"my test message\\\"
> logger -p user.notice -t TESTTAG_NOTICE \\\"my test message\\\"
> logger -p user. debug -t TESTTAG_DEBUG \\\"my test message\\\"
>
> (tail -f your syslog file from another session to see how it works)
>
> It would be interesting to chek your syslog.conf and see if those messages
> go
> somewhere else too.
> Any questions, please, let me know.
>
> Regards,
> Javier
>
> On Sunday 22 May 2005 11:43, you wrote:
> > Good evening HP-UX admin people,
> >
> > I have a customer with an HP-UX 11.23 system which has strange entries
in
> > /var/adm/syslog/syslog.log:
> >
> >
> > Feb 16 17:07:06 rx1600 syslog: Error reading field (3) for AN
> > Feb 16 17:17:28 rx1600 syslog: read error (Error 0)
> > Feb 16 17:17:28 rx1600 syslog: Abnormal end
> > Feb 16 17:17:28 rx1600 syslog: Error reading field (3) for AN
> > Feb 16 17:17:28 rx1600 syslog: Error reading field 21
> > ...........
> > so far too
> >
> >
> > These messages happen every few minutes.
> > By debug of syslogd (with -d option) I could understand they are at the
> > user level messages:
> >
> >
> > logmsg: pri 15, flags 0, from rx1600, msg May 18 17:22:02 syslog: read
> > error (Error 0) Logging to FILE /var/adm/syslog/syslog.log readfds =
0xe8
> > 0x3 0x6 0x7 0x5 got a message (1, 0x8)
> > logmsg: pri 16, flags 0, from rx1600, msg May 18 17:22:02 syslog:
> > Abnormal end Logging to FILE /var/adm/syslog/syslog.log readfds = 0xe8
> > 0x3 0x6 0x7 0x5 got a message (1, 0x8)
> > logmsg: pri 14, flags 0, from rx1600, msg May 18 17:22:02 syslog: Error
> > reading field (3) for AN Logging to FILE /var/adm/syslog/syslog.log
>
> readfds
>
> > = 0xe8 0x3 0x6 0x7 0x5 got a message (1, 0x8)
> > logmsg: pri 14, flags 0, from rx1600, msg May 18 17:22:02 syslog: Error
> > reading field 21 Logging to FILE /var/adm/syslog/syslog.log readfds =
> > 0xe8 0x3 0x6 0x7 0x5 got a message (1, 0x8)
> > About the /usr/include/syslog.h file:
> >
> > /*
> > * Facility codes
> > */
> > #define LOG_KERN (0<<3) /* kernel messages */
> > #define LOG_USER (1<<3) /* random user-level messages */
> > #define LOG_MAIL (2<<3) /* mail system */
> > #define LOG_DAEMON (3<<3) /* system daemons */
> > #define LOG_AUTH (4<<3) /* security/authorization messages */
> > #define LOG_SYSLOG (5<<3) /* messages generated internally by syslogd
> > */
> > #define LOG_LPR (6<<3) /* line printer subsystem */
> > ....................................................
> >
> > /*
> > * Priorities (these are ordered)
> > */
> > #define LOG_EMERG 0 /* system is unusable */
> > #define LOG_ALERT 1 /* action must be taken immediately */
> > #define LOG_CRIT 2 /* critical conditions */
> > #define LOG_ERR 3 /* error conditions */
> > #define LOG_WARNING 4 /* warning conditions */
> > #define LOG_NOTICE 5 /* normal but signification condition */
> > #define LOG_INFO 6 /* informational */
> > #define LOG_DEBUG 7 /* debug-level messages */
> >
> >
> >
> > In bold you will find the facilities whence come those messages. So for
> > example
> >
> > logmsg: pri 15
> >
> > means
> >
> > facility code = 1
> > priorities = 5
> >
> > Then all messages come from \\\\\\\"random user-level messages\\\\\\\"
and no
>
> daemons
>
> > or other known facilties of the operating system.
> >
> > I think the reason is a batch file of an user running on the system so I
> > suggested him for example:
> >
> > find / -type f | xargs grep -l \\\\\\\"read error\\\\\\\"
> > find / -type f | xargs grep -l \\\\\\\"abnormal end\\\\\\\"
> >
> > to looking for those strings in the some scripts or programs but nothing
> > yet, no useful information found.
> >
> > Please I ask your experience about how to discover what or who is
sending
> > those messages in the syslog.log file.
> >
> > I already provided to customer information about how to separate those
>
> user
>
> > level messages from syslog.log to another log file but it is not enough
> > because customer wants to know where they come from.
> >
> > Thanks in advance for your replies, I will summarize.
> >
> > Best regards,
> > Fabio Porcelli
> > --
> > Email.it, the professional e-mail, gratis per te: http://www.email.it/f
> >
> > Sponsor:
> > Audio, Video, HI-FI...oltre 2.000 prodotti di alta qualità a prezzi da
> > sogno solo su Visualdream.it
> > Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=2955&d=20050522
>
> --
> Email.it, the professional e-mail, gratis per te: http://www.email.it/f
>
> Sponsor:
> Biscotti perfetti? Metti la pasta dentro allo Sparabiscotti e...click
> click... biscotti pronti per essere infornati!
> Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=2745&d=20050522
 --
 Email.it, the professional e-mail, gratis per te: http://www.email.it/f
 
 Sponsor:
 Scarica Sweety sul tuo cellulare.
La prima suoneria è GRATIS!
 Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=3536&d=20050527

--
             ---> Please post QUESTIONS and SUMMARIES only!! <---
        To subscribe/unsubscribe to this list, contact majordomo@dutchworks.nl
       Name: hpux-admin@dutchworks.nl     Owner: owner-hpux-admin@dutchworks.nl
 
 Archives:  ftp.dutchworks.nl:/pub/digests/hpux-admin       (FTP, browse only)
            http://www.dutchworks.nl/htbin/hpsysadmin   (Web, browse & search)


This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 11:02:48 EDT