CIT 470: Advanced Network and System Administration
Slide #1
CIT 470: Advanced Network and System Administration
Logging
CIT 470: Advanced Network and System Administration
Slide #2
Topics
System logsLogging policiesFinding logsSyslogSyslog serversLog monitoring
CIT 470: Advanced Network and System Administration
Slide #3
System Logs
Logs record status and error conditions.Where do log messages come from?KernelAccounting systemSystem servicesLogging methods:Service records own logs (apache, cron).Service usessyslogservice to manage logs.
CIT 470: Advanced Network and System Administration
Slide #4
Logging Policies
Throw away log data.Save for a while, then throw away.Rotate log filesArchive log files
CIT 470: Advanced Network and System Administration
Slide #5
How to choose a logging policy?
Are there any data retention requirements?How much disk space do you have?How quickly do you need to retrieve logs?Could you find the source of a security issue with the logs you keep?
CIT 470: Advanced Network and System Administration
Slide #6
Throwing Away
Not recommended.Leaves you unaware of:Software and hardware problemsSecurity incidentsIt may take time to detect an incident.Keep logs for at least a month or two.
CIT 470: Advanced Network and System Administration
Slide #7
Rotation
Keep backup files for each day/weeklogfilelogfile.1logfile.2logfile.3Rename files each day/week to move old ones back in list.Compress rotated logs to save disk space.Remove/archive logs that are X days old.
CIT 470: Advanced Network and System Administration
Slide #8
Rotation
#!/bin/shcd /var/logmv logfile.2 logfile.3mv logfile.1 logfile.2mv logfile logfile.1cp /dev/null logfilechmod 600 logfile
CIT 470: Advanced Network and System Administration
Slide #9
logrotate
Program to handle log rotation.Run via/etc/cron.daily.Configured via/etc/logrotate.conf.OptionsHow often to rotateHow long to keep logsCompression or notLog file permissionsPre- and post-rotate scripts
CIT 470: Advanced Network and System Administration
Slide #10
logrotate.conf
# rotate log files weeklyweekly# keep 4 weeks worth of backlogsrotate 4# create new (empty) log files after rotating oldcreate# uncomment if you want your log files compressed#compress# RPM packages drop log rotation information intoinclude /etc/logrotate.d# no packages own wtmp -- we'll rotate them here/var/log/wtmp {monthlycreate 0664 root utmprotate 1}
CIT 470: Advanced Network and System Administration
Slide #11
Archiving Logs
Store logs to archival media (tape.)Archive after X days/weeks.Should be part of regular backup plan.May want to save logs for all hosts together.
CIT 470: Advanced Network and System Administration
Slide #12
Finding Logs
Most logs are stored under/var/log/var/admCheck syslog's configuration/etc/syslog.confTo find other logs, read startup scripts/etc/init.d/*and manuals for services started by scripts.
CIT 470: Advanced Network and System Administration
Slide #13
Finding Logs
CIT 470: Advanced Network and System Administration
Slide #14
Syslog
Comprehensive logging system.Frees programmers from managing log files.Gives sysadmins control over log management.Sorts messages bySourcesImportanceRoutes messages to destinationsFilesNetworkTerminals
CIT 470: Advanced Network and System Administration
Slide #15
Syslog Components
SyslogDaemon that does actual logging.Additional daemon,klog, gets kernel messages.openlog, syslog, closelogC library routines to submit logs to syslog.loggerUser-level program to submit logs to syslog.Can use from shell scripts.
CIT 470: Advanced Network and System Administration
Slide #16
Example Syslog Messages
Feb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run-parts --report /etc/cron.hourly)Feb 11 10:37:22 localhost -- MARK --Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to 192.168.1.1 port 67Feb 11 10:51:11 localhost dhclient: DHCPACK from 10.42.1.1Feb 11 10:51:11 localhost dhclient: bound to 10.42.1.55 -- renewal in 35330 seconds.Feb 11 14:37:22 localhost -- MARK --Feb 11 14:44:21 localhost mysqld[7340]: 060211 14:44:21 /usr/sbin/mysqld: Normal shutdownFeb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT!Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:218.38.30.101
CIT 470: Advanced Network and System Administration
Slide #17
Configuring Syslog
Configured in/etc/syslog.confFormat:selector<Tab>actionEx:mail.info /var/log/mail.logSelector componentsSource (facility)List of facilities separated by commas or *.Importance (level)Can be none or *
CIT 470: Advanced Network and System Administration
Slide #18
/etc/syslog.conf
# Log anything (except mail) of level info or higher.# Don't log private authentication messages!*.info;mail.none;authpriv.none;cron.none /var/log/messages# The authpriv file has restricted access.authpriv.* /var/log/secure# Log all the mail messages in one place.mail.* /var/log/maillog# Log cron stuffcron.* /var/log/cron# Everybody gets emergency messages*.emerg *# Save news errors of level crit and higher in a special file.uucp,news.crit /var/log/spooler# Save boot messages also to boot.loglocal7.* /var/log/boot.log
CIT 470: Advanced Network and System Administration
Slide #19
Syslog Facilities
CIT 470: Advanced Network and System Administration
Slide #20
Syslog Levels
CIT 470: Advanced Network and System Administration
Slide #21
Syslog Actions
CIT 470: Advanced Network and System Administration
Slide #22
Testing Syslog
stu> for i in {debug,info,notice,warning,err,crit,alert,emerg}> do> logger -p daemon.$i "Test message for daemon, level $i"> donestu> tail /var/log/daemon.logFeb 11 15:57:00 localhost stu: Test message for daemon, level debugFeb 11 15:57:00 localhost stu: Test message for daemon, level infoFeb 11 15:57:00 localhost stu: Test message for daemon, level noticeFeb 11 15:57:00 localhost stu: Test message for daemon, level warningFeb 11 15:57:00 localhost stu: Test message for daemon, level errFeb 11 15:57:00 localhost stu: Test message for daemon, level critFeb 11 15:57:00 localhost stu: Test message for daemon, level alertFeb 11 15:57:00 localhost stu: Test message for daemon, level emerg
CIT 470: Advanced Network and System Administration
Slide #23
Syslog Variants
Some use m4 macrosauth.notice ifdef(‘LOGHOST’, ‘/var/log/authlog’, ‘@loghost’)Red Hat Linux variantsAllows spaces as separators.New operators: = (this priority only)Ex:mail.=infoNew operators: ! (except this pri and higher)Ex:mail.info,mail.!err
CIT 470: Advanced Network and System Administration
Slide #24
Syslog NG
Free drop-in replacement for syslog.More configurableSave logs to templated location (auto-rotates.)Filter logs based on program, time, message, etc.Message format customization.Allows easy logging to remote database.Improved networkingTCP support as well as UDP.Improved securityDoesn’t trust hostnames in remote messages.TCP transmission permits encrypted tunneling (stunnel.)
CIT 470: Advanced Network and System Administration
Slide #25
Log Servers
Collect all syslog data on one server.Allows logging to scale to large networks.Logs can be correlated across machines.Security-sensitive logs not on compromised host.Routers and diskless-hosts must log to a server.Need twosyslog.conffilesClient: sends all logs across network to server.Server: saves logs to database or local files.
CIT 470: Advanced Network and System Administration
Slide #26
Log Monitoring
Too much data for a human to process.Logs arrive 24x7 too.Use an automatic monitoring programTriggers on patterns found in log.Examples:logwatch,swatch# 3ware logswatchfor /(?i)3w-xxxx.+no longer fault tolerant/mail=root,subject=LW warn: disk 3ware RAID not fault tolerantthrottle 1:00:00,use=regex
CIT 470: Advanced Network and System Administration
Slide #27
References
Michael Bower,Building Secure Servers with Linux, O’Reilly, 2005.Aeleen Frisch, Essential System Administration, 3rdedition, O’Reilly, 2002.Jeremy Mate, “Log Analysis with Swatch,”http://sial.org/howto/logging/swatch/, 2005.Jeremy Mate, “Logging with syslog-ng,”http://sial.org/howto/logging/syslog-ng/, 2005.Evi Nemeth et al,UNIX System Administration Handbook, 3rdedition, Prentice Hall, 2001.Shelley Powers et. al.,UNIX Power Tools, 3rdedition, O’Reilly, 2002.Syslog-ng FAQ, http://www.campin.net/syslog-ng/faq.html.
0
Embed
Upload