www.routermonkey.org

Cisco PIX 7.2 logging to php-syslog-ng on linux RH FC4

Mon, 04 Feb 2008 15:21:41 -0600

Cisco PIX Syslog To PHP-Syslog-ng

PIX v7.2(2)

Barney Gaumer – 02/04/2008

Hopefully this document will help you to install/configure and use PHP-Syslog-ng with your Pix to keep log history for security infrastructure in a manageable format.

Prerequisites:

Linux (preferably RedHat FC4)

mySQL 4.x

Apache 2.x

PHP 5

Syslog-ng 1.6.x

PHP-Syslog-ng 2.8 or higher (I’m using 2.8)
http://downloads.sourceforge.net/php-syslog-ng/phpsyslogng-2.8.tar.gz?modtime=1120681704&big_mirror=0

Starting off, if you don’t have all of the above mentioned packages installed, you can use YUM on many linux distro’s to download and install packages and their dependencies.

Example:
yum install mysqld (this will install mysql client and server)

yum install httpd (this will install Apache web server)

yum install php-mysql (this will install PHP with mysql support)

yum install syslog-ng (this will install syslog-ng)

Now at the shell prompt on your linux box, type the following to automatically start mysqld, apache and syslog-ng.

chkconfig --levels 235 mysqld on
chkconfig --levels 235 syslog-ng on
chkconfig --levels 235 httpd on

This will set each service to start for run levels 2,3 and 5. Note: If you have problems with syslog-ng writing to mysql, type chkconfig --levels 235 syslog-ng off to stop the /etc/init.d/syslog-ng itteration and start it in /etc/rc.local by adding the following:

/sbin/syslog-ng

Next, cd into /etc/syslog-ng and using vi, edit syslog-ng.conf

Comment out the following:

# source s_sys {
# file ("/proc/kmsg" log_prefix("kernel: "));
# unix-stream ("/dev/log");
# unix-stream ("/etc/log/log");
# internal();
# udp(ip(0.0.0.0) port(514));
# };

And comment out:

#log { source(s_sys); filter(f_filter1); destination(d_cons); };
#log { source(s_sys); filter(f_filter2); destination(d_mesg); };
#log { source(s_sys); filter(f_filter3); destination(d_auth); };
#log { source(s_sys); filter(f_filter4); destination(d_mail); };
#log { source(s_sys); filter(f_filter5); destination(d_mlal); };
#log { source(s_sys); filter(f_filter6); destination(d_spol); };
#log { source(s_sys); filter(f_filter7); destination(d_boot); };
#log { source(s_sys); filter(f_filter8); destination(d_cron); };

Then add the following:

destination d_mysql {

   pipe("/var/log/mysql.pipe"
    template("INSERT INTO logs
    (host, facility, priority, level, tag, datetime, program, msg)
      VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
       '$PROGRAM', '$MSG' );n") template-escape(yes));

};

# };

log { source(net); destination(d_mysql);

};

source src { unix-dgram("/dev/log"); internal(); };
#source src { unix-dgram("/etc/log/log"); internal(); };
source net { udp(); };

In this last implimentation, I had some trouble with the unix socket file /etc/log/log so I used the standard /dev/log which seemed to work fine?

Now tweak the destination/filters:

destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog" sync(10)); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_mlal { usertty("*"); };

#filter f_filter1   { facility(kern); };
filter f_filter2   { level(info..emerg) and
                           not facility(mail,authpriv,cron); };
filter f_filter3   { facility(authpriv); };
filter f_filter4   { facility(mail); };
filter f_filter5   { level(emerg); };
filter f_filter6   { facility(uucp) or
                         (facility(news) and level(crit..emerg)); };
filter f_filter7   { facility(local7); };
filter f_filter8   { facility(cron); };

Now you need to create a directory located in / called scripts
mkdir /scripts then cd into /scripts

Create a file called syslog2mysql.sh by typing vi syslog2mysql.sh

This is were we define the fifo file mysq.pipe that allows syslog-ng to write to your mysql database:

#!/bin/bash

if [ ! -e /var/log/mysql.pipe ]
then
mkfifo /var/log/mysql.pipe
fi
while [ -e /var/log/mysql.pipe ]
do
mysql -u root --password=password syslog < /var/log/mysql.pipe >/dev/null
done

Make the directory phpsyslog in /var/www/html. You should have downloaded phpsyslogng-2.8.tar.gz to your system. Copy the file to /var/www/html/phpsyslog. CD into /var/www/html/phpsyslog and unpack phpsyslogng-2.8.tar.gz by typing tar zxvf phpsyslogng-2.8.tar.gz.

Now is a good time to setup the root password for mysql:

your-box# mysql -u root
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('PASSWD');

Whatever you changed the root password for mysql to is what you will most likely populate config.php db authentication parms with.

Ok - Now you need to CD into the directory /var/www/html/phpsyslog/scripts and run the following:

mysql -u root -p password < ./dbsetup.sql

This will setup the syslog database in mysql. If any of this "bonks" on you, type /etc/init.d/mysqld restart just to make sure that mysql is running.

Now cd into /var/www/html/phpsyslog/config and edit the file config.php to change the database user, password and site URL:

config.php

<?php
// Copyright (C) 2005 Claus Lund, clauslund@gmail.com

//========================================================================
// BEGIN: MISC FUNCTIONALITY
//========================================================================
define('PAGETITLE', 'php-syslog-ng');
define('VERSION', '2.8');

// COUNT_ROWS determines if results also display the number of total
// entries in the query. You need to have MySQL 4.0.0 or later for this
// to work. If you are using an older version of MySQL then disable this
// feature.
define('COUNT_ROWS', TRUE);

// DEFAULTLOGTABLE is the name of the table where new syslog entries are
// stored.
define('DEFAULTLOGTABLE', 'logs');

// MERGELOGTABLE is the name of the merge table. This feature should
// generally only be used in combination with the logrotate script.
// If it is enabled then the logrotate script will create a merge table
// of all the log tables. Be aware that you need to be a little careful
// when using merge tables so it is recommended that you only use it with
// the logrotate script. The merge table will also be the one used to
// populate the HOSTS and FACILITY fields on the search form.
define('MERGELOGTABLE', 'all_logs');

// If LOGROTATERETENTION is defined then it determines how many days logs
// are kept. Old log tables with a datestamp older than LOGROTATERETENTION
// days will be dropped.
define('LOGROTATERETENTION', 90);
//========================================================================
// END: MISC FUNCTIONALITY
//========================================================================

//========================================================================
// BEGIN: DATABASE CONNECTION INFO
//========================================================================
// DBUSER is the name of the basic user.
define('DBUSER', 'root');

// DBUSERPW is DBUSER's database password.
define('DBUSERPW', 'password');

// DBADMIN is the name of the admin user.
define('DBADMIN', 'root');

// DBADMINPW is DBADMIN's database password.
define('DBADMINPW', 'password');

// DBNAME is the name of the database you are using.
define('DBNAME', 'syslog');

// DBHOST is the host where the MySQL server is running.
define('DBHOST', 'localhost');

// DBPORT is the port where the MySQL server is listening.
// The default port is 3306.
define('DBPORT', '3306');
//========================================================================
// END: DATABASE CONNECTION INFO
//========================================================================

//========================================================================
// BEGIN: AUTHENTICATION
//========================================================================
define('REQUIRE_AUTH', TRUE);
define('AUTHTABLENAME', 'users');

// Authentication has two modes:
// 1) You renew the session on every page view. This means you can have a
//    tail screen running and the session will never expire as long as you
//    refresh the screen before SESSION_EXP_TIME. This is the default.
// 2) The session is timed from the time you login. The session is only
//    refreshed on login.
define('RENEW_SESSION_ON_EACH_PAGE', TRUE);

// SESSION_EXP_TIME is seconds until the session expires.
define('SESSION_EXP_TIME', '3600');

// Set the URL to php-syslog-ng. If you don't then the login screen will
// not be able to redirect users automatically after a successful login.
define('URL', 'http://172.20.1.7/phpsyslog/');
//========================================================================
// END: AUTHENTICATION
//========================================================================

//========================================================================
// BEGIN: ACCESS CONTROL
//========================================================================
// Access Control Lists allows you to specify what individual users have
// access to. Access Control requires Authentication to have any effect.
<?php
// Copyright (C) 2005 Claus Lund, clauslund@gmail.com

//========================================================================
// BEGIN: MISC FUNCTIONALITY
//========================================================================
define('PAGETITLE', 'php-syslog-ng');
define('VERSION', '2.8');

// COUNT_ROWS determines if results also display the number of total
// entries in the query. You need to have MySQL 4.0.0 or later for this
// to work. If you are using an older version of MySQL then disable this
// feature.
define('COUNT_ROWS', TRUE);

// DEFAULTLOGTABLE is the name of the table where new syslog entries are
// stored.
define('DEFAULTLOGTABLE', 'logs');

// MERGELOGTABLE is the name of the merge table. This feature should
// generally only be used in combination with the logrotate script.
// If it is enabled then the logrotate script will create a merge table
// of all the log tables. Be aware that you need to be a little careful
// when using merge tables so it is recommended that you only use it with
// the logrotate script. The merge table will also be the one used to
// populate the HOSTS and FACILITY fields on the search form.
define('MERGELOGTABLE', 'all_logs');

// If LOGROTATERETENTION is defined then it determines how many days logs
// are kept. Old log tables with a datestamp older than LOGROTATERETENTION
// days will be dropped.
define('LOGROTATERETENTION', 90);
//========================================================================
// END: MISC FUNCTIONALITY
//========================================================================

//========================================================================
// BEGIN: DATABASE CONNECTION INFO
//========================================================================
// DBUSER is the name of the basic user.
define('DBUSER', 'root');

// DBUSERPW is DBUSER's database password.
define('DBUSERPW', 'password');

// DBADMIN is the name of the admin user.
define('DBADMIN', 'root');

// DBADMINPW is DBADMIN's database password.
define('DBADMINPW', 'password');

// DBNAME is the name of the database you are using.
define('DBNAME', 'syslog');

// DBHOST is the host where the MySQL server is running.
define('DBHOST', 'localhost');

// DBPORT is the port where the MySQL server is listening.
// The default port is 3306.
define('DBPORT', '3306');
//========================================================================
// END: DATABASE CONNECTION INFO
//========================================================================

//========================================================================
// BEGIN: AUTHENTICATION
//========================================================================
define('REQUIRE_AUTH', TRUE);
define('AUTHTABLENAME', 'users');

// Authentication has two modes:
// 1) You renew the session on every page view. This means you can have a
//    tail screen running and the session will never expire as long as you
//    refresh the screen before SESSION_EXP_TIME. This is the default.
// 2) The session is timed from the time you login. The session is only
//    refreshed on login.
define('RENEW_SESSION_ON_EACH_PAGE', TRUE);

// SESSION_EXP_TIME is seconds until the session expires.
define('SESSION_EXP_TIME', '3600');

// Set the URL to php-syslog-ng. If you don't then the login screen will
// not be able to redirect users automatically after a successful login.
define('URL', 'http://172.20.1.7/phpsyslog/');
//========================================================================
// END: AUTHENTICATION
//========================================================================

//========================================================================
// BEGIN: ACCESS CONTROL
//========================================================================
// Access Control Lists allows you to specify what individual users have
// access to. Access Control requires Authentication to have any effect.
// Currently only the Configure screen uses this options.
define('USE_ACL', TRUE);
define('USER_ACCESS_TABLE', 'user_access');
define('ACTION_TABLE', 'actions');
//========================================================================
// BEGIN: ACCESS CONTROL
//========================================================================

//========================================================================
// BEGIN: SEARCH CACHE
//========================================================================
// Enabling the search cache will create a small table with the values
// needed to fill in the HOSTS and FACILITY fields on the search page.
// The cache table has to filled/updated by either clicking the refresh
// cache option or periodically running the updateCache.php script (from
// cron).
// If you use the MERGELOGTABLE then the cache will be updated using that
// table. If you do not use MERGELOGTABLE then the cache is updated for
// each table with log data.
define('USE_CACHE', TRUE);
define('CACHETABLENAME', 'search_cache');
//========================================================================
// END: SEARCH CACHE
//========================================================================

//========================================================================
// BEGIN: REG EXP ARRAY USED FOR INPUT VALIDATION
//========================================================================
$regExpArray = array(
        "username"=>"(^w{4,}$)",
        "password"=>"(^.{4,}$)",
        "pageId"=>"(^w+$)",
        "sessionId"=>"(^w{32}$)",
        "date"=>"/^yesterday$|^today$|^now$|^(d){4}-([01]*d)-([0123]*d)$/i",
        "time"=>"/^now$|^([012]*d):([012345]*d):([012345]*d)$/i",
        "limit"=>"(^d+$)",
        "orderby"=>"/^seq$|^host$|^facility$|^priority$|^datetime$/i",
        "order"=>"/^asc$|^desc$/i",
        "offset"=>"(^d+$)",
        "collapse"=>"/^1$/",
        "table"=>"(^w+$)",
        "excludeX"=>"(^[01]$)",
        "host"=>"(^[w-.]+$)",
        "facility"=>"(^w+$)",
        "priority"=>"/^debug$|^info$|^notice$|^warning$|^err$|^crit$|^alert$|^emerg$/i",
);
//========================================================================
// END: REG EXP ARRAY USED FOR INPUT VALIDATION
//========================================================================
?>

Now it's time to login to ASDM to setup and define syslog server and rate limiting. Important Note: If you don't set a rate limit for notices on your Pix, your syslog-ng/mysql WILL get pounded!

Let's start off in Configuration > Properties > Logging > Logging Setup

Check the check boxes for "enable sysloging" and "send logs in EMBLEM format. EMBLEM format is compatible with standard unix style syslog servers and syslog-ng.

Now move down to "Configuration > Properties > Logging > Logging Filters" Click on the Syslog Servers item and choose edit

In syslog from all event classes, click the radio button for filter on severity and select informational from the list

Next move to "Configuration > Properties > Logging > Rate Limit"

in the section "Rate Limits for Syslog Logging Levels" click on informational and choose edit

I'm being fairly conservative, sending 100 Informational messages every five seconds. Next, move to "Configuration > Properties > Logging > Syslog Servers" Choose ADD.

Choose the approprate firewall interface and enter the IP Address of your syslog server, select udp/514 then click OK. Now move to "Configuration > Properties > Logging > Syslog Setup" choose "Local7" as the facility code

Click on the Advanced button to make changes to Device ID if desired

Now type the following in a terminal window on your syslog server:

/etc/init.d/mysqld restart

/etc/init.d/httpd restart

/sbin/syslog-ng &

/scripts/syslog2mysql.sh &

Make sure that the last two are in /etc/rc.local

rc.local

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/sbin/syslog-ng
/bin/bash /scripts/syslog2mysql.sh

Now launch your browser and type in the URL to php-syslog-ng, you should see something similar to the following:

Doc under construction!

mini HowTo on getting a RANCID implementation on it feet

Fri, 17 Aug 2007 11:33:57 -0500

RANCID - Really Awesome New Cisco confIg Differ

This is a mini HowTo on getting a RANCID implementation on it feet, for complete documentation please visit the official website for RANCID found @ http://www.shrubbery.net/rancid/

Download the package at ftp://ftp.shrubbery.net/pub/rancid/rancid-2.3.1.tar.gz

You will want to run this on BSD or some flavor of Linux.

Dependencies:

Expect needs to be installed, try “yum -y install expect”

rancid will also need to have the following packages:

cvs            Code revision system available from prep.ai.mit.edu:/pub/gnu
gnudiff        gnudiff provides the uni-diff (-u) option. If you do not have
                 a diff that supports -u, configure will set-up rancid to use
                 'diff -c' or 'diff -C'.
perl5           perl version 5 or greater available from www.cpan.org
expect        http://expect.nist.gov/ We highly suggest that you stick to
                 expect 5.24.1 (or so). This seems to work best. Note that
                 you need to have the accompanying tcl &/ tk.
svn            Code revision system, an alternative to cvs. Available from
                 http://subversion.tigris.org/tarballs/. Use the configure
                 option --enable-svn to configure for Subversion.
tcl             Required by expect.

Lets get started!

Drop the gzipped tarball into /opt and untar it using “tar zxvf rancid-2.3.1.tar.gz”

Cd to the directory where rancid was extracted and read the install and readme files.

Then execute:

[root@mybox rancid-2.3.x]# ./configure --prefix=/usr/local/rancid/

Then type make

Then make install

I’m just running this as root – I know, shame on me but I just wanted it up fast!

Edit the file /usr/local/rancid/etc/rancid.conf

(I am using SVN for the repo so I changed "RCSSYS=svn; export RCSSYS")

here is what mine looks like

# rancid 2.3.2a6
# This file sets up the environment used for rancid. see rancid.conf(5)
#
# This will be site specific
#
TERM=network;export TERM
#
# Create files w/o world read/write/exec permissions, but read/exec permissions
# for group.
umask 027
#
# Under BASEDIR (i.e.: --localstatedir), there will be a "logs" directory for
# the logs from rancid and a directory for each group of routers defined in
# LIST_OF_GROUPS (below). In addition to these, there will be a "CVS"
# directory which is the cvs (or Subversion) repository.
#
# Use a full path (no sym-links) for BASEDIR.
#
TMPDIR=/tmp; export TMPDIR
# Be careful changing this, it affects CVSROOT below.
BASEDIR=/usr/local/rancid/var; export BASEDIR
PATH=/usr/local/rancid/bin:/usr/bin:/usr/sbin:.:/bin:/usr/kerberos/bin:/usr/local/bin:/usr/bin; export PATH
# Location of the CVS/SVN repository. Be careful changing this.
CVSROOT=$BASEDIR/CVS; export CVSROOT
# Location of log files produced by rancid-run(1).
LOGDIR=$BASEDIR/logs; export LOGDIR
#
# Select which RCS system to use, "cvs" (default) or "svn". Do not change
# this after CVSROOT has been created with rancid-cvs. Changing between these
# requires manual conversions.
#RCSSYS=cvs; export RCSSYS
RCSSYS=svn; export RCSSYS
#
# if NOPIPE is set, temp files will be used instead of a cmd pipe during
# collection from the router(s).
#NOPIPE=YES; export NOPIPE
#
# FILTER_PWDS determines which passwords are filtered from configs by the
# value set (NO | YES | ALL). see rancid.conf(5).
#FILTER_PWDS=YES; export FILTER_PWDS
#
# if NOCOMMSTR is set, snmp community strings will be stripped from the configs
#NOCOMMSTR=YES; export NOCOMMSTR
#
# How many times failed collections are retried (for each run) before
# giving up. Minimum: 1
#MAX_ROUNDS=4; export MAX_ROUNDS
#
# How many hours should pass before complaining about routers that
# can not be reached. The value should be greater than the number
# of hours between your rancid-run cron job.
OLDTIME=4; export OLDTIME
#
# How many hours should pass before complaining that a group's collection
# (the age of it's lock file) is hung.
#LOCKTIME=4; export LOCKTIME
#
# The number of devices to collect simultaneously.
#PAR_COUNT=5; export PAR_COUNT
#
# list of rancid groups
LIST_OF_GROUPS="ISI_Prod_Routers ISI_Prod_Switchgear ISI_Prod_Firewalls ISI_ATM_Backbone ISI_Lab_Routers ISI_Lab_Switchgear ISI_Lab_Firewalls"
# more groups...
#LIST_OF_GROUPS="$LIST_OF_GROUPS noc billybobisp"
#
# For each group, define a list of people to receive the diffs.
# in sendmail's /etc/aliases.
# rancid-group: notoriousBG@isiisi.com
# rancid-admin-group: notoriousBG@isiisi.com
# be sure to read ../README regarding aliases.
#
# If your MTA configuration is broken or you want mail to be forwarded to a
# domain not the same the local one, define that domain here. "@" must be
# included, as this is simply appended to the usual recipients. It is NOT
# appended to recipients specified in rancid-run's -m option.
# MAILDOMAIN="@isiisi.com"; export MAILDOMAIN
#
# By default, rancid mail is marked with precedence "bulk". This may be
# changed by setting the MAILHEADERS variable; for example no header by setting
# it to "" or adding X- style headers. Individual headers must be separated
# by a \n.
MAILHEADERS=""; export MAILHEADERS

Run the rancid-cvs command to create the /usr/local/var/networking directory and its database and network device list files.

Creat the file "/root/.cloginrc"

Example:

[root@blzsfed60bg ~]# cat .cloginrc
#Routers
add password * consolepass enablepass

Now, there is a file called Router.db located in each group directory, for me that location is:

/usr/local/rancid/var

My groups are:

ISI_ATM_Backbone
ISI_Lab_Firewalls
ISI_Lab_Routers
ISI_Lab_Switchgear
ISI_Prod_Firewalls
ISI_Prod_Routers
ISI_Prod_Switchgear

You can call them whatever you want - that was done in the rancid.conf.

Below is a sample of one of my Router.db files

[root@blzsfed60bg ISI_Lab_Routers]# cat router.db
10.10.107.1:cisco:up
10.10.107.2:cisco:up
172.30.100.26:cisco:up
[root@blzsfed60bg ISI_Lab_Routers]#

Here is my crontab for the root user, you can see that rancid is schedule to run every hour.

[root@blzsfed60bg etc]# crontab -l
*/1 * * * * /scripts/cacti_poller.sh > /dev/null 2>&1
1 * * * * /usr/local/rancid/bin/rancid-run > /dev/null 2>&1
50 23 * * * /usr/bin/find /usr/local/rancid/var/logs -type f -mtime +2 -exec rm {} \;
[root@blzsfed60bg etc]#

make sure you start SVN by issuing the command "/usr/bin/svnserve -d"

Download Tortoise SVN client and install it on the computer you will access the repo from.

To access your REPO - right click on your desktop, you should see the following:

Click on Repo-browser an you see the following:

Type in the path to your REPO and click OK, you should see something similar to the following:

(note - I have right clicked on 7.7.7.3 and selected "Log Messages" in the screenshot below)

Right click on the Modified item and choose "Show didderences" You well see a diff such as the following:

There you have it!

Disclamer - If you want detailed instruction on setting up rancid, you should go to:

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch1_:_Network_Backups_With_Rancid

Peter Harrison has done a great job of putting this information together and his howto is great.

Also, I have some troubleshooting notes @ forums.routermonkey.org regarding the mail function of Rancid.