User Tools

  • Logged in as: anonymous (anonymous)
  • Log Out

Site Tools


mantisbt:irc_logs

IRC logs archive setup

Thanks to mantisbot - MantisBT's IRC bot, we keep an archive of all activity on the official MantisBT channels.

This data is available on the IRC Logs page

Initial setup

mantisbt.org pages

A new page was created, and the following ones modified, as well as the CSS:

See mantisbt-web repository for details.

Converting from the old setup

  • Create a new directory for the logs
    mkdir /srv/www/irclogs
    cd !$
  • Copy the log files from jreese's old setup to the new location
    f=$(mktemp)
    cd /home/jreese/irclogs
    find -name 'm*.log' |xargs tar rf $f
    cd /home/supybot/mantisbot/logs/ChannelLogger
    tar xf $f
    chmod -R supybot:supybot m*
    rm $f
  • Reorganize the log files into a year-based archive. To minimize risk of losing log data, this should be done when there is no traffic in the channels
    Save the following code as a script (e.g. /tmp/convert.sh) and run it:
    #!/bin/bash
    
    backup=$(mktemp)
    
    find -maxdepth 1 -type d |while read dir
    do
            if [[ $dir == '.' ]]
            then
                    continue
            fi
            channel=$(basename $dir)
            echo "Processing $dir"
    
            # Backup recent log files
            find $channel -name "*$(date +'%Y')*" |sort |tail -n 2 |xargs tar rf $backup
    
            cd $channel
    
            years=$(find -name '*.log' -maxdepth 1 |cut -d_ -f2 |cut -d- -f1 |sort -u)
            for year in $years
            do
                    echo "  $year ($(ls *$year*.log|wc -l) files to move)"
                    mkdir -p $year
    
                    # Rename files while moving (jreese has a '_' between channel name and
                    # date, mantisbot-generated files have a '.')
                    find -name "mantis*_$year*.log" |sed -r "s/^(.*)_(.*)$/\0 $year\/\1.\2/" |xargs -n 2 mv
            done
    
            cd - >/dev/null
            echo
    done
    
    # Check for differences in today's log files
    restore=$(mktemp -d)
    logdir=$PWD
    cd $restore
    tar xf $backup
    find -name "mantis*_*.log" |sed -r "s/^(.*)_(.*)$/\0 \1.\2/" |xargs -n 2 mv
    # Find differences and apply them as a patch
    find -type f |xargs -n 1 -I FILE diff -u $logdir/FILE FILE |patch -d$logdir -p1
    cd - >/dev/null
    rm -rf $restore
    rm $backup

TODO check the dates in the logs - jreese has [hh:mm:ss] (on PST apparently), mantisbot has yyyy-mm-ddThh:mm:ss (on server time, EST)

– import completed on 02-May-2013 19:53 (EST, server time)

Generate the HTML pages

  • Update scripts repository
    cd /srv/mantisbt-tools
    git pull
  • make sure the irclogs.py script is executable
  • Run the script
    ./irclogs.py

Setup the cron job

  • Edit the cron job
    vi /etc/cron.d/mantis-irclogs
  • Type in the following text
    PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
    
    # Generate IRC log html pages every 5 minutes
    */5 * * * * root /srv/mantisbt-tools/irclogs.py 2>&1 |cat >>/var/log/irclogs.log
  • Save the file

To test, generate some activity on one of the channels, wait a few minutes and check that the

  • Edit the log rotation config file
    vi /etc/logrotate.d/mantis
  • Add the following section (daily rotation, keep 5 days)
    # IRC logs
    /var/log/irclogs.log {
        daily
        rotate 5
        compress
        missingok
        nocreate
    }

Web server setup

  • Setup redirection from /irclogs to the new irclogs.php page
    • Go to the logs directory
      cd /home/supybot/mantisbot/logs/ChannelLogger
    • Create a redirection page
      vi index.php
    • Paste the following contents and save the file
      <?php
              header('HTTP/1.1 301 Moved Permanently');
              header('Location: http://mantisbt.org/irclogs.php');
              header('Content-Type: text/html');
      ?>
      
      <!DOCTYPE HTML>
      <html lang="en-US">
              <head>
                       <meta charset="UTF-8">
                       <meta http-equiv="refresh" content="1;url=/irclogs.php">
                       <script type="text/javascript">
                           window.location.href = "/irclogs.php
                       </script>
                       <title>Page Redirection</title>
              </head>
              <body>
                      This page has moved.
                      If you are not redirected automatically, follow the <a href='/irclogs.php'>link</a>.
              </body>
      </html>
      
  • Create symbolic links to redirect URLs of primary legacy bookmarks
    ln -s index.php latest.log.html
    find -maxdepth 1 -name "m*" -exec sh -c "ln -svf ../index.php {}; ln -svf index.php {}/latest.log.html" \;
  • Update the Apache config file
    • Edit the file
      vi /etc/apache2/sites-enabled/mantisbt
    • Change references to irclogs directory
      %s:home/jreese:srv/www
    • Save the file and exit
    • Reload apache configuration
      service apache2 reload

Cleanup

When everything is confirmed to be working as expected

  • Inform jreese so he stops his cron job
  • Remove the old irclogs directory /home/jreese/irclogs

Operations

The channel logs are saved on an ongoing basis by *mantisbot*.

The HTML pages are generated from these logs and then posted on the web site by a Python script which is scheduled to run every 5 minutes via a cron job [TODO]

There should be no need for any manual intervention.

Known Issues / TODO

  • FIXED: new log directories' permissions at the turn of a year 17994
mantisbt/irc_logs.txt · Last modified: 2015/02/13 18:01 by dregad