Apache access log parser

[with reverse DNS check and colors]

Nothing special here really. Just a few lines of code to make the logs review a little bit easier.

Displayed columns in order from left to right:

  • Date and time of access
  • HTTP CODE of response [200 in green, 404 in blue, rest in red]
  • IP address
  • Reverse DNS hostname [last 30 chars] [empty if NXDOMAIN]
  • Request [first 30 chars]

 


The output:

apache log parsing output


Script:


#!/bin/bash

while read line
do
        # IP
          ip=$(echo $line | cut -d " " -f 1)

        # HOST
          host=$(host $ip | cut -d " " -f 5 | tail -1)
          if [[ ${#ip} -lt 15 ]]; then
                for (( i=$(echo "15-${#ip}"|bc); i>0; i-- )) do
                        ip="$ip "
                done
          fi

        # IF I DO NOT GET DOMAIN NAME
          if [[ $(echo "$host" | grep "NXDOMAIN" | wc -l ) -ne 0 ]]; then
                host=" - "
          fi

        # EVEN UP THE HOSTNAME TO SEE LAST 30 CHARS
          if [[ ${#host} -lt 30 ]]; then
                for (( i=$(echo "30-${#host}"|bc); i>0; i-- )) do
                        host="$host "
                done
          else
                host=${host:$(echo "${#host}-30"|bc)}
          fi

          dhost="\033[01;30m$host\033[00m"

        #   DISPLAY GOOGLEBOT CUSTOM DNS
          if [[ $(echo $host | grep google |wc -l) -eq 1 ]]; then
                dhost="\033[01;30mGOOGLEBOT\033[00m                     "
          fi

        # DATE
          date=$(echo $line | cut -d "[" -f 2 | cut -d "]" -f 1 | cut -d "+" -f 1)
                day=$(echo $date | cut -d ":" -f 1 | tr -d " ")
                dtime=$(echo $date | cut -d ":" -f 2- | tr -d " ")

        # REQUEST
          req=$(echo $line | cut -d "]" -f 2 | cut -d "\"" -f 2 | cut -d " " -f -2)
        # CUT REQUEST TO 30 CHARS
          dreq=${req:0:30}
        # CUSTOM REQUEST INFO IN CASE OF ADMIN PANEL
          if [[ $(echo $req | grep "admin.php" | wc -l) -eq 1 ]]; then
                dreq="\033[01;31mFAV\033[00m"
          fi

        # HTTP CODE
          code=$(echo $line | cut -d "\"" -f 3 | cut -d " " -f 2)
          hcode="\033[01;31m$code\033[00m";
          if [[ "$code" -eq "200" ]]; then
                hcode="\033[01;32m$code\033[00m";
          fi
          if [[ "$code" -eq "404" ]]; then
                hcode="\033[01;34m$code\033[00m";
          fi

        # DISPLAY
          # I DONT WANT TO DISPLAY FAVICON REQUESTS
          if [[ $(echo $req | grep "favicon.ico" | wc -l) -eq 1 ]]; then
                echo -n ""
          else
                echo -e "$day $dtime $hcode $ip $dhost $dreq"
          fi
done < /var/log/apache2/access.log

Obrazek

Advertisements

outlook rules with powershell script (moving emails to PST file)

Have you ever reached the size limit of the rules that you can setup in Microsoft Outlook? Well, I did. By default it is 32 Kb which can be extended to 256 Kb. Still, if you receive couple of hundreds emails per day – like myself – maybe you will appreciate the fact that you don’t have to be limited to mentioned thresholds. There is a fairly simple solution and it’s name is – Powershell.
Below you will find a powershell script with a couple of example rules and a bonus function for displaying balloon tooltips. It is designed to work with PST file but if you have large mailbox quota you don’t have to move emails outside of your exchange mailbox.


Script is based on function display:

function display( [string]$subject, [string]$color )  {
  $len = 20
  if ( $subject.length -lt 20 ){
    $toadd=20-$subject.length;
    for ( $i=0; $i -lt $toadd; $i++ )
    {
      $subject=$subject+" ";
    }
    $len = $subject.length
  }
  else { $len = 20 }

  $index=$index+1
  Write-host -ForegroundColor $color -nonewline " |" ((($subject).ToString()).Substring(0,$len)).ToUpper()
}

It takes two parameters:

  • subject – the Email subject
  • color – the color that should be used to display particular email subject

Here is how it looks during execution (sorry for cuts and blurs – censorship was necessary):

Powershell moving outlook emails


Script:

# VARIABLES #

 # CREATE OUTLOOK INSTANCE
        $outlook = New-Object -comobject outlook.application
        $namespace = $outlook.GetNameSpace("MAPI")

 # PATH TO PST FILE
        $pstPath = "D:\PST\ALERT.pst"

 # ACCESS THE PST FILE AND MAP DESIRED FOLDERS
        $pst = $namespace.Stores | ?{$_.FilePath -eq $pstPath}
        $pstRoot = $pst.GetRootFolder()
        $pstFolders = $pstRoot.Folders
        $nagiosFolder = $pstFolders.Item("NAGIOS")

 # DEFAULT FOLDER 6 IS INBOX, 3 IS DELETED ITEMS
        $DefaultFolder = $namespace.GetDefaultFolder(6)
        $InboxFolders = $DefaultFolder.Folders
        $DeletedItems = $namespace.GetDefaultFolder(3)

        $Emails = $DefaultFolder

# BALLOON FUNCTION

function balloon([string]$text, [string]$title)
{
    if ($objBalloon)
    {
      # DELETE EXISTING BALLOON
      $objBalloon.Dispose()
    }

    [void] [System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
    $objBalloon = New-Object System.Windows.Forms.NotifyIcon
    $objBalloon.Icon = "C:\Windows\ServicePackFiles\i386\msnms.ico"

     # INFO, WARNING AND ERROR VALUES ARE ALLOWED
      $objBalloon.BalloonTipIcon = "Error"
      $objBalloon.BalloonTipTitle = "$title"
      $objBalloon.BalloonTipText = "$text"
      $objBalloon.Visible = $True

     # HOW LONG TO SHOW THE BALLOON
      $objBalloon.ShowBalloonTip(5000)
}

# DISPLAY FUNCTION

function display( [string]$subject, [string]$color )  {
  $len = 20
  if ( $subject.length -lt 20 )
  {
    $toadd=20-$subject.length;
    for ( $i=0; $i -lt $toadd; $i++ ){
      $subject=$subject+" ";
    }
    $len = $subject.length
  }
  else { $len = 20 }
  $index=$index+1
  Write-host -ForegroundColor $color -nonewline " |" ((($subject).ToString()).Substring(0,$len)).ToUpper()
}

# PROCESSING EMAILS

foreach ($Email in $Emails) {
  $index=$index+1
        # FOUR SUBJECTS PER LINE
  if ( ($index % 4) -eq 0 ) {  Write-host -nonewline -ForegroundColor DarkGray " |`n" }

  if ($Email.To -eq "Surname, Name" -and $Email.Subject -notmatch "SPAM") {
      $Email.Move($nagiosFolder) | out-null
      display  ([string]$Email.Subject ) ([string]"Cyan")
      continue
  }
  if ($Email.Subject -match "SiteScope Alert, error") {
      $Email.Move($pstSubFolder.Item("SITESCOPE")) | out-null
      display  ([string]$Email.Subject ) ([string]"Yellow")
      continue
  }
    # DEFAULT BEHAVIOR IF NO MATCH IN RULES ABOVE
    display  ([string]$Email.Subject ) ([string]"DarkGray")
}

Obrazek

Generating html file from directory list with powershell

There was a day when I was searching for some files inside a directory which had multiple sub-directories. Of course there is a Windows search tool that can help you to some extent but – it will not allow you to view multiple directories at the same time while highlighting files of particular kind – this is what I wanted.
Below there is a powershell script that does exactly that listing directory content and creating a html file to let you see and (in some cases) easily open files.

Each file is displayed as a tile like this one:

Default file icon

On each tile there is a filename, file size and date of last modification.

The web page generated will look like this:

html result of directory view

Tiles are grouped as they appear in directories. They have different colors – based on their extensions. In my case the black ones are txt files, yellow ones are html/htm files, violet ones are pdfs and pictures of common types are purple. The rest are displayed as gray.
The files that have colors other than gray can be opened in the browser by clicking on a tile. Most browsers do not support opening files like docx and others just by clicking on the url pointing at particular file. And the reason for this is that browser do not know a protocol that could handle such file. Thankfully you can create your own protocol but i will show it some other time.

For now you can open txt, cpp, c, hpp, h, gif, tiff, jpg, jpeg, png, bmp, pdf, html, htm files from the web page generated by the script.
File size is displayed in either Bytes, KiloBytes or MegaBytes – if the size is bigger than 1024 Bytes it will be shown in KiloBytes.
You can easily modify the script to add your coloring preference just by creating or altering a CSS style.


The script:

$global:current_dir="";
$global:open=0;
$root=($(pwd).Path).Length;

$out = "
<html>
  <head>
  <style>
    body {
      padding: 0px;
      margin: 0px;
      background-color: rgb(30,30,30);
      background-image: -ms-linear-gradient(top left, #3F0F63 0%, #930AEF 100%);
      background-image: -moz-linear-gradient(top left, #3F0F63 0%, #930AEF 100%);
      background-image: -o-linear-gradient(top left, #3F0F63 0%, #930AEF 100%);
      background-image: -webkit-gradient(linear, left top, right bottom, color-stop(0, #3F0F63), color-stop(1, #930AEF));
      background-image: -webkit-linear-gradient(top left, #3F0F63 0%, #930AEF 100%);
      background-image: linear-gradient(to bottom right, #3F0F63 0%, #930AEF 100%);
    }
    div.directory {
      border: 2px solid rgb(155,155,155);
      border-bottom: 0px;
      padding: 5px;
      overflow:auto;
    }
    div.dirtitle {
      font-family: Verdana;
      font-size: 11px;
      color: rgb(140,140,140);
      text-align:left;
      padding: 3px;
      margin: 3px;
      width: 95%;
    }
    div.file {
      border: 2px solid rgb(30,30,30);
      width: 80px;
      height: 80px;
      float: left;
      position: relative;
      padding: 3px;
      text-align:center;
      color: white;
    }
    span.filename {
      font-family: Verdana;
      font-size: 12px;
      line-height: 40px;
    }
    span.filesize {
      font-family: Verdana;
      font-size: 10px;
    }
    span.mod_date {
      font-family: Verdana;
      font-size: 10px;
    }
    .word {
      background-color: rgb(82,82,82);
    }
    div.file:hover {
        border: 2px solid rgb(72,72,72);
    }
    .orange {
      background-color: rgb(218, 83, 44);
    }
    .green {
      background-color: rgb(0, 163, 0);
    }
    .purple {
      background-color: rgb(185, 29, 71);
    }
    .blue {
      background-color: rgb(45, 137, 239);
    }
    .yellow {
        background-color: rgb(255, 196, 13);
    }
    .violet {
      background-color: rgb(159, 0, 167);
    }
    .black {
      color: white;
      background-color: rgb(0, 0, 0);
    }
    .gray {
      background-color: rgb(100, 100, 100);
    }
    .red {
      background-color: rgb(224, 25, 61);
    }
  </style>
</head>
<body>";

Write-Output $out >> test.html ;

Get-ChildItem -recurse | where { $_.PSIsContainer -eq $false } |% {
 # NO TILES FOR DIRECTORIES
 if ( [string]$global:current_dir -ne [string]$_.Directory ){
  if ( $global:open -eq 0 )
  {
   $global:open=1;
   $out= "<div id='directory' class='directory'> <div id='dirtitle' class='dirtitle'>" + $_.Directory + "</div>";
   Write-Output $out >> test.html ;
  }
  else
  {
   $out = " </div><div id='directory' class='directory'> <div id='dirtitle' class='dirtitle'>" + $_.Directory + "</div> ";
   Write-Output $out >> test.html;
  }
  $global:current_dir=$_.Directory;
 }

 # CORRECTING PATH
 $relPath = $_.FullName.Remove(0, $root + 1)
 $relPath = $relPath -replace "\\", "/"

 # HTML
 if ( $_.Extension -eq ".html" -or $_.Extension -eq ".htm")
 {
  $out= "<div class='file yellow' onclick='window.location=`"" + ($relPath).trim() + "`"'>";
  Write-Output $out >> test.html ;
 }
 # TEXT FILES
 elseif ( $_.Extension -eq ".txt" -or $_.Extension -eq ".cpp" -or $_.Extension -eq ".c" -or $_.Extension -eq ".h" -or $_.Extension -eq ".hpp")
 {
  $out= "<div class='file black' onclick='window.location=`"" + $relPath + "`"'>";
  Write-Output $out >> test.html ;
 }
 # PDF
 elseif ( $_.Extension -eq ".pdf" )
 {
  $out= "<div class='file violet' onclick='window.location=`"" + $relPath + "`"'>";
  Write-Output $out >> test.html ;
 }
 # PICTURES
 elseif ( $_.Extension -eq ".jpg" -or $_.Extension -eq ".gif" -or $_.Extension -eq ".png" or $_.Extension -eq ".jpeg" -or $_.Extension -eq ".bmp" -or $_.Extension -eq ".tiff" )
 {
  $out= "<div class='file purple' onclick='window.location=`"" + $relPath + "`"'>";
  Write-Output $out >> test.html ;
 }
 else
 {
  $out = "<div class='file gray' >";
  Write-Output $out >> test.html
 }

 # REMOVING HYPHEN FROM FILENAMES
 # SINCE IT CAUSES MULTIPLE LINES
 # TO BE GENERATED PER FILENAME
 $sanit= $_.BaseName -replace "-", "_"

 # SHOWS ONLY FIRST 9 CHARACTERS FROM FILE NAME
 # TO AVOID OVERFLOW
 if ( ($sanit).Length -gt 9 )
 {
  $out= "`t<span id='filename' class='filename'>" + ($sanit).Substring(0,9) + "</span>";
 }
 else
 {
  $out= "`t<span id='filename' class='filename'>" + $sanit + "</span>";
 }
 Write-Output $out >> test.html;

 $var=$_.Length;

 # SIZE BIGGER THAN MB
 if ( $var -gt 1048576 )
 {
  $var=$var/1048576;
  $var = "{0:N2}" -f $var
  $out = "`t<span id='filesize' class='filesize'>" + ([string]$var).trim() + " MB</span>";
  Write-Output $out >> test.html;
 }
 else
 {
  # SIZE BIGGER THAN KB
  if ( $var -gt 1024 )
  {
   $var=$var/1024;
   $var = "{0:N2}" -f $var
   $out = "`t<span id='filesize' class='filesize'>" + ([string]$var).trim() + " KB</span>";
   Write-Output $out >> test.html;
  }
  else
  {
   $out = "`t<span id='filesize' class='filesize'>" + ([string]$var).trim() + " B</span>";
   Write-Output $out >> test.html;
  }
 }
 $out = "`t<span id='mod_date' class='mod_date'>" + $_.LastWriteTime + "<span>"
 Write-Output $out >> test.html ;
 Write-Output "</div>" >> test.html;
 Write-Output "`n" >> test.html;
}
Write-Output "</div></body></html>" >> test.html;

Obrazek

cpu usage graph in console


License and source code is available at google code.


Just a simple script that displays processor usage graph in command line.

This is how it will look like when running:
CPU graph in console. …or longer version from more real-like usage:

CPU graph with more real like usage

There are two thresholds defined – above 40 % will be shown in yellow and above 60 % will be red.


Requirements (the ones that i can think of):

  • sleep (sleep for a fraction of second is not supported by older versions of sleep)

  • mpstat

Example function used to draw in console:

function green()
  {
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;32m|\033[0m"
        echo -en "\033[u"
  }

33[s saves the current position of the cursor and 33[u restores it to saved position.
tput cup places the cursor in specific row and column.


Full script:

#!/bin/bash
 
# VARIABLES
  columns=40
 
# FUNCTIONS #
function draw()
{
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;3$3m$4\033[0m"
        echo -en "\033[u"
}
function wipe()
{
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;31m \033[0m"
        echo -en "\033[u"
}
#############
 
# HIDE CURSOR AND CLEAR SCREEN #
  tput civis & clear
 
while true; do
 
# COLLECT UPDATE
#
 
        idle=`mpstat 1 1 | grep "Average" | tail -1 | sed 's/ \+/ /g' | cut -d " " -f 11 | tr -d"\n"`;
 
        usage=`echo "scale=0;(100-$idle)/10" |bc`;
 
        # BACKUP PREVIOUS DATA #
        # CUT THE OLDEST LINE  #
        cat cpu.log | tail -$columns >> temp.tmp
 
        # UPDATE STATS #
        if [[ $usage -eq 0 ]];
        then
                # IF USAGE LESS THEN 20% DRAW SINGLE BAR #
                echo "1" >> temp.tmp
        else
                echo $usage >> temp.tmp
        fi
 
        # UPDATE LOG #
        cat temp.tmp > cpu.log
        rm temp.tmp
 
        # DRAW GRAPH #
 
        var=6;
        # BEGIN FROM COLUMN 1 #
        j=1
 
        while read usage
        do
                # DRAW USAGE #
                for i in `seq 1 $usage`
                do
                        top=`echo "11-$usage" | bc`
                        var=`echo "11-$i"     | bc`
 
                        if [[ $usage -gt 3 ]]; then
                                if [[ $usage -gt 6 ]]; then
                                        if [[ $var -eq $top ]];then
                                                draw    $var $j "1" "+"
                                        else
                                                draw    $var $j "1" "|"
                                        fi
                                else
                                        if [[ $var -eq $top ]];then
                                                draw    $var $j "3" "+"
                                        else
                                                draw    $var $j "3" "|"
                                        fi
                                fi
                        else
                                if [[ $var -eq $top ]]; then
                                        draw     $var $j "2" "+"
                                else
                                        draw     $var $j "2" "|"
                                fi
                        fi
                done
 
                # WIPE PREVIOUS BAR REMAINNING IF THEY EXIST #
                usage=`echo "$usage+1" | bc`
 
                for k in `seq $usage 11`;
                do
                        var=`echo "11-$k" | bc`
                        wipe $var $j
                done
 
                # PROCEED TO NEXT COLUMN #
                j=`echo "$j+1" | bc`
 
                # ADD LATENCY IF NEEDED #
                sleep 0.1
 
        done < "cpu.log"
        tput cup 11 0
#
done