console

[Another] Apache access log parser in Python

I must admit – Python is mighty cool. I’ve started learning it yesterday and today I’ve managed to create a simple parser of Apache2 access log – and with colors!

Nothing fancy so i’m just going to drop it here:

 

#!/usr/bin/env python3.4

# IMPORTS
import re
import socket
from colorama import init, Fore, Back, Style

# VARS
regex = '([(\d\.)]+) - - \[(.*?)\] "(.*?)" (\d+) (\d+) "(.*?)" "(.*?)"'
log_path = '/var/log/apache2/access.log'
previous_ip = " "
previous_host = " "

# FUNCTIONS
def adjust_item( str, i ):
	if len(str) < i:
		for j in range(i-len(str)):
			str = str + " "
	return str

def get_hostname( ip ):
	global previous_ip
	global previous_host
	if previous_ip == ip:
		return previous_host
	else:
		try:
			new_host = socket.gethostbyaddr(ip)
			previous_ip = ip
			previous_host = new_host[0]
			return new_host[0]
		except Exception:
			new_host = ip
			previous_ip = ip
			previous_host = ip
			return new_host

# READ FILE
with open(log_path, 'rt') as log:
	text = log.read();

# FOR EACH LINE
for line in text.split("\n"):
	if len(line) > 5:

		# PARSE LINE AND ADJUST FIELD LENGTH
		ip 		= adjust_item( re.match( regex, line ).group( 1 ), 15 )
		hostname 	= adjust_item( str(get_hostname(ip.strip())), 30 )
		date		= (re.match( regex, line ).group( 2 )).split(" ")[0]
		request 	= adjust_item( re.match( regex, line ).group( 3 ), 40 )
		code		= adjust_item( re.match( regex, line ).group( 4 ), 4 )
		size		= adjust_item( re.match( regex, line ).group( 5 ), 8 )
		ref		= adjust_item( re.match( regex, line ).group( 6 ), 30 )
		agent		= adjust_item( re.match( regex, line ).group( 7 ), 3 )

		# HTTP 200 OK
		if code.strip()[0] == "2":

			print( date + " " , end="")
			print( Fore.GREEN + Style.BRIGHT + code[:4] + Fore.RESET + Style.NORMAL, end="" )
			print( ip[:15] + " " , end="")
			print( hostname[:30] + " " , end="")
			print( size[:8] + " " , end="")

			# CHECK IF METHOD USED IS GET | POST
			if request[0] == "G" or request[0] == "P" :
				print( request[:40] + " " , end="")
			else:
				# OTHER METHODS PRINT IN COLOR
				print( Back.BLACK + Fore.RED + Style.DIM + request[:40] + Fore.RESET + Back.RESET + Style.NORMAL + " ", end="" )

			print( ref[:30] + " ", end="")
			print( agent[:3])

		# HTTP 300
		elif code.strip()[0] == "3":
			print( date + " " , end="")
			print( Fore.YELLOW + Style.BRIGHT + code[:4] + Fore.RESET + Style.NORMAL,  end="" )
			print( ip[:15] + " " , end="")
			print( hostname[:30] + " " , end="")
			print( size[:8] + " " , end="")
			print( request[:40] + " " , end="")
			print( ref[:30] + " ", end="")
			print( agent[:3])

		# HTTP 400
		elif code.strip()[0] == "4":
			print( date + " " , end="")
			print( Fore.BLUE + Style.BRIGHT + code[:4] + Fore.RESET + Style.NORMAL,  end="" )
			print( ip[:15] + " " , end="")
			print( hostname[:30] + " " , end="")
			print( size[:8] + " " , end="")

			# CHECK IF METHOD USED IS GET | POST
			if request[0] == "G" or request[0] == "P" :
				print( request[:40] + " " , end="")
			else:
				# OTHER METHODS PRINT IN COLOR
				new_request=Fore.RED + request[:40] + Fore.RESET
				print( new_request + " " , end="")
			print( ref[:30] + " ", end="")
			print( agent[:3])

		# HTTP 500
		elif code.strip()[0] == "5":
			print( date+ " " , end="")
			print( Fore.MAGENTA + Style.BRIGHT + code[:4] + Fore.RESET + Style.NORMAL ,  end="" )
			print( ip[:15] + " " , end="")
			print( hostname[:30] + " " , end="")
			print( size[:8] + " " , end="")
			print( request[:40] + " " , end="")
			print( ref[:30] + " ", end="")
			print( agent[:3])

		# OTHER
		else:
			print( date + " " , end="")
			print( code[:4],  end="" )
			print( ip[:15] + " " , end="")
			print( hostname[:30] + " " , end="")
			print( size[:8] + " " , end="")
			print( request[:40] + " " , end="")
			print( ref[:30] + " ", end="")
			print( agent[:3])

You will see output like this:
log

cpu usage graph in console


License and source code is available at google code.


Just a simple script that displays processor usage graph in command line.

This is how it will look like when running:
CPU graph in console. …or longer version from more real-like usage:

CPU graph with more real like usage

There are two thresholds defined – above 40 % will be shown in yellow and above 60 % will be red.


Requirements (the ones that i can think of):

  • sleep (sleep for a fraction of second is not supported by older versions of sleep)

  • mpstat

Example function used to draw in console:

function green()
  {
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;32m|\033[0m"
        echo -en "\033[u"
  }

33[s saves the current position of the cursor and 33[u restores it to saved position.
tput cup places the cursor in specific row and column.


Full script:

#!/bin/bash
 
# VARIABLES
  columns=40
 
# FUNCTIONS #
function draw()
{
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;3$3m$4\033[0m"
        echo -en "\033[u"
}
function wipe()
{
        echo -en "\033[s"
        tput cup $1 $2
        echo -en "\033[1;31m \033[0m"
        echo -en "\033[u"
}
#############
 
# HIDE CURSOR AND CLEAR SCREEN #
  tput civis & clear
 
while true; do
 
# COLLECT UPDATE
#
 
        idle=`mpstat 1 1 | grep "Average" | tail -1 | sed 's/ \+/ /g' | cut -d " " -f 11 | tr -d"\n"`;
 
        usage=`echo "scale=0;(100-$idle)/10" |bc`;
 
        # BACKUP PREVIOUS DATA #
        # CUT THE OLDEST LINE  #
        cat cpu.log | tail -$columns >> temp.tmp
 
        # UPDATE STATS #
        if [[ $usage -eq 0 ]];
        then
                # IF USAGE LESS THEN 20% DRAW SINGLE BAR #
                echo "1" >> temp.tmp
        else
                echo $usage >> temp.tmp
        fi
 
        # UPDATE LOG #
        cat temp.tmp > cpu.log
        rm temp.tmp
 
        # DRAW GRAPH #
 
        var=6;
        # BEGIN FROM COLUMN 1 #
        j=1
 
        while read usage
        do
                # DRAW USAGE #
                for i in `seq 1 $usage`
                do
                        top=`echo "11-$usage" | bc`
                        var=`echo "11-$i"     | bc`
 
                        if [[ $usage -gt 3 ]]; then
                                if [[ $usage -gt 6 ]]; then
                                        if [[ $var -eq $top ]];then
                                                draw    $var $j "1" "+"
                                        else
                                                draw    $var $j "1" "|"
                                        fi
                                else
                                        if [[ $var -eq $top ]];then
                                                draw    $var $j "3" "+"
                                        else
                                                draw    $var $j "3" "|"
                                        fi
                                fi
                        else
                                if [[ $var -eq $top ]]; then
                                        draw     $var $j "2" "+"
                                else
                                        draw     $var $j "2" "|"
                                fi
                        fi
                done
 
                # WIPE PREVIOUS BAR REMAINNING IF THEY EXIST #
                usage=`echo "$usage+1" | bc`
 
                for k in `seq $usage 11`;
                do
                        var=`echo "11-$k" | bc`
                        wipe $var $j
                done
 
                # PROCEED TO NEXT COLUMN #
                j=`echo "$j+1" | bc`
 
                # ADD LATENCY IF NEEDED #
                sleep 0.1
 
        done < "cpu.log"
        tput cup 11 0
#
done