Activities

October 2014
M T W T F S S
« Sep   Nov »
 12345
6789101112
13141516171819
20212223242526
2728293031  

IPtables : Blocking bots/spam attack based on the Log file output

Recently I had a client who wants to move more than 300 domains hosted servers to AWS server. His application uses a product which can serve multiple domains based on the request. After a successful migration, Im noticed that web server is hitting badly and causing server load to become high most of the time. After analyzing the Apache log file, I found that one of the comment posting area was compromised and spammers are inserting tons of comments in to the database. This cause Webserver and MySQL load become high most of the time.

After I given some instructions to DevOps, they are fixed the issue by a capcha validation. But still I’m getting same hits but it is showing 301 error message now and getting same rush BAD traffic.

So I decided to create a custom script to analyse the log file output, find the bad network and add it to firewall blocked list.

Note : The outcome of this script is, we can reduce the server load up to 70% after denying the bad network. Now we can get faster throughput from on application servers and database server and hence platform become more stable.

bots-attack

Here is the script action,

1. You need to set the Apache access log file.
2. Set a bash command to filter bad IP from your log file.
3. It will create a log folder “/home/bots_fw” and a file (/home/bots_fw/web_access_log/blocked_networks.txt) which having the blacklisted network.
4. Once your Apache log files are processed, a copy of this file stored on /home/bots_fw/web_access_log and preserved for 10 days.
5. When this script is executed from cron, it will check the network from the blocked_networks.txt file and add the network in to iptables once it can not find any entries.

You can download the files here

#!/bin/bash
#Written by Liju Mathew,liju@serveridol.com
# This script is used for blocking IP network those are hitting the web server to hack something or used for
# post something using application urls. These attempts would cause server resource become fully used for  bad traffic
access_log_path="/var/httpd/access.log"
access_log_backup="/home/bots_fw/web_access_log"

#This file contains the network those are added earlier
blocked_networks_file=$access_log_backup"/blocked_networks.txt"
# creating logs directory
[ ! -d $access_log_backup ] && mkdir -p $access_log_backup
[ ! -f $blocked_networks_file ] && touch $blocked_networks_file

#Minimum no of hits coming from a bad ip address
bad_access_limit=50

#Script variables
network_counter=0
TMP="/tmp"
touch $TMP/bad_ip_report.txt
cat /dev/null > $TMP/bad_ip_report.txt

#Deleting the access log backup which is 10 days old
/usr/bin/find $access_log_backup  -mtime +10 -exec rm {} \;

# Creating IP filter file. You can put any filter statement to get the IP address of suspected user
#But the resultant output should have list of ip address

cat $access_log_path | grep "/blogdetailspage" | awk ' { print $1 }'  | sort |uniq -c | sort -g -r | head -n 100 >> $TMP/bad_ip_report.txt
ip_filter_file=$TMP/bad_ip_report.txt

while read line
do
no_occurance="$(echo $line  |  awk '{print $1}')"
bad_ip="$(echo $line |  awk '{print $2}')"

if [[ $no_occurance -ge $bad_access_limit ]]
then
      #check it is a ip number
      if [[ $bad_ip == *.* ]]
         then
         clear
         echo " Processing the IP $bad_ip.."
         #Finding IP network
         bad_network="$(echo $bad_ip | awk -F. '{print $1"."$2}')"
         bad_network=$bad_network".0.0/16"
     #check the given IP block aganist our added firewall list   
    network_exists=0   
         for entry in `cat $blocked_networks_file`;
        do
         
                if [[ $entry == $bad_network ]]
                    then
                        #echo "match found. Skipping $entry"
                        network_exists=$((network_exists+1))
                        break
                        else
                             echo ""
                        fi
         done          
                                   
              # If network_exits value is Zero added it to firewall
            if [[ $network_exists -eq 0 ]]
                        then
                            # IP is added to Fireall
                            network_counter=$((network_counter+1))
                            iptables -I INPUT -s $bad_network   -p tcp --dport 80 -j DROP
                           echo "$bad_network" >> $blocked_networks_file
                       
                        else
                        echo ""
                        fi

          fi
fi

done <"$ip_filter_file"

echo " Total $network_counter Networks are added to firewall"          

# following commands are optional. I've truncated the log file to reduce the computing power which taking my cron when the apache logs
#are heavy
# Moving logs to backup location
BKP_File=$access_log_backup"/Web-Accesslog-""$(date +"%m-%d-%y-%H-%M").log"
cp $access_log_path $BKP_File

#Flushing Apache log file to avoid duplicate entries
cat /dev/null > $access_log_path

#cronjob Entries
##00  */6 * * * /bin/bash /home/installation/scripts/deny_bots_attack.sh   >/dev/null 2>&1

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>