Apache: Analyse Logs Spam Bots

If you admin a Apache Webserver, you see often weekly thousand of visits a day on your Blogs.

Background:
These are no real users, this visits are made by Spam Bots in my Logs like Xovi.de or xovibot.net Bots!
On info pages this Company says Admins should disallow crawl by robots.txt, but they IGNORE the settings!
This x-guys is in my opinion against German Law „Datenschutz“.

"Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)"

Solution:

  • On Linux Setup a Firewall like ufw and block these IP Ranges
  • To find out the IPs do:

$sudo cat /var/log/apache2/access.log|grep xovibot.net| awk '{ print $2 }' | sort | uniq -c | sort -n > x.log

  • Now read x.log with cat

     46 212.224.119.143
     52 185.53.44.101
     54 212.224.119.140
     59 185.53.44.104
     62 212.224.119.142
     71 185.53.44.102
     75 185.53.44.103
     80 185.53.44.67
     80 212.224.119.141
     83 185.53.44.68
     87 185.53.44.43
     87 185.53.44.69
     96 185.53.44.70
    106 185.53.44.73
    108 185.53.44.51
    110 185.53.44.74
    113 185.53.44.55
    116 185.53.44.45
    116 185.53.44.53
    120 185.53.44.56
    131 185.53.44.71
    132 185.53.44.97
    137 185.53.44.46
    137 212.224.119.144
    141 212.224.119.182
    142 185.53.44.47
    146 185.53.44.41
    150 185.53.44.93
    152 185.53.44.188
    152 185.53.44.203
    152 185.53.44.64
    152 185.53.44.99
    153 185.53.44.184
    154 185.53.44.181
    154 185.53.44.82
    155 212.224.119.139
    156 185.53.44.92
    158 185.53.44.160
    159 185.53.44.202
    160 185.53.44.177
    160 185.53.44.178
    161 185.53.44.175
    163 185.53.44.187
    165 185.53.44.186
    166 185.53.44.189
    168 185.53.44.200
    172 185.53.44.90
    173 185.53.44.159
    173 185.53.44.72
    175 185.53.44.98
    176 185.53.44.96
    177 185.53.44.149
    179 185.53.44.157
    179 185.53.44.183
    183 185.53.44.148
    185 185.53.44.158
    185 185.53.44.63
    186 185.53.44.152
    188 185.53.44.201
    191 185.53.44.176
    191 185.53.44.80
    193 185.53.44.61
    193 185.53.44.94
    202 185.53.44.62

  • And insert the IP ranges of them into the ufw settings by:

$sudo ufw insert 1 deny from 185.53.44.0/24 to any       # insert rule
$sudo service ufw force-reload                           # force update firewall
$sudo ufw status numbered                                # test status

  • Where the „insert 1“ is important cause ufw must see first the deny entry
  • Check the logs manual weekly again with the „cat“ filter.. Kick them out!
  • Remark: This Howto works with every bot entry! There are more Marketing Scan Bots on the net!

More Infos:
http://webrobots.de/xovibot/