Apache: Analyse Logs Spam Bots

If you admin a Apache Webserver, you see often weekly thousand of visits a day on your Blogs.

Background:
These are no real users, this visits are made by Spam Bots in my Logs like Xovi.de or xovibot.net Bots!
On info pages this Company says Admins should disallow crawl by robots.txt, but they IGNORE the settings!
This x-guys is in my opinion against German Law „Datenschutz“.

"Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)"

Solution:

  • On Linux Setup a Firewall like ufw and block these IP Ranges
  • To find out the IPs do:

$sudo cat /var/log/apache2/access.log|grep xovibot.net| awk '{ print $2 }' | sort | uniq -c | sort -n > x.log

  • Now read x.log with cat

     46 212.224.119.143
     52 185.53.44.101
     54 212.224.119.140
     59 185.53.44.104
     62 212.224.119.142
     71 185.53.44.102
     75 185.53.44.103
     80 185.53.44.67
     80 212.224.119.141
     83 185.53.44.68
     87 185.53.44.43
     87 185.53.44.69
     96 185.53.44.70
    106 185.53.44.73
    108 185.53.44.51
    110 185.53.44.74
    113 185.53.44.55
    116 185.53.44.45
    116 185.53.44.53
    120 185.53.44.56
    131 185.53.44.71
    132 185.53.44.97
    137 185.53.44.46
    137 212.224.119.144
    141 212.224.119.182
    142 185.53.44.47
    146 185.53.44.41
    150 185.53.44.93
    152 185.53.44.188
    152 185.53.44.203
    152 185.53.44.64
    152 185.53.44.99
    153 185.53.44.184
    154 185.53.44.181
    154 185.53.44.82
    155 212.224.119.139
    156 185.53.44.92
    158 185.53.44.160
    159 185.53.44.202
    160 185.53.44.177
    160 185.53.44.178
    161 185.53.44.175
    163 185.53.44.187
    165 185.53.44.186
    166 185.53.44.189
    168 185.53.44.200
    172 185.53.44.90
    173 185.53.44.159
    173 185.53.44.72
    175 185.53.44.98
    176 185.53.44.96
    177 185.53.44.149
    179 185.53.44.157
    179 185.53.44.183
    183 185.53.44.148
    185 185.53.44.158
    185 185.53.44.63
    186 185.53.44.152
    188 185.53.44.201
    191 185.53.44.176
    191 185.53.44.80
    193 185.53.44.61
    193 185.53.44.94
    202 185.53.44.62

  • And insert the IP ranges of them into the ufw settings by:

$sudo ufw insert 1 deny from 185.53.44.0/24 to any       # insert rule
$sudo service ufw force-reload                           # force update firewall
$sudo ufw status numbered                                # test status

  • Where the „insert 1“ is important cause ufw must see first the deny entry
  • Check the logs manual weekly again with the „cat“ filter.. Kick them out!
  • Remark: This Howto works with every bot entry! There are more Marketing Scan Bots on the net!

More Infos:
http://webrobots.de/xovibot/

WordPress: Monitor the Size of the MYSQL Database

If you use WordPress or a other Blog Software with a MySQL Database over years it is useful to do some things regularly:

  • Check the Database Size weekly, that no Skript Kid has found a Backdoor and fills up the Database silent
  • Purge Database Caches with default maintenance tools of the Software
  • Dont forget to create a MySQL Dump Backup weekly with cron

To Check the Size login on mysql command prompt do:

$mysql -u dbuser -p #Enter Password

mysql > use dbwordpress

mysql > SELECT table_schema "Data Base Name", sum( data_length + index_length) / 1024 / 1024 
"Data Base Size in MB" FROM information_schema.TABLES GROUP BY table_schema;

System Echo a Table of all of your Databases with size!

If the size is much more bigger than last week and you haven’t changed anything you can try to cleanup the Database by for example with a upgrade.php, update.php or other commandline PHP Scripts of the PHP Kit (WordPress, Drupal..) Maintenance Tools. Often many little things are cached into the MySQL Databases. If this doesn’t help, rewind to older MySQL Dump and test again. Don’t forget to keep the old WordPress-DB and rewinded WordPress-DB MySQL Dump secure! If the Database grows again fast, login to WordPress, check comments area of the posts sites. If comments are allowed by Guests, disable them to prevent Spammers.

Major Webserver Setup Rules: The „must“ do!

If you want to setup a fresh secure Webserver then use this list

  • dont ever upload data, files, images on the Webserver who are classified as „secret“
  • setup daily full backup with Cron
  • view daily the system logs, auth, www, errors …
  • setup a local firewall with less opened ports 80,25 ..
  • setup daily automatic updates by cron
  • reduce the count of users who can login
  • use no logical usernames
  • force long passwords by rules min 15 digits
  • setup a daily load monitor by „uptime“ to log
  • setup a realtime network monitor by „iftop“
  • use „nmap“ as local portscan to test settings
  • remove unneeded software packages and services, less is more..
  • change monthly passwords forced by rules
  • upload only via encrypted SFTP and use Login Keys
  • copy hourly your Logs on /var/log to a other external place (scp/rsync) by cron
  • use ECC-RAM to have save RAM usage againts RAM attacks
  • use 2 HDD’s as Raid 1 and setup mail of root to post failures to your box
  • mount the Webserver root file system readonly, that no one can modify /etc
  • dont use Java, PHP, Tomcat, or other Adminpanels if you really need them!

I hope this rules help you to protect your Server..