Schlagwort: stop crawling

Apache: Analyse Logs Spam Bots

If you admin a Apache Webserver, you see often weekly thousand of visits a day on your Blogs. Background: These are no real users, this visits are made by Spam Bots in my Logs like Xovi.de or xovibot.net Bots! On info pages this Company says Admins should disallow crawl by robots.txt, but they IGNORE the settings! This x-guys is in my opinion against German Law „Datenschutz“. "Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)" Solution: On Linux Setup a Firewall like ufw and block these IP Ranges To find out the IPs do: $sudo cat /var/log/apache2/access.log|grep xovibot.net| awk '{ print $2 }' | sort | […]