Apache: Count Visits on Console

If you use a Webserver like Apache, you can use a small script to Analyse your Logs. Create a analyse-web.sh Script with:

 $sudo nano /home/user/analyse-web.sh 

insert:
#!/bin/bash
cat /var/log/apache2/access.log | awk '{ print $1 }' | sort | uniq -c
exit 0

System Output:
1573  www.domain2.de
3568  www.domain3.de
..

If you change the „$1“ to other value like „$8“ you will get the count of touched files or folders! This shows you attacks on single Files by abnormal high counts! You can use cron to run it every 15Minutes and send it to mailbox of a user. And this way does not need a PHP Tool with special PHP rights like webalizer or else..

Apache: Analyse Logs Spam Bots

If you admin a Apache Webserver, you see often weekly thousand of visits a day on your Blogs.

Background:
These are no real users, this visits are made by Spam Bots in my Logs like Xovi.de or xovibot.net Bots!
On info pages this Company says Admins should disallow crawl by robots.txt, but they IGNORE the settings!
This x-guys is in my opinion against German Law „Datenschutz“.

"Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)"

Solution:

  • On Linux Setup a Firewall like ufw and block these IP Ranges
  • To find out the IPs do:

$sudo cat /var/log/apache2/access.log|grep xovibot.net| awk '{ print $2 }' | sort | uniq -c | sort -n > x.log

  • Now read x.log with cat

     46 212.224.119.143
     52 185.53.44.101
     54 212.224.119.140
     59 185.53.44.104
     62 212.224.119.142
     71 185.53.44.102
     75 185.53.44.103
     80 185.53.44.67
     80 212.224.119.141
     83 185.53.44.68
     87 185.53.44.43
     87 185.53.44.69
     96 185.53.44.70
    106 185.53.44.73
    108 185.53.44.51
    110 185.53.44.74
    113 185.53.44.55
    116 185.53.44.45
    116 185.53.44.53
    120 185.53.44.56
    131 185.53.44.71
    132 185.53.44.97
    137 185.53.44.46
    137 212.224.119.144
    141 212.224.119.182
    142 185.53.44.47
    146 185.53.44.41
    150 185.53.44.93
    152 185.53.44.188
    152 185.53.44.203
    152 185.53.44.64
    152 185.53.44.99
    153 185.53.44.184
    154 185.53.44.181
    154 185.53.44.82
    155 212.224.119.139
    156 185.53.44.92
    158 185.53.44.160
    159 185.53.44.202
    160 185.53.44.177
    160 185.53.44.178
    161 185.53.44.175
    163 185.53.44.187
    165 185.53.44.186
    166 185.53.44.189
    168 185.53.44.200
    172 185.53.44.90
    173 185.53.44.159
    173 185.53.44.72
    175 185.53.44.98
    176 185.53.44.96
    177 185.53.44.149
    179 185.53.44.157
    179 185.53.44.183
    183 185.53.44.148
    185 185.53.44.158
    185 185.53.44.63
    186 185.53.44.152
    188 185.53.44.201
    191 185.53.44.176
    191 185.53.44.80
    193 185.53.44.61
    193 185.53.44.94
    202 185.53.44.62

  • And insert the IP ranges of them into the ufw settings by:

$sudo ufw insert 1 deny from 185.53.44.0/24 to any       # insert rule
$sudo service ufw force-reload                           # force update firewall
$sudo ufw status numbered                                # test status

  • Where the „insert 1“ is important cause ufw must see first the deny entry
  • Check the logs manual weekly again with the „cat“ filter.. Kick them out!
  • Remark: This Howto works with every bot entry! There are more Marketing Scan Bots on the net!

More Infos:
http://webrobots.de/xovibot/

WordPress: Why less plugins is more?

WordPress is a well known Blogging Platform to publish your Content to the world. But after some years of analysing of my Webserver Logs i remarked that a lot of used plugins produces more errors on inside linking than i would like to have. I could not explain why the Search Engines run into Errors or dead ends if they scan via the bots the Content. I didn’t realize that the plugins were the bad factor.

If you see same problems you can start up analyse your Blog for free with these steps:

  • Go Plugins disable all uneeded Plugins like Cache, Tags .. and check that the Cache Folder is empty on your Webspace!! Have seen that trash is always there if i disable or remove the plugins.
  • Go Settings, Reading, General and save all Settings again! If possible reset to WordPress defaults
  • Check URLs Settings! http / https, i prefer https cause google does rank it higher than http.
  • Check URLs of Images / Media Content, purge external Content!! Againt’s copyright violations!!
  • Purge outdated Posts!
  • Checkout if some older Plugins Functions are now integrated into the new Version of WordPress by default. If this happen purge the plugin! Cause Search Engines don’t like double tagging or linking, this can by calculated as Spam be the Search Engines.
  • You should know less plugins offers more Security, cause most of them are made by one developer (one man show like openssl bugs)
  • Try to use wordpress like more default, this offers you the security that bots can grab the page easy without errors.
  • Cache Plugins can speed up wordpress on low end Webspaces, but the static page creation can be a risc if the Plugin crashes on Caching and save destroyed Pages on the Cache Folder!
  • Run WordPress on a Webspace where you have full access to Logs and the Webspaces, that you can control and purge bugs by hand.
  • Check weekly Logs and if you use Search Engine Webmastertools for free.
  • Use a easy loadable and readable Theme, cause longer Load Times are ranked down too!

Now you know why less is more , have fun with blogging..

Ubuntu Debian: Installer Messages during Installation Realtime Debug Installer Mode

If you boot your PC on a Setup CD into „live“ mode after some time the Desktop Unity or Gnome starts. After successful testing the PC with the current Drivers and Kernel you can select „Install“ by pressing the icon left on the desktop. But here the Problem appear that you dont see any informations about bugs, crashes, or other major messages during the setup. Without this you are not able to find any errors later, or you cant tell a supporter what happend!

Workaround (#comments!):

  • Always open a gnome-terminal as first step instead of pressing the installer icon!
  • move the gnome-terminal window on lower screen area to 100% width!!!
  • Enter there:

$sudo tail -f /var/log/syslog   # some Linux uses /var/log/messages!

  • Now press Installer icon! and view Log in realtime

Hope this help to fix bugs like freezes of often seen SSD Errors I/O…