Apache MEMCACHED UDP Protection

Current a lot of sites blogging about memcached attacks on Servers here some details:

  • Memcached Servers need a installed and running Service called “memcached”
  • Websites need a php-plugin like php7.0-memcached to connect via API to the memcached Service
  • The Memcached Service uses a own Config File at debian /etc/memcached.conf
  • By default it MUST listen to localhost or socket
  • Admins MUST setup a FIREWALL like “ufw” (iptables) and MUST check own Server for OPEN PORTS with nmap
  • The Problem is that Attackers can run Scripts against to your Server in a 10^6 Range like a BOTNET !! with ONE PC cause MEMCACHED supports this high count of REQUESTS without going down.
  • DO NEVER HOLD CONFIDENTIAL DATA ON WEBSERVERS!!!

Test to open Port using nmap Port Scan with UDP Option NOT TCP:

sudo nmap -sU -p 11211 www.myserver.xyz

If the scan echo this YOU MUST check or install a FIREWALL!:
Host is up (0.10s latency).
PORT      STATE         SERVICE
11211/udp open|filtered unknown

if Echo shows this you are safe:
PORT      STATE    SERVICE
11211/udp filtered unknown

check your current Apache PHP Modules:

$sudo php -m

if memcached listed, the php api is active time to check more..

check for memcached service:

$sudo dpkg -l |grep mem

is memcached listed the service is installed, then do:
$sudo ps aux|grep mem

if the echo shows:
memcache ... /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid

the Service is active an listening..

Sample Config:
/etc/memcached.conf

# memcached default config file
# 2003 - Jay Bonci <jaybonci@debian.org>
# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.
# Run memcached as a daemon. This command is implied, and is not needed for the
# daemon to run. See the README.Debian that comes with this package for more
# information.
-d
# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log
# Be verbose
-v
# Be even more verbose (print client commands as well)
-vv
# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
-m 128
# Default connection port is 11211
-p 11211
# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
# Limit the number of simultaneous incoming connections. The daemon default is 1024
-c 300
# Lock down all paged memory. Consult with the README and homepage before you do this
# -k
# Return error when memory is exhausted (rather than removing items)
-M
# Maximize core file limit
# -r
# Use a pidfile
-P /var/run/memcached/memcached.pid

Setup Firewall (ufw):

$sudo apt-get install ufw
$sudo ufw allow 80/tcp
$sudo ufw allow 443/tcp
$sudo ufw enable

Retest with NMAP Port Scan your OPEN Ports! Do this monthly! Cause sometimes the Firewall can have unknown Problems!!

Check the Memcached Log at /var/log/memcached.log for Events

ENFORCE Google to DuckDuckgo SEARCH

If you want to enforce the use of DuckDuckgo.com instead of google.com do:

Edit at the PC the “hosts” File on:

Linux /etc/hosts
Windows C:\Windows\System32\drivers\etc

insert at last:

54.229.105.92  google.com  #ip of duckduckgo or 176.34.131.233
54.229.105.203 google.com #ip of duckduckgo
176.34.131.233 bing.com #ip of duckduckgo or 176.34.131.233
176.34.131.233 yahoo.com #ip of duckduckgo or 176.34.131.233

..reboot and test on a Browser Session after google.com you see duckduckgo.com

Remark:

  • Most DSL Routers do offer the edit of the hosts File too, do same there and ALL devices redirected!
  • Don’t forget to reboot!
  • This Solution works only on IPv4 Networks, to enforce the local net, disable IPv6 forwarding on your ISP Router!
  • Test the “addressbar” search, there the redir does not work, cause compiled in IP’s or IPv6 broadcast, then remove uneeded Search Engines on the Browser Settings!
  • Use the free fork of Firefox named icecat Browser

Locale Umloud Problems Cron

If you run scripts to handle text output by cronjobs your perhaps get problems with umlouds “ÖÄÜ” cause they are displayed by “**”.
This is a problem cause cron uses “C” setting as locale, you can test it by setting it into root crontab:

open crontab from root with:

$su - root
$crontab -

insert
* * * * * locale

This will mail cron’s locale echo to the mailbox of root! Read root’s mail!
After tests remove the locale entry at crontab!

Howto fix for Scripts:

open crontab from root with:

$su - root
$crontab -e

insert (for German):

LANG=de_DE.UTF-8 
LC_ALL=de_DE.UTF-8

for US:
LANG=en_US.UTF-8 
LC_ALL=en_US.UTF-8

Postfix: Automatic UFW Firewall Updates

If you use a Mail Server with Postfix you got daily Spam Attacks by Scripts:

How to fix?

  • Install ufw Firewall
  • Run a Scanner Script as  cronjob

On Debian/Ubuntu:

Install ufw:


sudo apt-get update && sudo apt-get install ufw && sudo ufw enable && sudo  ufw logging off

Scan Script:
sudo nano /home/user/firewall-update.sh:


#!/bin/bash
# scan rejected
cat /var/log/mail.log | grep rejected | cut -d"[" -f3 | cut -d"]" -f1|grep -v '^$' > /tmp/firewall.txt
# insert to Firewall
while read line; do sudo ufw insert 1 deny from $line to any; done < /tmp/firewall.txt
# scan "denied"
cat /var/log/mail.log | grep denied | cut -d"[" -f3 | cut -d"]" -f1|grep -v '^$' > /tmp/firewall2.txt
# insert to Firewall
while read line; do sudo ufw insert 1 deny from $line to any; done < /tmp/firewall2.txt
service ufw restart
exit 0

Remark:

  • add to root’s crontab run hourly
  • add lines cat..+ while.. replace rejected by other failed commands login etc..!

Test:


sudo ufw status numbered

Should echo list of banned IP’s! Screenshot of one DAY!

Status: active

To Action From
— —— —-
Anywhere DENY 87.98.131.120
Anywhere DENY 187.178.174.1
Anywhere DENY 46.148.88.115
Anywhere DENY 31.28.86.59
Anywhere DENY 200.6.213.125
Anywhere DENY 200.35.185.180
Anywhere DENY 37.49.227.221
Anywhere DENY 203.60.1.21
Anywhere DENY 78.110.2.2
Anywhere DENY 192.140.8.21
Anywhere DENY 66.240.219.146
Anywhere DENY 171.49.178.169
Anywhere DENY 186.179.219.145
Anywhere DENY 119.235.53.122
Anywhere DENY 95.177.213.224
Anywhere DENY 14.161.43.66
Anywhere DENY 203.143.23.69
Anywhere DENY 104.215.8.206
Anywhere DENY 191.96.249.84
Anywhere DENY 190.190.167.206
Anywhere DENY 95.97.176.158
Anywhere DENY 200.77.219.250
Anywhere DENY 191.248.224.38
Anywhere DENY 91.237.124.222
Anywhere DENY 31.27.32.18
Anywhere DENY 122.174.172.246
Anywhere DENY 208.92.136.194
Anywhere DENY 39.52.234.157
Anywhere DENY 202.188.23.209
Anywhere DENY 154.72.169.179
Anywhere DENY 88.247.177.95
Anywhere DENY 98.116.114.130
Anywhere DENY 203.45.1.236
Anywhere DENY 115.74.190.217
Anywhere DENY 183.129.160.229
Anywhere DENY 203.143.23.66
Anywhere DENY 201.187.101.222
Anywhere DENY 46.13.150.215
Anywhere DENY 104.215.11.242
Anywhere DENY 77.119.247.86
Anywhere DENY 94.46.187.190
Anywhere DENY 78.89.186.118
Anywhere DENY 185.218.184.95
Anywhere DENY 103.10.44.223
Anywhere DENY 167.114.226.176
Anywhere DENY 185.38.154.18
Anywhere DENY 85.105.58.91
Anywhere DENY 149.135.117.174
Anywhere DENY 104.236.166.245
Anywhere DENY 27.115.124.2
Anywhere DENY 181.60.254.53
Anywhere DENY 196.22.248.246
Anywhere DENY 165.49.18.249
Anywhere DENY 159.147.100.63
Anywhere DENY 192.95.17.132
Anywhere DENY 137.74.89.35
Anywhere DENY 193.70.87.209
Anywhere DENY 216.36.187.97
Anywhere DENY 101.187.124.125
Anywhere DENY 2.42.219.63
Anywhere DENY 185.109.169.71
Anywhere DENY 201.26.128.167
Anywhere DENY 133.130.74.177
Anywhere DENY 2.139.229.39
Anywhere DENY 177.1.7.49
Anywhere DENY 103.227.88.130
Anywhere DENY 46.102.196.66
Anywhere DENY 109.230.219.194
Anywhere DENY 180.94.114.47
Anywhere DENY 212.170.109.162
Anywhere DENY 190.24.136.122
Anywhere DENY 82.152.228.49
Anywhere DENY 181.49.39.70
Anywhere DENY 103.240.181.210
Anywhere DENY 189.51.83.246
Anywhere DENY 61.19.16.144
Anywhere DENY 45.76.95.222
Anywhere DENY 178.90.55.176
Anywhere DENY 87.139.234.44
Anywhere DENY 200.116.164.5
Anywhere DENY 24.222.140.202
Anywhere DENY 103.100.209.234
Anywhere DENY 82.201.54.152
Anywhere DENY 84.241.1.21
Anywhere DENY 110.145.123.120
Anywhere DENY 185.32.183.141
Anywhere DENY 200.49.145.161
Anywhere DENY 213.156.120.22
Anywhere DENY 95.59.137.196
Anywhere DENY 185.229.227.5
Anywhere DENY 188.225.171.58
Anywhere DENY 211.24.107.177
Anywhere DENY 186.233.80.51
Anywhere DENY 187.178.242.154
Anywhere DENY 190.223.59.18
Anywhere DENY 202.181.207.212
Anywhere DENY 41.87.95.33
Anywhere DENY 52.175.252.79
Anywhere DENY 103.252.220.20
Anywhere DENY 212.230.98.37
Anywhere DENY 41.180.72.44
Anywhere DENY 196.191.131.50
Anywhere DENY 120.150.227.127
Anywhere DENY 181.211.10.202
Anywhere DENY 218.255.233.114
Anywhere DENY 181.143.94.74
Anywhere DENY 196.38.89.85
Anywhere DENY 190.187.134.246
Anywhere DENY 76.65.196.40
Anywhere DENY 222.102.154.172
Anywhere DENY 221.121.148.77
Anywhere DENY 203.191.174.55
Anywhere DENY 190.25.46.42
Anywhere DENY 169.55.213.44
Anywhere DENY 86.16.10.224
Anywhere DENY 89.96.222.27
Anywhere DENY 202.131.203.163
Anywhere DENY 200.85.52.74
Anywhere DENY 94.23.73.132
Anywhere DENY 41.193.16.218
Anywhere DENY 175.136.232.97
Anywhere DENY 118.219.45.141
Anywhere DENY 205.151.252.203
Anywhere DENY 82.113.59.26
Anywhere DENY 178.33.107.200
Anywhere DENY 82.185.149.169
Anywhere DENY 220.130.186.101
Anywhere DENY 201.33.193.166
Anywhere DENY 178.159.36.60
Anywhere DENY 74.125.82.65
Anywhere DENY 74.125.82.67
Anywhere DENY 209.85.128.193
Anywhere DENY 209.85.128.195
Anywhere DENY 178.32.217.0/24
Anywhere DENY 74.125.82.66
Anywhere DENY 209.85.128.194
Anywhere DENY 209.85.128.196
Anywhere DENY 74.125.82.68
Anywhere DENY 104.236.142.81
Anywhere DENY 60.191.38.77
Anywhere DENY 5.101.0.34
Anywhere DENY 209.126.136.5
Anywhere DENY 145.249.104.109
Anywhere DENY 37.49.226.113
Anywhere DENY 189.112.109.185
Anywhere DENY 172.104.155.22
Anywhere DENY 23.227.207.153
Anywhere DENY 142.4.196.32
Anywhere DENY 210.72.142.7
Anywhere DENY 46.21.174.130
Anywhere DENY 13.91.5.211
Anywhere DENY 212.53.206.58
Anywhere DENY 167.114.60.66
Anywhere DENY 144.217.126.187
Anywhere DENY 144.217.210.228
Anywhere DENY 104.236.163.154
Anywhere DENY 61.236.111.38

Linux: Systemd ignore console-setup settings

If you work on older Laptops and you use a Console only System Setup like on Debian there is a Bug on systemd and the console-setup package since years. After reboot all Font Settings seems gone. But the Settings are not real gone, cause systemd does not pull the settings on boot!

How to fix?

  • edit the crontab of root by

sudo su -
crontab -e

  • insert the /bin/setupcon command on “@reboot” means on every boot!

@reboot     /bin/setupcon > /dev/null 2>&1 

  • save and exit, reboot now
  • now the PC should echo big Fonts for old eyes “Terminus 20×12 Frambuffermode”

 

Apache: Analyse Logs Spam Bots

If you admin a Apache Webserver, you see often weekly thousand of visits a day on your Blogs.

Background:
These are no real users, this visits are made by Spam Bots in my Logs like Xovi.de or xovibot.net Bots!
On info pages this Company says Admins should disallow crawl by robots.txt, but they IGNORE the settings!
This x-guys is in my opinion against German Law “Datenschutz”.

"Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)"

Solution:

  • On Linux Setup a Firewall like ufw and block these IP Ranges
  • To find out the IPs do:

$sudo cat /var/log/apache2/access.log|grep xovibot.net| awk '{ print $2 }' | sort | uniq -c | sort -n > x.log

  • Now read x.log with cat

46 212.224.119.143
52 185.53.44.101
54 212.224.119.140
59 185.53.44.104
62 212.224.119.142
71 185.53.44.102
75 185.53.44.103
80 185.53.44.67
80 212.224.119.141
83 185.53.44.68
87 185.53.44.43
87 185.53.44.69
96 185.53.44.70
106 185.53.44.73
108 185.53.44.51
110 185.53.44.74
113 185.53.44.55
116 185.53.44.45
116 185.53.44.53
120 185.53.44.56
131 185.53.44.71
132 185.53.44.97
137 185.53.44.46
137 212.224.119.144
141 212.224.119.182
142 185.53.44.47
146 185.53.44.41
150 185.53.44.93
152 185.53.44.188
152 185.53.44.203
152 185.53.44.64
152 185.53.44.99
153 185.53.44.184
154 185.53.44.181
154 185.53.44.82
155 212.224.119.139
156 185.53.44.92
158 185.53.44.160
159 185.53.44.202
160 185.53.44.177
160 185.53.44.178
161 185.53.44.175
163 185.53.44.187
165 185.53.44.186
166 185.53.44.189
168 185.53.44.200
172 185.53.44.90
173 185.53.44.159
173 185.53.44.72
175 185.53.44.98
176 185.53.44.96
177 185.53.44.149
179 185.53.44.157
179 185.53.44.183
183 185.53.44.148
185 185.53.44.158
185 185.53.44.63
186 185.53.44.152
188 185.53.44.201
191 185.53.44.176
191 185.53.44.80
193 185.53.44.61
193 185.53.44.94
202 185.53.44.62

  • And insert the IP ranges of them into the ufw settings by:

$sudo ufw insert 1 deny from 185.53.44.0/24 to any       # insert rule
$sudo service ufw force-reload                           # force update firewall
$sudo ufw status numbered                                # test status

  • Where the “insert 1” is important cause ufw must see first the deny entry
  • Check the logs manual weekly again with the “cat” filter.. Kick them out!
  • Remark: This Howto works with every bot entry! There are more Marketing Scan Bots on the net!

More Infos:
http://webrobots.de/xovibot/

Rsync: Performance NFS Boost

If you use two Linux Servers with NFS Shares connect over Gigabit Interfaces for Backups, you perhaps remark performance problems on using rsync.
It does often pause transmissions or reach only a rate of 32Mb/s. That’s bad and waste time and energy.

Background:
After some Tests of running Backups with single files and compressed big archives, the scans of my tools like iftop, systat, iptraf found out that the used Option “rsync -avz” was the bottle neck, cause the Data Rate break down at intervals. The “z” compress option was described on howtos as must have for slower networks did not work like a “on the fly permanent transmission”. My two Test Servers use each 3 Gigabit bonded NIC Cards, if the rsync command compress the Files, they were first pulled to RAM Cache and compressed there later after a small pause send.

Solution:
If you disable and remove the “z” option and on used bash scripts as “batch run” removing the “v” Option too, i was able to send Data at range of 133MB/s.

On fast Network do:

$rsync -a /source/ /nfs-mounted-destination