Rsync: Performance NFS Boost

If you use two Linux Servers with NFS Shares connect over Gigabit Interfaces for Backups, you perhaps remark performance problems on using rsync.
It does often pause transmissions or reach only a rate of 32Mb/s. That’s bad and waste time and energy.

Background:
After some Tests of running Backups with single files and compressed big archives, the scans of my tools like iftop, systat, iptraf found out that the used Option „rsync -avz“ was the bottle neck, cause the Data Rate break down at intervals. The „z“ compress option was described on howtos as must have for slower networks did not work like a „on the fly permanent transmission“. My two Test Servers use each 3 Gigabit bonded NIC Cards, if the rsync command compress the Files, they were first pulled to RAM Cache and compressed there later after a small pause send.

Solution:
If you disable and remove the „z“ option and on used bash scripts as „batch run“ removing the „v“ Option too, i was able to send Data at range of 133MB/s.

On fast Network do:

$rsync -a /source/ /nfs-mounted-destination

Amazon: Cloud Drive Sharing with NFS on Local Network as Photo Uploader Backup

Current the acd_cli tool for amazon drive mount offers no „fsid“ (device /dev) point, that you cant export and share the amazon drive at your home local network.

On the latest raspberry pi OS (jessie) you can use a small workaround to get a NFS Shared Amazon Cloud Backup Uploader (needs python3.4 !)

  • install acd_cli tools, create the auth file (read) and  run $ sudo acd_cli mount /amazoncloud
  • now create a second Folder /sendtoamazon
  • install nfs-kernel-Server and share this Folder with NFS exports to your LAN
  • on the raspberrypi you can use a cron „move“ Job running every 10 Minutes:
    #crontab -e
    */10 * * * * mv /sendtoamazon/* /amazoncloud
    

If you now put files on the /sendtoamazon nfs share from a PC,  the raspberry pi will grab and push it into the cloud via the „acd_cli mount“ a python3 script, remark 10 minutes is ok for small files! on bigger files use over „40“ minutes between pushes. If acd_cli fails check for correct python version! and last python modules!!! (python3-appdirs python3-dateutil python3-requests python3-sqlalchemy python3-pip pip3)

If you have a SMALL BANDWITH for upload you can use „rsync -avz“ instead mv (move) with „throttle“ option to upload files slowly!!

Links:

Howto acd_cli for amazon mount on git