Tag Archives: bash

Admin, Security, Tools

Do We Trust Cloud Storage For Privacy?

Leave a comment

With more generic offerings from  cloud storage providers –  up to 50GB free,   cloud storage is tempting alternative to store some of our data. I have some data, which I really do not want to loose. I already have them stored on several devices, however additional copy in cloud could help.  But how much I can trust cloud providers to keep my data private, even from their own employees.  Not that I have something super secret, but somehow I do not like idea, that some bored sysadmin, will be browsing my family photos.  Or provider  use my photos for some machine learning algorithms.

Main providers like Dropbox, Google do use some encryption, however they control  encryption keys, so they can theoretically access your data any time and in worst case provide them to third parties – like government agencies.   From what I have been looking around only few providers like Mega or SpiderOak  offer privacy  by design – which means  all encryption is done on client and they should not have any access to your keys (zero knowledge).   However how much we can trust that their implementation is flawless or that there are not intentional back-doors left? There has been some concerns about Mega security couple years ago,  but no major issues appeared since then.

So rather then trusting those guys fully, why not to take additional step and also encrypt our data, before sending them to cloud?  Additional encryption will not cost us much CPU time on current hardware (from tests – 11% of one core of old AMD CPU) and will not slow down transfers, because they are rather limited by Internet connection bandwidth.  And on Linux we have quite few quality encryption  tools like gpg or openssl, which can be relatively easily integrated into our backup/restore chains. In the rest of this article I’ll describe my PoC shell script, that backs up/ restores  whole directory to MEGA, while providing additional encryption / decryption on client side.  Continue reading Do We Trust Cloud Storage For Privacy? →

Admin

Simple statistics from nginx access logs

Leave a comment

I required some simple statistics (selected page visits per day)  from web-server logs.   I looked at some web log analyzer packages like AWStats, but it looked to me like as an overkill in my case – I’d probably spent more time to trying make it work then putting together some small script. So here it is – a simple bash script that will take all available access logs (by default on Debian nginx is using logrotate to rotate logs daily and keeps 52 daily logs, old logs are gzipped) and calculate page visits for certain request pattern: Continue reading Simple statistics from nginx access logs →

Admin

Converting files in directory structure

Leave a comment

Often I needed to convert a set of files (audio files to different format,  text files to different encoding, …)  in some directory structure and save results to a  new destination while retaining directory structure.   In order to walk through directory structure we can use find command,   but if we are to create output in new place, subdirectories have to be created, which is where find command line gets bit complicated (especially if we have to consider spaces in files/directories names).   So here is an example for converting text files encoding:

find . -name "*.txt" -exec bash -c 'mkdir -p "/dest-dir/`dirname "{}"`"; iconv -f utf-8 -t windows-1250 -o "/dest-dir/{}" "{}"' \;