I required some simple statistics (selected page visits per day) from web-server logs. I looked at some web log analyzer packages like AWStats, but it looked to me like as an overkill in my case – I’d probably spent more time to trying make it work then putting together some small script. So here it is – a simple bash script that will take all available access logs (by default on Debian nginx is using logrotate to rotate logs daily and keeps 52 daily logs, old logs are gzipped) and calculate page visits for certain request pattern: Continue reading Simple statistics from nginx access logs
Category Archives: Admin
Poor man’s backup for XenServer
I’m running several XenServer hosts and wanted to provide some basic backup of VMs. I decided to use USB disk – XenServer 6.2 provides great support for external disks. I was looking around for some simple free tool for backup (to backup several VMs from different servers in scheduled batches), but did not find anything suitable (simple scripts were not flexible enough, bigger solutions were overkill in my case) – so I created my own solution xapi-back
My setup is:
- I created small Debian VM and attached USB disk to it (in XS 6.2 this external disk will stay connected to VM after VM or host reboot)
- Installed xapi-back
- Created special user for backup
- Scheduled VM backups with cron
Main advantages of xapi-back compared to other similar solutions:
- easy to install – just download and run
python setup.py - easy to configure – just one simple configuration file with details of xen servers and some basic backup parameters
- self-contained – does not need xe or other tools (as many other solutions) and can run on any computer ( not only in xenserver Dom0 as some scripts, generally I think it’s not good practice to run backups in Dom0, better is to have it separately).
- complete – you can do all basic tasks from xapi-back via simple command line interface
xb– list VMs and their last backups, backup, restore, set VMs for scheduled backup ( with help of cron). You’ll not need any other management tools (xe, XenCenter, ….) to make backups. - self-maintaining – xapi-back can be scheduled and run automatically. It maintains backups’ storage, keeps N last backups and removes old backups so it can run unattended for months.
- compact – it’s very small solution so it can run on any machine, only python is needed (it can run easily on minimal Debian install or even on NAS)
- universal – can run on any POSIX system, where python is running ( any Linux, FreeBSD, Solaris …)
- multiple servers – can handle multiple XenServers and server’s pools
Replacing failed disk in Linux RAID1
In this past article I’ve written about setting a sofware based RAID1 on Linux. After less then two years one disks failed, so I had to replace it – here are my experiences with this procedure. Continue reading Replacing failed disk in Linux RAID1
Changing Management Interface on XenServer 6.2
If you are changing management interface on XS (from eth1 to eth2, in my case, because eth1 was not connect to right subnet), be aware that management console must not update routing table for local subnet access appropriately – it keeps the record for previous interface there with same metric value, so actually old record will have preference. So you might end up in situation, where you cannot access local subnet.
You can check with route command:
route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.163.127.32 * 255.255.255.224 U 0 0 0 xenbr1 10.163.127.32 * 255.255.255.224 U 0 0 0 xenbr2 default prague03-lab-co 0.0.0.0 UG 0 0 0 xenbr2
Temporarily it can be fixed by:
route del -net 10.163.127.32 netmask 255.255.255.224 dev xenbr1
However this change is not saved – so it’ll not survive reboot.
Final solution is to use xsconsole and “Network and Management Interface” / “Emergency Network Reset” – this function will reboot host.
Migrating xend virtual machines to xapi platform (XCP/XenServer)
Xen hypervisor has currently 3 tool stacks : xend + xm (which is now deprecated), xl (new low level tool) and xapi+xe. xapi is most advanced and it is used in XenServer (and XCP , but it is now also deprecated because XenServer is now open source). Recently I have been migrating some virtual machines from Xen 4 with xend to XenServer 6.2. Below are details of migrating linux machines to new environment. Continue reading Migrating xend virtual machines to xapi platform (XCP/XenServer)
Not Always PyPy Is Faster
PyPy is an alternative Python interpreter, which is known for it’s speed. However it does not have to be always faster as ‘classic’ Python interpreter (called here CPython). For one small project of mine – PDF Checker – I was testing PyPy hoping to speed up PDF document processing (basically parsing to extract text – pdfminer library is used and document parsing takes majority of time). Below are results from running program for two different files and in CPython interpreter or in PyPy (with JIT and without JIT compilation):
| CPython | PyPy | PyPy with JIT disabled | |
| Small PDF (110kB) | 1.1 s | 2.4 s | 2.5 s |
| Big PDF (996kB) | 16.6 s | 10.9 s | 36.5 s |
Running uWSGI for gevent enabled application
Gevent is a great library that uses greenlets (a Python co-routine library) to enable asynchronous I/O, while providing API that looks like normal synchronous API, so it’s easier to use and understand. The async magic is done automatically by Gevent, which is running an event loop on background and switching between coroutines as necessary.
This approach can be very useful for concurrent applications, which spend a lot of time in waiting for I/O. Like web applications – so Gevent is popular there. For certain type of workloads it can be quite useful – it can enable higher concurrency, while using less resources (greenlet is much lighter then thread or process). Continue reading Running uWSGI for gevent enabled application
Unity – adding unknown applications to Launcher/Dash
It’s described in numerous posts how to add new application to Unity so it’s searchable in Dash. Unity works with .desktop files, which define how applications should be launched from Unity – these files are located in /usr/share/applications (system wide definitions) or ~/.local/share/applications (user specifics application). So if you add well formated .desktop file to any of these locations Unity will be aware of it (may need to restart unity).
Recently I found one more interesting behavior of Unity – if you start unknown application from terminal it will appear in Launcher (in ‘Running applications’ section), Unity even makes some effort to find correct icon for it. Now you can lock it to Launcher (right click and chose ‘Lock to Launcher’ from menu). On background Unity creates new .desktop entry for this program in ~/.local/share/applications, so it can stay locked to Launcher in future. This new .desktop file contains title of the application from window title (in which application is running) executable path and parameters are taken from process properties, even icon path is stored if Unity was able to find one. And when you unlock application from launcher, .desktop file will still remain in users applications – so you can search it in Dash. So this approach can make adding new unknown application easier – just run, then lock and unlock from Launcher and you’ll have new entry in ~/.local/share/application. You can then edit it a bit manually to make it perfect and this is it.
Oracle DB – Compression of LOBs
We have some APEX applications for our colleagues, where people can attach files (usually MS Word documents or PDF files). A straightforward way to implement it was to use a table with BLOB field and connect it with the file item in an upload form. All works well, but as times goes on BLOBs are starting to take significant amount of space. I have been looking for some solution, how to reduce space taken by attached documents and explored Oracle 11g compression capabilities.
For LOBs 11g (Enterprise Edition) provides new feature called SecureFiles, which also enables compression of LOB data. We have migrated our tables to use SecureFiles and saved approximately 50% of space. Continue reading Oracle DB – Compression of LOBs
Why GMail is not changing all server certificates in synch?
I’m accessing my Gmail account from behind HTTPS proxy – it was described is this post. Thunderbird does not support it, for IMAP and SMTP only SOCKS proxy can work. To cope with it I’m using a small local proxy, that redirects any connection via proxy CONNECT method to remote host:port.
This works fine, but in email client I had to set IMAP server as localhost and SMTP server also as localhost. Thunderbird is cautious about it and since both connections are using TLS/SSL then there is a security issue – I’m connecting to localhost, but certificates are for *.gmail.com domain. Luckily Thunderbird enables me to set security exception – it asks me if I’ll allow that certificate for that host address, if I confirm everything works like charm until Gmail changes certificate on servers (which happens about couple time per year or so). Continue reading Why GMail is not changing all server certificates in synch?