Category Archives: Admin

Accesing Oracle from Python (with proper unicode support)

It’s not obvious to set it right, so I’m putting some notes here:

Installation is described here.
Few comments:

  • ORACLE_HOME is needed just for installation
  • If you add client library path to /etc/ld.so.conf.d/oracle.conf   and update ldconfig, you don’t need to export modified LD_LIBRARY_PATH
  • when you install Oracle client library and set environment,  you can install cx_oracle also via pip install cx_Oracle

The crucial step not mentioned in the installation guide is  to set NLS_LANG environment variable – this should be in the environment of your python program using cx_oracle.  So for instance for Flask+SQLAlchemy you can have:

Without this variable oracle client is using 7bits ASCII! So any unicode character will raise “UnicodeEncodeError: ‘ascii’ codec can’t encode character” error.

Simple statistics from nginx access logs

I required some simple statistics (selected page visits per day)  from web-server logs.   I looked at some web log analyzer packages like AWStats, but it looked to me like as an overkill in my case – I’d probably spent more time to trying make it work then putting together some small script. So here it is – a simple bash script that will take all available access logs (by default on Debian nginx is using logrotate to rotate logs daily and keeps 52 daily logs, old logs are gzipped) and calculate page visits for certain request pattern: Continue reading Simple statistics from nginx access logs

Poor man’s backup for XenServer

I’m running several XenServer hosts and wanted to provide some basic backup of VMs. I decided to use USB disk – XenServer 6.2  provides great support for external disks.   I was looking around for some simple free tool for backup (to backup several VMs from different servers in scheduled batches), but did not find anything suitable (simple scripts were not flexible enough, bigger solutions were overkill in my case) – so I created my own solution xapi-back

My setup is:

  • I created small Debian VM and attached USB disk to it (in XS 6.2 this external disk will stay connected to VM after  VM or host reboot)
  • Installed xapi-back
  • Created special user for backup
  • Scheduled VM backups with cron

Main advantages of xapi-back  compared to other similar solutions:

  • easy to install – just download and run python setup.py
  • easy to configure –   just one simple configuration file with details of xen servers and some basic backup parameters
  • self-contained  – does not need xe or other tools (as many other solutions) and can run on any computer ( not only in xenserver Dom0 as some scripts,  generally I think it’s not good practice to run backups in Dom0, better is to have it separately).
  • complete – you can do all basic tasks from xapi-back via simple command line interface xb –  list VMs and their last backups,  backup, restore,  set VMs for scheduled backup ( with help of cron). You’ll not need any other management tools (xe, XenCenter, ….) to make backups.
  • self-maintaining – xapi-back can be scheduled and run automatically. It maintains backups’ storage, keeps N last backups and removes old backups so it can run unattended for months.
  • compact – it’s very small solution so it can run on any machine, only python  is needed (it can run easily on minimal Debian install or even on NAS)
  • universal – can run on any POSIX system, where python is running ( any Linux, FreeBSD, Solaris …)
  • multiple servers – can handle multiple XenServers and server’s pools

Changing Management Interface on XenServer 6.2

If you are changing management interface on XS (from eth1 to eth2, in my case, because eth1 was not connect to right subnet), be aware that management console must not update routing table for local subnet access appropriately – it keeps the record for previous interface there with same metric value, so actually old record will have preference.   So you might end up in situation, where you cannot access local subnet.

You can check with route command:

Temporarily it can be fixed by:

However this change is not saved – so it’ll not survive reboot.

Final solution is to use xsconsole and “Network and Management Interface” /  “Emergency Network Reset” –  this function will reboot host.

Migrating xend virtual machines to xapi platform (XCP/XenServer)

Xen hypervisor has currently 3 tool stacks : xend + xm (which is now deprecated),   xl (new low level tool) and xapi+xe.   xapi is most advanced and it is used in XenServer (and XCP , but it is now also deprecated because XenServer is now open source).  Recently I have been migrating some virtual machines from Xen 4 with xend to XenServer  6.2. Below are details of migrating linux machines to new environment. Continue reading Migrating xend virtual machines to xapi platform (XCP/XenServer)

Not Always PyPy Is Faster

PyPy is an alternative Python interpreter, which is known for it’s speed.  However it does not have to be always faster as ‘classic’ Python interpreter (called here CPython). For one small project of mine – PDF Checker – I was testing PyPy hoping to speed up PDF document processing (basically parsing to extract text – pdfminer library is used and document parsing takes majority of time).  Below are results from running program for two different files and in CPython interpreter or in PyPy (with JIT and without JIT compilation):

CPython PyPy PyPy with JIT disabled
Small PDF (110kB)  1.1 s  2.4 s  2.5 s
Big PDF (996kB)  16.6 s  10.9 s  36.5 s

Continue reading Not Always PyPy Is Faster

Running uWSGI for gevent enabled application

Gevent is a great library that uses greenlets (a Python co-routine library) to enable asynchronous I/O, while providing  API that looks like normal synchronous API, so it’s  easier to use and understand.  The async magic is done automatically by Gevent, which is running an event loop on background and switching between coroutines as necessary.

This approach can be very useful for concurrent applications, which spend a lot of time in waiting for I/O.  Like web applications – so Gevent is popular there.  For certain type of workloads it can be quite useful – it can enable higher concurrency,  while using less resources (greenlet is much lighter then thread or  process). Continue reading Running uWSGI for gevent enabled application

Unity – adding unknown applications to Launcher/Dash

It’s described in numerous posts how to add new application to Unity so it’s searchable in Dash.   Unity works with .desktop files, which define how applications should be launched from Unity – these files are located in /usr/share/applications (system wide definitions) or ~/.local/share/applications (user specifics application). So if you add well formated .desktop file to any of these locations Unity will be aware of it (may need to restart unity).

Recently I found one more interesting behavior of Unity – if you start unknown application from terminal it will appear in Launcher (in ‘Running applications’ section), Unity even makes some effort to find correct icon for it.  Now you can lock it to Launcher (right click and chose ‘Lock to Launcher’ from menu).   On background Unity creates new .desktop entry for this program in ~/.local/share/applications, so it can stay locked to Launcher in future.   This new .desktop file contains title of the application from window title (in which application is running) executable path and parameters are taken from process properties, even icon path is stored if Unity was able to find one.  And when you unlock application from launcher, .desktop file will still remain in users applications – so you can search it in Dash.   So this approach can make adding new unknown application easier – just run, then lock and unlock from Launcher and you’ll have new entry in ~/.local/share/application. You can then edit it a bit manually to make it perfect and this is it.