How much better is the thread pool?

Is thread pool worth to consider for my project?   I was  looking for some opinions around the net and as usual they  differ and most common wisdom is it matters. Generally it’s “known” that creating and tearing down thread is “significant” overhead, so if you have a lot of small tasks thread pool is much better solution then spawning new thread for each task.  But what is significant overhead?  According to what I read time to create thread on Linux should be about 10μs (which does not look as too much to me) and app. 2MB of memory allocated for stack (configurable).   I was considering thread pool in context of audioserve project, where I started with simplest possible solution (e.g. spawning individual threads ) and was wondering how much I’m loosing by not using thread pool. So I implemented simple thread pool (as learning exercise – long term audioserve solution should use tokio-threadpool)  and add it to audioserve.  In the remainder of this short article I’d like to share my findings and  roughly quantify benefits of thread pool for such small project. Continue reading How much better is the thread pool?

audioserve Android Client Early Beta Is Available

So finally there is something. I’m using it myself now to listen to audiobooks and it have almost all functionality I wanted it to have. It might be still bit unstable and few things is not well behaving (keep up long time in paused state, navigation between notifications and activities is still bit messy and few more issues), but generally it works. Continue reading audioserve Android Client Early Beta Is Available

Stupidly Complex or Completely Stupid – Android Development

As part of audioserve project I decided to create an Android client. To make this exercise more interesting I decided to write it in Kotlin language (see previous article about Kotlin language).  This client should provide roughly similar interface as web client (see this article for web client interface demo), but I hoped to achieve much better caching features, which will enable to play smoothly even when connectivity is temporarily lost ( for half an  hour, an hour) and possibility to download and play audiobooks completely offline.  Further in this article I’d like to share my experiences with Android development, as beginner in this area.  And spoiler is I do not like it. Continue reading Stupidly Complex or Completely Stupid – Android Development

From Russia with Love – Kotlin Language

One may think that we have already more then enough programming languages. But community thinks differently and new languages keeps popping up and thanks for that, because some  bring cool innovative features, others focus on more streamlined and effective development, but all of them contribute to the evolution of the IT industry. Because as in the nature progress comes through the variety of species and competition between them.  As I have written in my past article I looked recently into Rust language – very interesting language, which comes with novel approach to memory management without garbage collection, but still assuring high security of the program.  Nowadays I met another member of ever-growing happy family of programing languages – Kotlin and I’d like to share my first experiences in this article. Continue reading From Russia with Love – Kotlin Language

Audioserve Audiobooks Server – Stupidly Simple or Simply Stupid?

If you read some of my previous articles you know that I’m quite fond of audiobooks.   In past I was looking for good media server, which supports audiobooks  and ended with  airsonic (a subsonic clone).   The problem with majority of media servers is that they rely totally on audio tags, which are often messed in audiobooks. Also many of “advanced” functionalities are not applicable to audiobooks (random play, shuffle tracks, moods, etc.)  I’m bit of old school so I rely more on reasonable directory structure and do not want to mess with tags for every audiobook I download. I also think that for personal collection I do not need likes, favorites, sharing and similar stuff,  as I can easily remember which books are good and which I have listened or want to listen, but I do need few things, which are usually missing – like bookmarks. Interesting function is to continue listen on a device, when I left  on previous device, but since I basically listen only on my mobile phone, it does not seems to be critical. So ideal audiobooks server actually requires much less functionality than today’s media servers provide.  As I’m progressing with Rust language I decided two weeks ago to create simple audio streaming server adhering to KISS principle – Keep It Simple, Stupid, – Result of this exercise is an  application that provides minimum viable functionalities for streaming and listening of audiobooks – it’s called Audioserve.  In this article I’ll show it’s basic design and demo current application in a video. Continue reading Audioserve Audiobooks Server – Stupidly Simple or Simply Stupid?

The Different Approach to A Parser – Parser Combinators

Some time ago I’ve looked into building parser for simple logical language used by imap_detach Tool. I used there parsimonious module which uses PEG grammars. Recently I learned about another parsing technique – Parser Combinators. The idea comes from functional programming and is about combining (with help of higher order functions – so called combinators) simple parsing functions, which parse primitive tokens, into more complex parsing functions, which parse parts of text and further combine those into more complex ones, ending finally with one function, which parses all given data. I first met with parser combinators in Rust library nom. Here parsers and  combinators are express as macros, which is convenient on one side (concise syntax), but can lead to pretty cryptic error messages and one cannot rely much on editor’s help with auto-completions, context help etc..  I used nom to build very simple URI parser.  I was wondering also if parser combinators are available for Python and how they would compare with other approaches – like above mentioned PEG grammars. Continue reading The Different Approach to A Parser – Parser Combinators

CI/CD Environment for A Smaller Project

Advantages of Continuous Integration (CI) and Continuous Delivery (CD) are obvious even for small projects with few contributors and are easily achievable with help of  free cloud tools – like for instance with mighty combo of Github plus Travis. But what if we want to achieve similarly convenient  environment inside of our private network, available only to our internal teams. Luckily open source is here again to help us with another great tool – GitLab –  GitLab  is a similar platform to GitHub, but the code is open source and we can easily install it in our environment. In this article I’ll summarize my experiences and guidelines how to build convenient environment for a small project  with automatic testing and deployment. Continue reading CI/CD Environment for A Smaller Project

Asynchronous Again – Rewriting ptunnel in Rust

Asynchronous programing model is quite popular for I/0 intensive tasks – it enables you effective use of resources, while maintaining agility of and assuring scalability of the application. I myself used asynchronous programming many times –   in JavaScript (where it’s omnipresent) , Python ( mainly  in asyncio recently, but also bit in twisted, which was one of first network asynchronous libraries I met) and also in OCAML with  lwt or Core Async. The concept is always similar for all implementations –  I/O operations are returning handles to future results – they are called either  Futures, Promises, or Deferred  – and they are returned immediately.  These futures can have functions attached to them, which are executed later, when I/O result becomes available.  Asynchronous  programming is very much about functions, it requires first class functions  and anonymous functions are very useful here, that’s why asynchronous model flourishes in functional languages.  Apart of I/O deferred processing usually there are other utilities for later execution – like timeouts, pausing execution for some time (sleep), tasks synchronization (events, locks). Futures are executed in an “event loop”,   a loop that monitors various events from OS (availability of data from I/O), timers, etc. to execute futures (meaning functions attached to them), when appropriate. It’s also very common to chain futures, executing second one with result of first one , when first one is resolved and result is available and the third one with results from the second one and so on. Apart of this basic scheme languages may provide some syntactic sugar around asynchronous model like await and async keywords in Python or C#, which makes it easier to write the code.

Recently, as I’m progressing in learning of Rust,  I wondered how asynchronous programing is done in Rust. I decided to remake my old project ptunnel (written in Python) into Rust – ptunnel is a program that tunnels arbitrary connection/protocol through HTTPS proxy, so it can be used to connect IMAP, SMTP or SSH through proxy. In the rest of this article I”l share my experiences from this project. Continue reading Asynchronous Again – Rewriting ptunnel in Rust

MyBookshelf2 Beta2

MyBookshelf2 – ebooks management and sharing solution moved to next version – Beta 2 – apart of few small fixes, the main change is an internal revamp to make it compatible with latest Asexor and thus removing dependence on and Autobahn library.  It makes deployment of MyBookshelf2 easier and removes components that did not add much value to the solution, just made it bit overcomplicated.  Some effort was given to provide easy implementation in Docker – now there is a script that will guide you through two pre-configured scenarios – development (local code, monitoring of code changes) and stage (code in a volume, JS client built and packed, app server running behind nginx proxy with SSL termination, nginx serving static content).

Check latest code on github. To give it a try just clone repo and run script in deploy directory (assuming you have Docker installed).

Splitting Large Audio Books

I’m big fan of audio books.   During past years I’ve been using setup described in this article (libresonic server, android client, audio encoded with opus codec) for audio books listening.  It works well  for me , but it’s best with audio books split to chapters or to parts not longer then 1 hour. However some audio books come in  one large file (m4b format, or  aax proprietary file  from Audible).  To listen to such audio books conveniently I need to split them. Luckily with ffmpeg tool and a bit of bash scripting it is not difficult. Continue reading Splitting Large Audio Books

My Digital Bits And Pieces