Category Archives: Programming

CI/CD Environment for A Smaller Project

Advantages of Continuous Integration (CI) and Continuous Delivery (CD) are obvious even for small projects with few contributors and are easily achievable with help of  free cloud tools – like for instance with mighty combo of Github plus Travis. But what if we want to achieve similarly convenient  environment inside of our private network, available only to our internal teams. Luckily open source is here again to help us with another great tool – GitLab –  GitLab  is a similar platform to GitHub, but the code is open source and we can easily install it in our environment. In this article I’ll summarize my experiences and guidelines how to build convenient environment for a small project  with automatic testing and deployment. Continue reading CI/CD Environment for A Smaller Project

Asynchronous Again – Rewriting ptunnel in Rust

Asynchronous programing model is quite popular for I/0 intensive tasks – it enables you effective use of resources, while maintaining agility of and assuring scalability of the application. I myself used asynchronous programming many times –   in JavaScript (where it’s omnipresent) , Python ( mainly  in asyncio recently, but also bit in twisted, which was one of first network asynchronous libraries I met) and also in OCAML with  lwt or Core Async. The concept is always similar for all implementations –  I/O operations are returning handles to future results – they are called either  Futures, Promises, or Deferred  – and they are returned immediately.  These futures can have functions attached to them, which are executed later, when I/O result becomes available.  Asynchronous  programming is very much about functions, it requires first class functions  and anonymous functions are very useful here, that’s why asynchronous model flourishes in functional languages.  Apart of I/O deferred processing usually there are other utilities for later execution – like timeouts, pausing execution for some time (sleep), tasks synchronization (events, locks). Futures are executed in an “event loop”,   a loop that monitors various events from OS (availability of data from I/O), timers, etc. to execute futures (meaning functions attached to them), when appropriate. It’s also very common to chain futures, executing second one with result of first one , when first one is resolved and result is available and the third one with results from the second one and so on. Apart of this basic scheme languages may provide some syntactic sugar around asynchronous model like await and async keywords in Python or C#, which makes it easier to write the code.

Recently, as I’m progressing in learning of Rust,  I wondered how asynchronous programing is done in Rust. I decided to remake my old project ptunnel (written in Python) into Rust – ptunnel is a program that tunnels arbitrary connection/protocol through HTTPS proxy, so it can be used to connect IMAP, SMTP or SSH through proxy. In the rest of this article I”l share my experiences from this project. Continue reading Asynchronous Again – Rewriting ptunnel in Rust

Splitting Large Audio Books

I’m big fan of audio books.   During past years I’ve been using setup described in this article (libresonic server, android client, audio encoded with opus codec) for audio books listening.  It works well  for me , but it’s best with audio books split to chapters or to parts not longer then 1 hour. However some audio books come in  one large file (m4b format, or  aax proprietary file  from Audible).  To listen to such audio books conveniently I need to split them. Luckily with ffmpeg tool and a bit of bash scripting it is not difficult. Continue reading Splitting Large Audio Books

Secret Sharing Is Caring Too

In todays digital world passwords and other types of secrets are omnipresent and they secure access to various assets dear to our hearts, some of those can have tremendous tangible or moral value. For such assets it’s worth to select really good and strong password, which basically means long and hard to remember. How to ensure ourselves in case of memory failure? We can write it down and lock in secure place, share with trusted person etc., but still there is one point of of failure – secure place can be robbed, that person can betray us. Can cryptography  provide us with better options?  Yes it can with help of method called Secret sharing – we can split secret into n parts – called shared secrets – and distribute them to different places/people. Later we (or someone else) need to collect k (k > 0 and k <= n) shared secret to recover original secret. k is called threshold and it is defined when generating shared secrets – so we for instance generate n=5 shared secrets, but only k=3 will be needed to recover original secret.

I believe you can easily imagine  many other real life scenarios where secret sharing can be useful and for sure it’s used in many applications and systems today. Cryptography provides several algorithms for secure (by design) secret sharing.  Most common is Shamir’s Secret Sharing based on linear algebra approach. There are many tools and libraries for Shamir’s scheme (and further advancements of original algorithm),  you can for instance try ssss, which provides command line tool that you can easily install into your Linux and also there is an online demo. Another family of secret sharing schemes is based on Chinese Reminer Theorem, where especially Asmuth-Bloom scheme is interesting.  I have not seen many implementation for Asmuth-Bloom secret sharing so I created one in Rust. Continue reading Secret Sharing Is Caring Too

Ethereum local playground

In past article I’ve talked generally about blockchain technologies, in this article we will look into Ethereum from user perspective. We will build local playground, where we can test many functions of Ethereum(Ethers transfers, using and writing smart contracts and more) without spending real Ethers (and thus real money). This guide in intended for users with Linux OS. Continue reading Ethereum local playground

In RUST We Trust

Having been programing recently mostly in dynamic untyped languages (aka Python and JavaScript) I though that it would be nice to try something else, bit different –  meaning compiled and statically typed. Last adventures in this area were with OCAML, which I used for few learning projects couple years ago( like this one).  OCAML is very nice language indeed, and learning exercise was very valuable for me (especially getting more accustomed to functional programming style),  but apart  of that learning experience I did not follow it  further (mainly due limited ecosystem of OCAML).

Looking recently to languages and technology survey on Stackoverflow  where Rust is leading the list of most “loved” languages (meaning developers who used the language like it and want to use it for their next projects) with head start on   second one (SmallTalk) .   This caught my attention and looking quickly at Rust site I decided to give it a try.  Below are my first experiences learning this language. Continue reading In RUST We Trust

Run and monitor tasks via WebSocket with ASEXOR

Many modern web applications require more then just displaying data in the browser.  Data may need to be processed and transformed in various ways, which require intensive processing tasks on server side. Such processing is best done asynchronously outside of web application server, as such tasks can be relatively  long running. There are already many existing solutions for asynchronous task scheduling, some of them are quite sophisticated general frameworks like Celery, Kafka, others are build in features of application servers ( like mules and spoolers in uWSGI).  But what if we need something simpler, which can work  directly with Javascript clients and is super simple to use in a project.  Meet asexor – ASynchronous EXecuOR,  a small project of mime. Continue reading Run and monitor tasks via WebSocket with ASEXOR

Comparison of JSON Like Serializations – JSON vs UBJSON vs MessagePack vs CBOR

Recently I’ve been working on some extensions to ASEXOR, adding there direct support for messaging via WebSocket and I use JSON for small messages that travels between client (browser or standalone)  and backend.  Messages looks like these:

I wondered, if choosing different serialization format(s) (similar to JSON, but binary) could bring more efficiency into the application –  considering  both message size and encoding/decoding processing time.  I run small tests  in python 3.5 (CPython and PyPy)  (see tests here on gist) with few established serializers, which can be used as quick replacement for JSON and below are results (updated Dec 2nd 2017 thanks to comment below, as situation changed a bit with new libraries versions): Continue reading Comparison of JSON Like Serializations – JSON vs UBJSON vs MessagePack vs CBOR