Tag Archives: asyncio

Run and monitor tasks via WebSocket with ASEXOR

Many modern web applications require more then just displaying data in the browser.  Data may need to be processed and transformed in various ways, which require intensive processing tasks on server side. Such processing is best done asynchronously outside of web application server, as such tasks can be relatively  long running. There are already many existing solutions for asynchronous task scheduling, some of them are quite sophisticated general frameworks like Celery, Kafka, others are build in features of application servers ( like mules and spoolers in uWSGI).  But what if we need something simpler, which can work  directly with Javascript clients and is super simple to use in a project.  Meet asexor – ASynchronous EXecuOR,  a small project of mime. Continue reading Run and monitor tasks via WebSocket with ASEXOR

Asyncio Proxy for Blocking Functions

File operations and other IO operations can block asyncio loop and  unfortunately  python does not support true asynchronous disk operations (mainly due to problematic state of async disk IO in underlying os – aka linux – special library is need for true asynchronous disk operations  so normally select (or other IO event library) always reports file as ready to read and write and thus file IO operations block). Current solution is to run such operations in thread pool executor. There is asyncio wrapper library for file object – aiofiles, but there are also many blocking functions in other python modules – like os, shutil etc.  We can easily write wrappers for such methods, but it can be annoying and time consuming if we use many of such methods.   What about to write a generic proxy, which will assure that methods are executed in thread pool and use this proxy for all potentially blocking methods within the module. Continue reading Asyncio Proxy for Blocking Functions

WAMP Is WebSocket on Steroids

If you look for WAMP abbreviation over Internet, you will probably find that WAMP = Windows + Apache + MySQL + PHP – which was popular web stack some time ago (but who wants to run web server on Windows today?  And other components  now also have  viable alternatives).   But in this article I’d like to talk about WAMP = Web Application Messaging Protocol.  WAMP is available  as WebSocket subprotocol, but also can work on plain TCP or Unix domain sockets. Continue reading WAMP Is WebSocket on Steroids

Functional Fun with Asyncio and Monads

Python 3.4+ provides excellent Asyncio library for asynchronous tasks scheduling and asynchronous I/O operations.   It’s similar to gevent, but here tasks are implemented by generator based coroutines.  Asynchronous I/O is useful for higher I/O loads, where it usually achieves better performance and scalability then other approaches (threads, processes). About a year ago I played with OCaml, where light weight threads/ coroutines and asynchronous I/O  approaches  are also very popular (Ocaml has same limitation for threading as Python – a global lock) and there were two great libraries – lwt and core async.  Both libraries use monads as a programming style to work with asynchronous tasks. In this article we will try to implement something similar on basis of asyncio library. While our solution will  probably not provide “pure” monads it’ll still be fun and we’ll learn something about asyncio. Continue reading Functional Fun with Asyncio and Monads