Mybookshelf2 (MBS2) is basically in beta stage and provides all functionality available in previous version and more. As I went through programing the new application using several new technologies, I’ve realized few things, which I’d like to share here.
Web client development’s got quite complex
- NodeJS – to run all required tools
- NPM – to install tools and required packages
- Babel transpiler( compiler that compiles JS to JS) to compile next generation JS (or TypeScript) into ES5 code understandable by current and legacy browsers
- Gulp – a build tool
- BrowserSync – to run and test front end application
- JSPM – to package the code for the web browsers and load modules in the web browsers
- Karma and Jamine – to run tests
All above was basically needed to to build, test and run my code. On top of that you indeed need UI framework itself and numerous other libraries (like above mentioned jQuery, which is far from being obsolete by new frameworks). I also found that good knowledge of developer tools in browser (Firefox and Chrome) is essential.
Aurelia framework has done good job to hide this tool chain complexity from me, but nevertheless as my project evolved I had to cope directly with some of these tools anyhow (browser packing and JSPM especially gave me quite hard time – I actually made a round trip from JSPM to bare SystemJS, then to WebPack, to finally return back to JSPM, which turned out to be best solution for me after all) .
By no means I’d like to make impression that these tools are not good. Actually they all are pretty good (some of them are truly impressive, for instance BrowserSync, which saved me a lot of time during development). But still all these new tools were additional things I had to learn to be able to develop modern web applications.
As a part of this project I had to learn ES6 and since there are some many changes I’d recommend to read a book about it. I have found this free book quite good.
Is ORM needed for implementation of RESTful API?
On server side MBS2 application is mainly a kind of RESTful API implementation, which is then used by rich SPA JS client. I implemented it pretty conservatively using Flask-RESTful, SQLAlchemy and Flask-SQLAlchemy. SQLAlchemy is pretty impressive package, very powerful (much more then Django ORM), but also quite complex and it took me some time to learn it. And as I was progressing through server API implementation a heretic thought pop up – Why I need all that ORM (Object Relational Mapping) thingy in my API functions. Of course it’s very convenient, but as for advanced functionality of ORM layer like sessions, unit of work, caching etc. I had a little use for it. Usually it was just about grabbing an object or lists of objects and sending them out ( or insert, update, delete of single object). Sometimes even advanced ORM functionality stood in the way of correct implementation ( auto-flushing of object to database too early).
It would be probably OK if there were no additional consequences of using ORM, however as I’ve shown in this article, ORM means significant drop in performace – ORM queries are several times slower (5x- 10x) then direct access using SQL. It’s understandable, considering complex machinery under the hood, but in case of RESTful API, where we are almost not using it, it looks like quite a waste. Actually more compelling problem for me with RESTful API for me was serialization and deserialization of data, where ORM did helped only little (in case of nested objects).
So if ORM is superfluous (in my opinion), what else we can use. My proposal would be to use custom DAL (Data Access Layer), using SQL directly in DBAPI driver (or through excellent SQLAlchemy SQL Expressions Language, which makes construction of SQL statements more general and pythonic). Writing DAL could be bit boring, but in the end probably not much more time consuming then using ORM (considering simple scenarios I have for RESTful API).
I’ve already done something similar for another part of the application – back-end ebooks processing engine, which is responsible for ebooks conversions and metadata extractions. As this program is written as completely asynchronous (with asyncio), there is no possibility to use ORM, only core SQLAlchemy for SQL construction. DAL layer is also asynchronous (using aiopg) and generally works fine.
Actually what I did missed in my project was a good comprehensive library for data management on client side – in the browser. I very quickly looked at BreezeJS, which looks interesting. So on my next project I might try different approach – very simplistic RESTful API (possibly auto-generated from database schema) and some advanced client side data library like Breeze, because now everything happens in the client anyhow.
WAMP and Crossbar.io usage was an overkill for such application
I have written about WAMP protocol in this article. In MBS2 it’s used to send requests to backend processing engine and monitor progress of such tasks. This functionality is implemented in separate package ASEXOR. I have no particular complains about Crossbar.io and used libraries, they work great and they are mature and reliable, however considering the simplistic needs of the application ( call one remote function and subscribe to one updates source) using full WAMP stack is just an overkill. Looking at it now I think more simple communication using WebSocket directly and some simple aiohttp server would be more appropriate. But I have to admit using WAMP and Crossbar.io was a good learning exercise ( as I needed to implement new transport (asyncio WAMP over rawsocket) into autobahn WAMP library).
Python becoming irrelevant as server platform for web applications