History

[Torflow] was the first Tor bandwidth scanner, started in 2011. Over time, it has become more difficult to install and to maintain, because the libraries it was built with are no longer maintained.

In 2016, [bwscanner] was started using more modern and maintained libraries, like [txtorcon], but it was discontinued.

In 2018, we started to develop “Simple Bandwidth Scanner” ([sbws]), for the same reason, using Python 3 and [stem] library.

Soon we started to find some difficulties with the initial sbws design, as:

  • confusing names for attributes, methods, classes, variables, functions

  • functions and methods very long (eg. main_loop)

  • repeated code in different functions

  • code that gets unnecessarily call twice

  • using threads instead of asynchronous programming or at least using a more modern and siple library than [multiprocessing.dummy] as [concurrent.futures].

  • many arguments passed to functions and many passed succesively to several other functions, methods and objects where class encapsulation would avoid that (eg. args, conf), which has made difficult to create tests.

  • passing a dynamic network objects to functions and methods (the tor controller) instead of the routers status or descriptors to facilitate tests.

  • converting json to dictionaries and different types of objects and converting back to json

  • parsing files that are not json but have json in each line and writing back to them

  • having to make statistics calculations on a big number of json objects (measurements) like means, count x attributes, compare objects attributes with dates, find a last object, etc. (mostly in the generator). This has also made difficult to add tests.

Specially the last 3 issues can be solved with a database management system, which is easy to maintain thanks to modern [Object-relationalMapping] (ORM).

The two most common ORMs frameworks in Python are [Django] and [SQLAlchemy]. While Django is oriented to create Web applications and it might sound crazy to use it in a non-Web application, the Web forms and templates can be ignored. It also has a great admin Web interface that it’s very helpful while developing. On the other hand, SQLAlchemy already supports python asyncio, but Django is starting to support it.