data

The Volmanac Data infrastructure is entirely hosted using AWS. As you can see from the picture above, there isn’t anything too complex going on with backend infrastructure - we tried to keep it as simple as possible. The most surprising fact may be that we are using ruby/rails, as opposed to python or a faster multi-threaded language such as java. There is no correct answer, but for Volmanac Rails and ActiveRecord as an ORM fit our needs given:

  1. Simplicity
  2. Data accuracy / validity is more important than write speed
  3. Separate Models - We have thousands of models/tables that generally are separate from one another as they refer to different investment ideas or data and therefore do not share any primary keys.
  4. Limited need for batch updates - If this changes in the future we will have no problem additionally using grails/hibernate.

The Volmanac Data Rails app is used by us to view and monitor data that we are ingesting from public data sources and third parties. We cache expensive queries on the model layer with Redis and other queries are made from the read DB. This app does ingest data from agents in the making reports, but the bulk of the work is done by the worker tier - these two applications share models via a rails engine.

While we do have some branches off of this stack, such as atomic counters with lambda / dynamoDB and a very basic neural network for one trade we are monitoring which we feel like we have enough data for and a good use case already, we really have made an effort to keep the data stack as simple as possible. And while the complexity of the data stack will ultimately be dictated by what data we collect and monitor, we feel strongly that there is currently so much low hanging fruit in this space that we can be very targeted on smaller/relevant data sets and not get caught up building out a large EMR backend for which we currently have a very limited need.