DBToaster
About

DBToaster creates query engines for embedding into applications that require real-time, low-latency data processing and monitoring capabilities.

DBToaster-generated engines are optimized for long-lived queries, where query results must be kept up-to-date with rapidly changing input data. Using database terminology, DBToaster engines maintain in-memory materialized views. Our performance claims refer to the speed at which DBToaster engines refresh views as the input data changes.

Applications include:

  • Algorithmic Trading
  • Real-time Data Warehousing
  • Network/Cluster Monitoring
  • Clickstream Analysis
  • and more...

The DBToaster compiler accepts queries written in SQL, and generates query engine code that can be incorporated directly into any C++ or Scala project (with support for more languages on the way). DBToaster-generated engines use each platform's native collection types, making integration with existing projects a breeze.

DBToaster at EcoCloud gives a high-level overview of the DBToaster project.

Is DBToaster right for you?

We do not claim that DBToaster is the right solution for everyone, but you should check out DBToaster if you need to

  • maintain materialized views of complex SQL queries,
  • read these views and care about very high refresh rates / low refresh latencies,
  • work with standing (aka continuous) rather than ad-hoc queries, i.e. you want to monitor the changing result of a given query over time, as the data changes, and
  • do not work with extremely large datasets (this is a temporary restriction until we release our parallel/secondary storage runtimes).

DBToaster may be also right for you even if you do not care about low view refresh latencies: DBToaster turns a set of queries into efficient specialized code for processing just these queries. DBToaster generates code that you can link into your applications. No further software (such as a separate database server or CEP engine) is required. Thus DBToaster is a very lightweight way of including fixed (parameterized) SQL queries in your applications.

Here is a white paper that may help you decide whether to try out DBToaster.

How can DBToaster be so fast?

Traditional relational databases are slow because they are designed to support arbitrary hand-written queries. Nowadays though, few people execute queries directly. Most queries are generated automatically based on templates. DBToaster custom-tailors each engine it creates to the needs of a specific application. This engine supports only query processing functionality that the application requires, avoiding the overhead of supporting unnecessary features.

DBToaster also employs an innovative technique that exploits incrementality to efficiently maintain query results in real-time as data changes. As a consequence, DBToaster-generated engines provide extremely low-latency access to query results, and efficiently support monitoring of result values.

DBToaster-generated code is typically by a factor of 3-4 orders of magnitude faster than existing state of the art data-management systems when we measure the time it takes to refresh a view given an update to the base data.