Some people also call this an OLAP database, or a big data database or a data warehouse. First, let’s recall there are 2 main types of databases for your data: transactional (OLTP database) and an analytical one (OLAP):
OLAP databases for short, differ from traditional transactional or OLTP databases in the sense that the queries that you would like it to process, mostly touch large parts of the data available to determine trends in the data, do comparative analysis between different periods and such. This is in contrast to OLTP databases where specific records are what you are after: a certain order with orderlines, a specific customer, etc. It depends on the use case you have which database you should choose.
A very well-known example of an open source transactional database that is widely used, is PostgreSQL. Most of the EU cloud providers offer PaaS services to deploy this database product, in which they take care of upgrades, availability and backups.
As said earlier, this is not the case for analytical databases. When we at Xomnia started exploring the area of analytical databases, some well-known products come to mind: Snowflake, Amazon Redshift, Google Bigquery and Databricks. All of which are not supported outside the big three cloud providers, AWS, Azure and GCP.
During our research on EU cloud providers, we stumbled upon Databend (https://www.databend.com/) and it caught our attention among the other alternatives. More on it is coming soon. Want to hear? Sign up for the newsletter!.
Databend offers an extended version of their product, Databend Cloud, as a service in the cloud, but this again has the same problems as the big three cloud providers: US-based companies, with deployment on a choice of those big three specifically.
However, Databend Cloud is based on the open source offering Databend (https://github.com/databendlabs/databend) which makes it easy to self-host an instance with limited capabilities. The differences between the enterprise version and the self-hosted version are described in this comparison (https://www.databend.com/databend-editions-details/).
Databend claims to be a real contender to Snowflake, the product which stood out as a prime example of how to do analytical database functionality the right way. The comparison we do here, though, is against PostgreSQL. Why? Isn't that a database that is not really suitable for analytical queries? The reason is as stated above: PostgreSQL IS available in EU cloud provider platforms and probably a first candidate to do some sort of analytical querying when one wants to deploy in the cloud and have management of the database outsourced to the cloud provider. And yes, there are better analytical databases than PostgreSQL that you can deploy yourselves, but that's mostly the point of this blog-sized comparison: why should you have a look at Databend?
Databend did their own comparison against Snowflake here. Based on what you see in the outcomes below, it is up to you to trust this comparison or not. I think it sticks.
TPC-H is a standardized benchmark for performance testing of analytical queries against a data model specifically from analytics data. It comes in different scale factors of which the version with scale factor 100 is probably the bare minimum to test with nowadays.
So, that's what I did: I essentially ran the scripts from the Snowflake comparison page. I did this on a system with AMD 5800X processor (8 CPU's, 16 cores), 64GB or RAM and some disk space running Fedora Silverblue. I used the Docker deployment description as written on this page. So, in essence, I ran a container for ‘minio’ (the S3 compatible object storage service), and one for Databend itself, for which I allocated 8 CPU's and 32GB of RAM. That should leave plenty of space for the system, including disk caching and such.
The TPC-H query durations are in the table below, as an average of three consecutive runs. The other column depicts the results of a test on PostgreSQL on the same dataset but on an Apple M1 Max (10 cores, 64GB) that I copied from here
TPC-H query | PostgreSQL result in s | Databend results in s | PostgreSQL/Databend ratio |
1 | 103,1 | 21,2 | 4,9 |
2 | 40,2 | 2,0 | 19,7 |
3 | 156,4 | 12,6 | 12,4 |
4 | 201,6 | 4,7 | 42,5 |
5 | 67,5 | 8,5 | 7,9 |
6 | 307,7 | 7,0 | 43,8 |
7 | 152,5 | 10,5 | 14,5 |
8 | 175,9 | 11,1 | 15,8 |
9 | 1264,6 | 18,4 | 68,8 |
10 | 266,4 | 11,3 | 23,6 |
11 | 15,3 | 1,6 | 9,3 |
12 | 93,9 | 8,6 | 11,0 |
13 | 201,4 | 7,1 | 28,4 |
14 | 117,7 | 7,4 | 16,0 |
15 | 627,9 | 7,2 | 87,0 |
16 | 42,6 | 1,2 | 36,3 |
17 | 469,3 | 9,7 | 48,6 |
18 | 700,2 | 12,7 | 55,1 |
19 | 15,0 | 16,3 | 0,9 |
20 | 531,5 | 10,7 | 49,9 |
21 | 199,9 | 27,4 | 7,3 |
22 | 58,1 | 1,5 | 39,7 |
Isn't this comparing apples and oranges? Yes, it is somewhat. But the main point is of course the difference in performance scale. Just look at the ratios! And CPU-wise these CPU's are not that different, the Apple M1 Max has a multi-thread rating of 22149, and the AMD 5800X has one of 27756. Memory-wise, PostgreSQL had its share of memory allocated too, just check the configuration on the github page.
My point is that the software architecture makes the difference here: Databend is multiple order of magnitudes faster for this use case. And don't get me wrong, PostgreSQL is a great database but not for analytical analysis. For OLTP use cases the results will probably be reversed, but that's not where you would apply Databend.
Does it really work that well? Might it be a contender against Snowflake? Could I use it on my EU cloud of choice where I am desperately trying to get some analytical databases up and running as an alternative to Snowflake, Databricks and BigQuery?
I say yes. But there are some drawbacks too: Databend is still very young, but the system looks solid and well designed. My results are in the same order of magnitude as the hot run published here. They differ, though, in the sense that my results are sometimes faster and sometimes slower. Mind you, I did nothing to optimize the performance. I just started those containers and made sure the Databend had a specific number of CPU's and enough memory. Load data and go!
What I essentially would hope for, is that one or more of the EU clouds would build a service offering around Databend, where they support it and arrange some maintenance services around it. So we can more easily deploy Databend, use it, and leave the nitty-gritty infrastructure details to the cloud providers.
That would be great! We will be closely watching how Databend develops at least.
In upcoming blogs we will delve deeper into other services offered by EU cloud providers and how those stand up against the Big Three.
Disclaimer: Xomnia is not affiliated with Databend in any way.
Solutions Architect and EU-Cloud topic lead at Xomnia
We are looking more closely recently at EU cloud providers due to geo-political tensions. See a previous blog for more info. In our research on the capabilities of EU clouds we noticed that the support for so-called analytical databases is mostly la...
We are looking more closely recently at EU cloud providers due to geo-political tensions. See a previous blog for more info. In our research on the capabilities of EU clouds we noticed that the support for so-called analytical databases is mostly la...
A common frustration among expats working at Xomnia is the occasional struggle with the communication style of the Dutch government. When a colleague accidentally missed a deadline on a fine which turned out to be a fine for missing a fine, this “fi...
