Learn how we ingest, process, and visualize data at a price point you never thought was possible.
BI Software is notoriously expensive. Today there are a lot of great tools to choose from, like Tableau, IBM Cognos, Microsoft PowerBI, etc. For a customer, we implemented Amazon QuickSight, and we are surprised by the low price point Amazon offers.
You only pay $18 per user in Amazon QuickSight for the enterprise edition per month. Every user gives you 10GB of SPICE capacity that you can use to load your data in. SPICE stands for Super-fast, Parallel, In-memory Calculation Engine. This is where your data will be stored in, to be able to visualize your data super fast.
To be able to ingest the data, we used API Gateway and Amazon Firehose, which can deliver the data to S3 in parquet format. Once in S3, you can run queries using Amazon Athena. Amazon QuickSight supports Athena, so you just need to connect QuickSight to Athena and you can load the data in SPICE.
The great thing about this setup is the low price at which this can be achieved:
- API gateway: $1.00 per million events
- Firehose: $0.029 per GB
- S3 Storage: $0.023 per GB (+ S3 API requests)
- Athena: $5 / TB processed
- QuickSight: $18 per user
- Let’s say in one month you have 10 million events and your event size is 1kb, you only pay approximately $25 to have this all processed, stored, and visualized.
In our actual setup the data is also partitioned to speed up the Amazon Athena queries (and reduce costs). How we do that we'll explain in a next blog post.
