In this talk I will share some of the lessons learnt while building a data product in the advertising domain to improve business decisions and leverage data for new revenue streams.

For large datasets with multiple dimensions, summaries computed for  each dimension can be quickly combined to obtain an accurate summary of various combinations of the dimensions (union, intersection, etc.). Summarization of data can be done in a fully distributed manner, by partitioning the data arbitrarily across many nodes, summarizing each partition, and  combining the results.

We will explore  some of the approaches and techniques for the summarisation like - hyperloglog, kth minimal value etc. We will explain how we solved this in the context of a real world problem using Data Sketches, an open source library that provides summarisation as  unique counts, quantiles, frequent items etc.

When dealing with massive datasets, as well as highly partitioned/fragmented data, it inherently has a long-tail distribution across all the fragments. In the above mentioned case study, the data pipeline ingests terabytes of data every week and if we were to create a sketch for every dimension-combination, we would end up with millions of fragments, each with its own sketch. We will explore methods to address the system level storage as well as compute challenges, as well as achieve lower error rates.