DAPCOM Data Services is a spin-off company participated by the University of Barcelona (UB) and the Technical University of Catalonia (UPC). We provide software engineering solutions and data compression strategies, including our own implementations, for both general and specific cases and sectors.
- Esteve Terrades
Currently a point has been reached where data compression is mandatory in most cases and scenarios when dealing with huge volumes of data in the so-called Big Data systems and universe. However, its importance is often forgotten when designing the data flow architecture and infrastructure. In most cases it is simply assumed that there will be some existing solution that will fit our needs. Indeed, there are interesting solutions, either universal or case specific, but they use to offer non-optimal compression efficiency or their computational cost is frequently too high.
On‐ground data compression systems are often based on complex algorithms, typically based on some kind of dictionary compression combined with statistically analysis. These algorithms typically require powerful processors or long compression times (or sometimes both). Our solution, FAPEC (Fully Adapted Prediction Error Coder), is based on an entropy coding that implements an adaptive layer that performs a smart and quick statistical analysis on small blocks of data. The benefit of our algorithm is its very quick operation and its capability to adapt to outliers in the data (values outside the typical statistical distribution).
FAPEC data compressor was developed within the frame of a space mission (ESA Gaia), taking into account the extreme requirements on ratios and speeds that we find in satellite payloads. We intend to bring these robustness and performance of our solution into ground‐based software systems.