Context

1. Internal data pipeline is around for about 3 years of rather hectic evolution, which was adjacent to the evolution of external data pipeline
2. Over the years, there were a couple of additions, which were later no longer used
3. As a result, there is a bunch of output files (parquets, jsons) which are no longer consumed, and similarly, potentially services / their components, which are no more used
4. On the code level, although there is a common similarity with external pipeline, code bases are different

Goals

• Strip off unused output files / services / components
• Unify code base with external data pipeline

Why

Although the maintenance of internal data pipeline is rather low, this will enable us to better maintain internal data pipeline and for the future, internal data pipeline will be in a shape for future unification with external data pipeline. We expect, that when this feature is done, a new one for unification of both pipeline will follow as a logical step, which will ensure only one environment to be under development and operations (not two like these days)