Data Lake

.https://cloud.google.com/blog/topics/developers-practitioners/framework-building-configuration-driven-data-lake-using-data-fusion-and-composer

Design components

1. Data Fusion pipelines for data movement

2. Custom pre-ingestion and post-ingestion tasks

3. Configurations to provide inputs to reusable components and tasks

4. Composer DAGs to execute the custom tasks and to call Data Fusion pipelines based on configurations

Usage scenario