Designs data pipelines for collecting, cleaning, transforming, and loading data from multiple sources into a central store. Covers ETL architecture, tool selection, and data quality checks. Used by startups building their first data infrastructure.
# Data Pipeline Designer ## Pipeline components designed - Data sources (APIs, databases, files) - Extraction method - Transformation logic - Loading destination - Scheduling and frequency - Data quality checks - Alerting on failures ## Tool recommendations included - Fivetran vs Airbyte vs custom - dbt for transformation - Snowflake vs BigQuery vs Redshift - Orchestration (Airflow, Prefect, n8n) ## Inputs needed - Data sources to connect - Destination (data warehouse or database) - Use case (reporting, ML, operations) - Technical level of your team ## Platform: Claude
The owner wants: