# ETL Developer Guide

Developer guide for getting started with DataSuite ETL workflows. This comprehensive documentation will guide you through understanding, setting up, and implementing ETL data pipelines using the DataSuite platform.

## What You'll Learn

* **System Architecture**: Understanding the complete DataSuite ETL ecosystem
* **Environment Setup**: Getting your local development environment running
* **Pipeline Development**: Building ETL workflows with LogStash and DBT
* **Testing & Validation**: Ensuring data quality and pipeline reliability

## Prerequisites

* Docker and Docker Compose installed
* Basic understanding of SQL and data concepts
* Familiarity with command line operations
* 8GB+ RAM recommended for full stack

## Navigation Guide

### 🏗️ Architecture & Design

Start here to understand the system before implementation.

* [**System Overview**](/data-platform/index/architecture/overview.md) - High-level architecture and data flow
* [**Component Details**](/data-platform/index/architecture/components.md) - Deep dive into each system component
* [**Data Flow**](/data-platform/index/architecture/data-flow.md) - How data moves through the pipeline

### ⚙️ Environment Setup

Get your development environment up and running.

* [**Prerequisites**](/data-platform/index/setup/prerequisites.md) - System requirements and dependencies
* [**Quick Start (Docker Compose)**](/data-platform/index/setup/docker-compose-setup.md) - Recommended for beginners
* [**Advanced Setup (Individual Containers)**](/data-platform/index/setup/individual-containers.md) - For custom configurations
* [**Service Verification**](/data-platform/index/setup/verification.md) - Confirm your installation is working

### 🛠️ Development Workflow

Build and customize your ETL pipelines.

* [**LogStash Configuration**](/data-platform/index/development/logstash-configuration.md) - Data ingestion setup
* [**DBT Getting Started**](/data-platform/index/development/dbt-getting-started.md) - Environment and project setup
* [**DBT Modeling Guide**](/data-platform/index/development/dbt-modeling-guide.md) - Creating dimensional models
* [**Testing & Validation**](/data-platform/index/development/testing-validation.md) - Data quality assurance

### 🚨 Troubleshooting

Resolve common issues and debug problems.

* [**Common Issues**](/data-platform/index/troubleshooting/common-issues.md) - Frequently encountered problems
* [**Debugging Guide**](/data-platform/index/troubleshooting/debugging-guide.md) - Systematic problem resolution

## Quick Start Path

**New to DataSuite?** Follow this recommended learning path:

1. 📖 Read [System Overview](/data-platform/index/architecture/overview.md) to understand the big picture
2. 🚀 Complete [Quick Start Setup](/data-platform/index/setup/docker-compose-setup.md) to get running fast
3. ✅ Run [Service Verification](/data-platform/index/setup/verification.md) to confirm everything works
4. 🔧 Follow [DBT Getting Started](/data-platform/index/development/dbt-getting-started.md) for your first pipeline
5. 📊 Build your first model with [DBT Modeling Guide](/data-platform/index/development/dbt-modeling-guide.md)

## Expected Outcomes

After completing this guide, you will be able to:

* ✅ Set up a complete DataSuite ETL development environment
* ✅ Configure LogStash for data ingestion from various sources
* ✅ Build dimensional data models using DBT
* ✅ Test and validate your data transformations
* ✅ Troubleshoot common pipeline issues

## Getting Help

* 🐛 **Issues**: Check [troubleshooting guides](https://github.com/FPT-Data-Suite/documents/blob/master/docs/troubleshooting/README.md) first
* 📚 **Documentation**: All guides include detailed examples and explanations
* 🔍 **Search**: Use your browser's search function to find specific topics

***

**Ready to get started?** Begin with the [System Overview](/data-platform/index/architecture/overview.md) to understand how all the pieces fit together.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.datasuite.vn/data-platform/index.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.