> For the complete documentation index, see [llms.txt](https://docs.datasuite.vn/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasuite.vn/data-platform/index.md).

# ETL Developer Guide

Developer guide for getting started with DataSuite ETL workflows. This comprehensive documentation will guide you through understanding, setting up, and implementing ETL data pipelines using the DataSuite platform.

## What You'll Learn

* **System Architecture**: Understanding the complete DataSuite ETL ecosystem
* **Environment Setup**: Getting your local development environment running
* **Pipeline Development**: Building ETL workflows with LogStash and DBT
* **Testing & Validation**: Ensuring data quality and pipeline reliability

## Prerequisites

* Docker and Docker Compose installed
* Basic understanding of SQL and data concepts
* Familiarity with command line operations
* 8GB+ RAM recommended for full stack

## Navigation Guide

### 🏗️ Architecture & Design

Start here to understand the system before implementation.

* [**System Overview**](/data-platform/index/architecture/overview.md) - High-level architecture and data flow
* [**Component Details**](/data-platform/index/architecture/components.md) - Deep dive into each system component
* [**Data Flow**](/data-platform/index/architecture/data-flow.md) - How data moves through the pipeline

### ⚙️ Environment Setup

Get your development environment up and running.

* [**Prerequisites**](/data-platform/index/setup/prerequisites.md) - System requirements and dependencies
* [**Quick Start (Docker Compose)**](/data-platform/index/setup/docker-compose-setup.md) - Recommended for beginners
* [**Advanced Setup (Individual Containers)**](/data-platform/index/setup/individual-containers.md) - For custom configurations
* [**Service Verification**](/data-platform/index/setup/verification.md) - Confirm your installation is working

### 🛠️ Development Workflow

Build and customize your ETL pipelines.

* [**LogStash Configuration**](/data-platform/index/development/logstash-configuration.md) - Data ingestion setup
* [**DBT Getting Started**](/data-platform/index/development/dbt-getting-started.md) - Environment and project setup
* [**DBT Modeling Guide**](/data-platform/index/development/dbt-modeling-guide.md) - Creating dimensional models
* [**Testing & Validation**](/data-platform/index/development/testing-validation.md) - Data quality assurance

### 🚨 Troubleshooting

Resolve common issues and debug problems.

* [**Common Issues**](/data-platform/index/troubleshooting/common-issues.md) - Frequently encountered problems
* [**Debugging Guide**](/data-platform/index/troubleshooting/debugging-guide.md) - Systematic problem resolution

## Quick Start Path

**New to DataSuite?** Follow this recommended learning path:

1. 📖 Read [System Overview](/data-platform/index/architecture/overview.md) to understand the big picture
2. 🚀 Complete [Quick Start Setup](/data-platform/index/setup/docker-compose-setup.md) to get running fast
3. ✅ Run [Service Verification](/data-platform/index/setup/verification.md) to confirm everything works
4. 🔧 Follow [DBT Getting Started](/data-platform/index/development/dbt-getting-started.md) for your first pipeline
5. 📊 Build your first model with [DBT Modeling Guide](/data-platform/index/development/dbt-modeling-guide.md)

## Expected Outcomes

After completing this guide, you will be able to:

* ✅ Set up a complete DataSuite ETL development environment
* ✅ Configure LogStash for data ingestion from various sources
* ✅ Build dimensional data models using DBT
* ✅ Test and validate your data transformations
* ✅ Troubleshoot common pipeline issues

## Getting Help

* 🐛 **Issues**: Check [troubleshooting guides](https://github.com/FPT-Data-Suite/documents/blob/master/docs/troubleshooting/README.md) first
* 📚 **Documentation**: All guides include detailed examples and explanations
* 🔍 **Search**: Use your browser's search function to find specific topics

***

**Ready to get started?** Begin with the [System Overview](/data-platform/index/architecture/overview.md) to understand how all the pieces fit together.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.datasuite.vn/data-platform/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.