Develop and maintain cloud-based data pipelines that support business intelligence, analytics, and data-driven decisions. Integrate AWS data sources into Delta Lake, monitor data workflows, optimize SQL queries, and contribute to data governance and security alongside IT and business teams.
Responsibilities
- Develop, maintain, and troubleshoot ETL/ELT pipelines using Databricks, Py
- Spark, and SQL.
- Integrate data from AWS S3, RDS, and APIs into Delta Lake.
- Monitor data workflows and resolve issues to ensure reliable data processing.
- Write and optimize SQL queries for data accessibility and performance.
- Assist with data governance, security, and access management using Unity Catalog.
- Collaborate with IT teams and business partners on data warehousing and analytics initiatives.
Required Skills
- Python
- SQLAWS S3
- AWS IAMAWS EC2
- AWS RDSDatabricks
- Py
- Spark
- Apache Spark
- Delta Lake
