Job Description
Summary
Prodigy Resources is seeking a Senior AI Data Engineer to design, build, and maintain cloud-native data platforms that power advanced analytics, real-time data applications, and AI/ML workflows. This role is critical to enabling machine learning in production environments by delivering robust, scalable, and high-quality data architectures.
Key Responsibilities
- Design and maintain scalable batch and real-time data pipelines using AWS-native tools.
- Develop and manage data lake and lakehouse architectures leveraging S3, Glue Catalog, Athena, EMR (Iceberg/Delta Lake), and Redshift.
- Collaborate with ML engineers and data scientists to operationalize ML models and support MLOps pipelines.
- Ensure data quality, observability, lineage tracking, and compliance across all data products.
- Design scalable data models for storage solutions including Amazon DynamoDB and other AWS-based services.
- Build and maintain online/offline feature stores to support low-latency AI applications.
- Contribute to decentralized data ownership and federated data governance models using tools such as Amazon DataZone.
- Partner with product managers, engineers, and analysts to align data infrastructure to strategic business goals.
- Design systems leveraging event-driven architectures and microservices principles.
- May lead projects and support cross-functional initiatives as needed.
- Occasional business travel may be required.
- Required Skills and Qualifications Expertise in Python and SQL for ETL/ELT, data transformation, and automation.
- Deep experience with AWS services, including S3, Glue, Athena, Redshift, DynamoDB, Kinesis/MSK, Lambda, Step Functions, and EMR.
- Strong understanding of modern data architectures, including data lakes, lakehouses, and data warehouses.
- Hands-on experience with Apache Spark, Kafka, and orchestration tools such as Airflow.
- Familiarity with DevOps and infrastructure-as-code (IaC) tools like Pulumi, CloudFormation, or AWS CDK.
- Knowledge of data governance, security, and compliance (e.g., HIPAA, GDPR).
- Proficiency in semantic layer design using AWS Glue Catalog or third-party platforms like Alation and Collibra.
- Ability to integrate semantic assets into lakehouse ecosystems.
- Strong communication skills with the ability to work cross-functionally in Agile environments.
- A problem-solving mindset, attention to detail, and a commitment to delivering high-quality solutions.
- Education and Experience Bachelor’s or Master’s degree in Computer Science, Engineering, or a related technical field required.
- AWS certifications (Data Analytics, Machine Learning) or equivalent credentials preferred.
- Core CompetenciesDrive for Results: Acts with urgency, persistence, and strategic focus to achieve ambitious goals.
- Customer Focus: Proactively anticipates needs and exceeds expectations through reliable, empathetic support.
- Self-Awareness: Reflects thoughtfully on actions and adapts behavior to drive better outcomes.
- Valuing Others: Builds trust and respect across diverse teams through transparent, inclusive collaboration.
- Learning Agility: Continuously seeks growth opportunities, feedback, and innovative ideas.
- Innovation: Brings forward and experiments with new ideas to drive efficiency, scalability, and value.
- Core Values Treat everyone with dignity and Respect
- Earn and maintain Trust
- Provide Reliable services that open doors
- Serve with courtesy and Compassion
- Prioritize Safety
- Communicate with Transparency
Job Tags