Client Name: Confidential

ABOUT THE BLACK BOX TALENT (TBBT)

The Black Box Talent (TBBT) is a talent acquisition and executive search firm helping organizations identify, attract, and hire high-quality professionals across diverse industries and geographies.

We partner closely with our clients to understand their business, culture, and hiring requirements, enabling us to deliver targeted, relevant, and quality talent for critical positions.

Our approach combines industry-focused recruitment expertise, strong candidate networks, structured sourcing, and personalized talent solutions to support organizations in building high-performing teams.

POSITION OVERVIEW

The Lead Data Engineer will be responsible for designing and developing scalable, modular, and fault-tolerant data engineering solutions.

The role involves large-scale data processing, cloud-based data pipelines, PySpark and SQL optimization, stakeholder collaboration, and adherence to data engineering best practices. The position requires 7 years of relevant IT experience.

Understood. I’ll keep all important technical details, while only combining repetitive points and making the wording concise.

KEY RESPONSIBILITIES

• Data Engineering & Pipelines: Design and develop scalable, modular, and fault-tolerant end-to-end data engineering solutions and cluster-based pipelines for batch and streaming data.
• Data Ingestion & Transformation: Develop data pipelines using Python (PySpark), Spark SQL, and ETL processes across AWS/Azure cloud environments.
• Cloud Data Technologies: Work with EMR, Kinesis, Kafka, Lambda, Glue, Athena, Redshift, and Step Functions for cloud-based data engineering solutions.
• Stakeholder & Business Collaboration: Gather requirements, define user stories and technical specifications, explain data modelling decisions, communicate business impact, and collaborate with users, technical teams, senior management, business owners, Enterprise Data & Analytics Architects, and Engineering practice leads.
• Big Data Processing: Design and develop large-scale data pipelines using Big Data, Hadoop, Hive, Python, and Advanced SQL.
• PySpark & SQL Optimization: Develop and optimize PySpark pipelines, SQL queries, and Hive processes, including DataFrames, joins, caching, memory management, partitioning, and parallelism.
• Spark Performance Tuning: Use Spark UI, Event Timelines, DAGs, and Spark configuration parameters to debug, tune, and optimize long-running pipelines.
• Hive & Data Modelling: Design Hive tables with partitioning for improved performance and develop effective data models and ETL pipelines.
• Technical Skills: Strong experience with Python, Pandas, NumPy, PySpark, Spark SQL, Advanced SQL, Hive, Hadoop, Shell scripting, Big Data, Data Modelling, and ETL.
• Cloud, DevOps & Delivery: Use Terraform, Git, CI/CD pipelines, AWS SDKs/native AWS services, and follow Agile implementation practices.
• Best Practices & Architecture: Ensure data engineering solutions follow established architecture, design principles, and engineering best practices, while influencing business and IT teams through strong communication.

CANDIDATE PROFILE

• 7 years of relevant IT experience.
• 4+ years of hands-on experience designing and developing data pipelines using Python (PySpark) / Spark SQL in AWS or Azure cloud.
• Advanced experience writing and optimizing SQL queries with Python and Hive for large datasets in Big Data environments.
• Strong experience in data modelling.
• Hands-on experience creating workflows using scheduling tools such as Autosys or CA Workload Automation.
• Strong understanding of ETL, ELT, and data modelling concepts.
• Experience building both streaming and batch data pipelines.
• Experience with Git and CI/CD pipelines.
• Excellent communication skills with the ability to influence client business and IT teams.