Senior Data Engineer – Python, SQL, Cloud & Databricks

Data Engineer

Location: PAN India
Experience: 8+ Years Overall (Minimum 5 Years in Data Engineering)
Band: BPO 5
Shift: General Shift
Qualification: Graduation Mandatory

About the Role

We are hiring an experienced Data Engineer with strong expertise in developing scalable data pipelines, integrating heterogeneous data sources, and building cloud-based data engineering solutions.

The ideal candidate should possess hands-on experience in Python/R, SQL, NoSQL, ETL/ELT, Apache Spark, Databricks and cloud platforms such as AWS, Azure or GCP.

The position involves building reusable data pipelines, managing structured and unstructured datasets, implementing data quality frameworks and supporting clinical data integration across enterprise systems.

Key Responsibilities

Data Engineering & Pipeline Development

• Design, develop and maintain scalable data pipelines for heterogeneous data sources.

• Build reusable ETL/ELT frameworks for efficient data extraction, transformation and loading.

• Automate manual data integration processes.

• Develop solutions to process large structured and unstructured datasets.

• Implement data integrity checks, quality validations and pipeline monitoring.

• Troubleshoot data ingestion and integration issues.

Cloud & Data Warehousing

• Develop and deploy data pipelines using AWS, Azure or GCP.

• Work with Apache Spark, Databricks, Snowflake, Amazon Redshift and data lake technologies.

• Design efficient data storage, staging and transformation workflows.

• Optimize data processing and delivery performance.

• Support enterprise-level data warehouse and data lake infrastructure.

Clinical Data Integration

• Develop end-to-end data extraction and transformation processes for clinical datasets.

• Coordinate with Statistical Programmers, SDTM Programmers, Clinical Data Programmers and Analytics teams.

• Support study-level data ingestion and integration requirements.

• Develop and validate APIs, data streams and data transfer specifications.

• Ensure accurate data formats, transfer frequency and quality deliverables.

• Follow applicable SOPs, computer system validation requirements and Good Clinical Practice (GCP) regulations.

Technical Leadership & Collaboration

• Collaborate with system architects, IT teams, vendors and business stakeholders.

• Translate business requirements into technical specifications.

• Provide technical guidance on data integration and pipeline development.

• Support project delivery, troubleshooting and continuous improvement.

• Develop reusable libraries, templates and data engineering standards.

• Participate in training, knowledge sharing and process optimization.

Mandatory Technical Skills

• Minimum 5 years of hands-on Data Engineering experience.

• Strong programming experience with Python and/or R.

• Strong SQL and NoSQL knowledge.

• Experience building and maintaining cloud-based data pipelines.

• Hands-on experience with AWS, Azure or GCP.

• Experience with ETL/ELT development.

• Apache Spark / Databricks experience.

• Data Warehousing and Data Lake experience.

• Experience with Snowflake, Amazon Redshift or similar platforms.

• Knowledge of GitHub, GitLab and CI/CD tools such as Jenkins.

• Experience processing structured and unstructured data.

• Strong understanding of XML, JSON, APIs and database concepts.

• Experience handling large datasets and implementing data quality checks.

Preferred Skills

• Clinical Data Engineering / Clinical Data Integration.

• R Shiny or Python application development.

• Hadoop.

• Agile development methodologies.

• Data visualization.

• Python or R certification.

• Experience working with cross-functional global teams.

Educational Qualification

Bachelor's degree in Computer Science, Statistics, Biostatistics, Mathematics, Biology, another health-related discipline, or a relevant equivalent qualification/experience.

Graduation is mandatory.

Important Eligibility Criteria

• Approximately 8+ years of overall experience.

• Minimum 5 years of relevant Data Engineering experience.

• Candidates must be comfortable working in General Shift.

• Former and current TCS/Tata employees are not eligible.

• PAN and Date of Birth are required for candidate profile creation during recruitment.