Impress employers and recruiters.
Choose from hundreds of resume examples.

Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

About the Role

We are seeking a Senior Big Data Developer with over 5 years of experience in big data engineering, API integration, and AI-assisted development to design, build, and maintain scalable data pipelines and backend systems within an enterprise environment.

Responsibilities

• Design and develop Spark-Scala applications for large-scale data processing on Hadoop/CDP clusters.
• Build and optimize ETL/ELT pipelines using Spark DataFrames, Datasets, and Spark SQL.
• Tune Spark jobs for performance, including partitioning, caching, broadcast joins, and shuffle optimization.
• Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms.
• Work with Parquet, ORC, and Avro file formats on HDFS.
• Write complex HiveQL/Spark SQL queries, including window functions, CTEs, subqueries, and aggregations.
• Design and maintain Hive external/managed tables and partitioned datasets.
• Optimize slow-running queries and resolve correlated subquery issues.
• Work with HDFS encryption zones and data governance requirements.
• Develop and maintain bash shell scripts for job orchestration and automation.
• Handle error management, return codes, logging, and alerting in shell scripts.
• Manage HDFS operations, file transfers, and data validation.
• Manage Kerberos authentication, including kinit and keytab handling.
• Build scripts and pipelines to extract data from REST APIs using curl and Python.
• Handle OAuth2 token generation, bearer token refresh, and API health checks.
• Parse and process JSON API responses and load them into HDFS/Hive.
• Manage pagination, error handling, and retry logic for API calls.
• Work with enterprise API gateways and URL parameter construction.
• Leverage GitHub Copilot/AI coding assistants to accelerate development.
• Use AI tools for code review, SQL generation, script debugging, and documentation.
• Contribute to AI-assisted data quality and anomaly detection pipelines.
• Explore and implement LLM-based automation for repetitive data engineering tasks.
• Schedule and manage jobs using AAP/Control-M/cron.
• Build and maintain Ansible playbooks for automated deployments.
• Manage deployment pipelines, including artifact versioning, Vault secret injection, and environment-specific configuration.
• Monitor job health, handle failures, and implement alerting.

Requirements

• 5+ years of experience in big data engineering, API integration, and AI-assisted development.
• Experience with Cloudera CDP (7.x) and migration from HDP is a plus.
• Knowledge of Kerberos, Vault, and HDFS encryption zones is a plus.
• Familiarity with CI/CD pipelines (Helios, GitHub Actions) is a plus.
• Experience with MSSQL/JDBC connectivity from Spark is a plus.
• Understanding of AML/Financial regulatory data domains is a plus.

Skills

• Big Data
• Spark
• Spark-Scala
• Hadoop
• CDP
• ETL
• ELT
• Spark DataFrames
• Spark SQL
• HiveQL
• Parquet
• ORC
• Avro
• HDFS
• Unix
• Shell Scripting
• Bash
• REST APIs
• curl
• Python
• OAuth2
• JSON
• GitHub Copilot
• AI coding assistants
• LLM
• AAP
• Ansible Automation Platform
• Control-M
• cron
• Ansible
• Vault
• CI/CD
• MSSQL
• JDBC

Experience Level

• Senior

Salary/Compensations

• 79,000 to 102,000

Benefits

• Paid time off based on employee grade (A-F): Vacation: 12-25 days, depending on grade
• Company paid holidays
• Personal Days
• Sick Leave
• Medical, dental, and vision coverage
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs

About the Company

• Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world, while creating tangible impact for enterprises and society.
• It is a responsible and diverse group of 340,000 team members in more than 50 countries.
• With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs.
• It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise and partner ecosystem.

Equal Opportunity

• Capgemini is an Equal Opportunity Employer encouraging inclusion in the workplace.
• Capgemini also participates in the Partnership Accreditation in Indigenous Relations (PAIR) program which supports meaningful engagement with Indigenous communities across Canada by promoting fairness, accessibility, inclusion and respect.
• We value the rich cultural heritage and contributions of Indigenous Peoples and actively work to create a welcoming and respectful environment.
• All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.
• This is a general description of the Duties, Responsibilities and Qualifications required for this position.
• Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed.
• Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodation does not pose an undue hardship.
• Capgemini is committed to providing reasonable accommodation during our recruitment process.
• If you need assistance or accommodation, please reach out to your recruiting contact.
• Please be aware that Capgemini may capture your image (video or screenshot) during the interview process and that image may be used for verification, including during the hiring and onboarding process.
• Click the following link for more information on your rights as an Applicant in the United States. http://www.capgemini.com/resources/equal-employment-opportunity-is-the-law