Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

Tailor your resume to this Senior Data Engineer (Web Scraping) role.

Rezi rewrites your resume against Oxford Data Plan's job description. Free.

Resume score gauge reading 58 out of 100

Don't guess if your resume is good enough.

See how it scores against the Senior Data Engineer (Web Scraping) posting at Oxford Data Plan — free, in seconds.

About the Role

We are seeking an experienced Data Engineer to own web scraping and data acquisition within our data platform. This senior, hands-on role involves improving the scale, robustness, and integration of scraping workloads into our lakehouse architecture, enhancing scheduling, monitoring, and storage, and establishing reusable patterns for reliable scraper development and operation.

Responsibilities

Own the development and operation of web-scraping and web-data ingestion pipelines.
Design and establish a scalable web-scraping framework, including common patterns for extraction, scheduling, storage, monitoring, validation and failure handling.
Design, build and maintain reliable production scrapers for new and existing data sources.
Evaluate and make build-versus-buy recommendations for scraping infrastructure and third-party services.
Investigate websites and determine the most appropriate acquisition approach.
Ensure scraping activities appropriately consider internal policies, website terms, robots.txt, access restrictions, privacy and intellectual-property constraints.
Diagnose and resolve issues caused by changing websites, dynamic content, authentication, rate limits and other operational challenges.
Use AI tooling to work more effectively while understanding, reviewing and being able to defend the code you ship.

Requirements

Demonstrated professional experience building and operating production web-scraping systems at scale.
Proven ability to independently own substantial scraping projects from initial investigation through production operation.
Strong production Python engineering skills, including building maintainable applications rather than standalone scripts.
Strong practical experience with scraping technologies such as Requests/httpx, BeautifulSoup, Scrapy, Playwright or Selenium.
Good understanding of HTTP, HTML, APIs, JavaScript-rendered websites and browser/network behaviour.
Experience handling common scraping challenges such as pagination, authentication, sessions, retries, rate limiting, concurrency and proxies.
Strong understanding of data pipelines, data quality and how collected data should be validated, stored and consumed downstream.
Experience deploying, monitoring and supporting production workloads in a cloud environment.
Strong debugging and problem-solving skills, with the ability to work independently and make sensible engineering decisions.
Experience working effectively within a remote engineering team, including code review, documentation and ticket-based workflows.

Skills

Python
Requests/httpx
BeautifulSoup
Scrapy
Playwright
Selenium
HTTP
HTML
APIs
JavaScript-rendered websites
Data pipelines
Data quality
Cloud environment
Debugging
Problem-solving
Code review
Documentation
Ticket-based workflows
AWS
Lakehouse architectures
Iceberg
PySpark
Distributed data-processing
Docker
Containerised workloads
Terraform
Infrastructure-as-code
Grafana
Observability platforms
High-volume crawling systems
Distributed crawling systems
Commercial scraping services
Proxy services
Browser-infrastructure services
Automated scraper testing
Canary runs
Source-drift detection
Legal
Compliance
Privacy
Data-governance

Work Type

Remote

Experience Level

Senior
Experienced