Dec 19, 2025
About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test-driven code delivery.
Key Responsibilities
Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies.
Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads.
Build and manage containerised environments using Docker for consistent development, testing, and deployment.
Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
Integrate Azure services, including:
Azure Functions for...
| IR35 Status: |
Unknown Status
|
CV-Library
London, UK
Contractor