Lead Data Platform Engineer
ContractHands-on lead role owning production-grade data pipelines and delivery. Set technical direction, run design/code reviews, coach engineers, and coordinate across time zones. Work across ingestion, transformation, orchestration, and APIs; automate with Terraform; instrument observability; and ensure reliability with testing. Azure preferred; strong AWS acceptable.
About the Role
We’re looking for a hands-on Lead Data Platform Engineer to both build/operate production-grade data pipelines and lead delivery. You’ll set technical direction, run design reviews and code reviews, coach engineers, and coordinate work across time zones. Day to day you’ll work across ingestion, transformation, orchestration, and APIs; automate environments with Terraform; instrument observability; and write tests to keep things reliable. Azure experience is preferred; strong AWS experience is also acceptable. You should communicate clearly (async chat and live sessions), learn fast, and use AI coding tools (e.g., Claude Code) effectively without relying on them exclusively.
What You'll Do
- Technical leadership & delivery: Lead a small squad or workstream; scope milestones; manage dependencies and risks; unblock engineers; communicate status to stakeholders; land outcomes on time.
- Architecture & direction: Define target states and data contracts; make build/buy calls; author RFCs; drive cost/performance guardrails and reliability patterns.
- Data pipelines: Design, build, and operate reliable batch and change-driven pipelines; schedule/orchestrate jobs; handle schema evolution and failures gracefully.
- Transformations & modeling: Implement clean, tested merge/transformation pipelines; produce models that are easy to consume for products and analytics; tune SQL for performance.
- APIs & integration: Build or collaborate on APIs for data access/updates; design request/response contracts; handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing.
- Infrastructure & DevOps: Provision/manage cloud resources with Terraform; automate build/test/deploy with CI/CD; write solid shell scripts for glue/ops tasks.
- Observability & reliability: Instrument logs/metrics/traces; define SLOs and on-call runbooks; lead incident reviews and drive root-cause prevention.
- Security & compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest; partner on compliance requirements.
- Quality & standards: Champion unit/integration/data-quality tests (including SQL); enforce code health via reviews, linters, and CI checks.
- Collaboration & docs: Operate async-first; produce concise docs (data contracts, mappings, runbooks, RFCs); collaborate closely with product/engineering/analytics across time zones.
What We're Looking For
- 9+ years in Data Engineering / Backend Engineering / DevOps, including team leadership or end-to-end ownership of large, multi-stakeholder projects.
- Proven leadership: led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback.
- Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL).
- Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable).
- Terraform and IaC fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion).
- Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background.
- Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays.
- Workflow orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable.
- APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
- Observability: Logs/metrics/traces and actionable alerts/dashboards.
- Testing mindset: habit of writing tests for code and SQL; fixtures/test data; CI.
- Communication: Clear written/verbal comms; practiced cross-timezone collaboration.
- AI tooling: Ready to use Claude Code (and similar assistants) to accelerate work—while exercising judgment and writing your own code/tests.
Preferred Qualifications
- Experience mentoring, interviewing, onboarding, and raising code quality standards.
- Regulatory/compliance awareness (e.g., data protection or public-sector standards) and how it impacts design/operations.
- Analytics exposure (dim/fact modeling, BI consumption patterns).
- Data platform architecture exposure (storage/layout choices, catalog/lineage, governance concepts).
- Familiarity with Kafka, streaming/message queues, and columnar formats (e.g., Parquet).
- Experience with cloud monitoring stacks (Azure Monitor/Application Insights, CloudWatch, Datadog, New Relic, etc.).
Engagement Details
- Duration: 10+ months (extension/full-time possible).
- Schedule: Flexible, with overlap for key meetings across time zones.
- Compensation: Competitive, based on experience.
- Mode: Remote; available for virtual collaboration and on-call windows as agreed.
Benefits & Perks
- Competitive contract compensation based on experience.
- Flexible remote work with time-zone collaboration.
- Work with cutting-edge data platform technologies.
- Leadership development and technical growth opportunities.
- Strong reliability/observability practices in production.
- Opportunity to use modern AI coding tools responsibly.
Ready to Join Our Team?
We're excited to hear from you! Click the button below to apply via email and take the next step in your career with Black Dog Labs.
Note: Clicking "Apply via Email" will open your default email client with a pre-filled application template. Please attach your resume/CV and fill in the requested details before sending.
If your settings aren't configured correctly for the button above, drop us a line at careers@blackdoglabs.io.