Senior Data Platform Engineer
ContractHands-on Senior Data Platform Engineer to build and operate production-grade data pipelines and services across ingestion, transformation, orchestration, and APIs; automate with Terraform; instrument observability; and write tests for reliability. Azure preferred; strong AWS also acceptable. Clear communication and effective use of AI coding tools (e.g., Claude Code) expected.
About the role
You'll work across ingestion, transformation, orchestration, and APIs; automate environments with Terraform; instrument observability; and write tests to keep things reliable. Azure experience is preferred; strong AWS experience is also acceptable. Communicate clearly (async + live), learn fast, and use AI coding tools effectively without relying on them exclusively.
What You'll Do
- Data pipelines: Design, build, and operate reliable batch and change-driven pipelines; schedule/orchestrate jobs; handle schema evolution and failures gracefully.
- Transformations & modeling: Implement clean, tested merge/transformation pipelines; produce consumable models; tune SQL for performance.
- APIs & integration: Build/collaborate on APIs; design request/response contracts; handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing.
- Infrastructure & DevOps: Provision/manage cloud resources with Terraform; automate build/test/deploy with CI/CD; write solid shell scripts for glue/ops tasks.
- Observability & reliability: Instrument logs/metrics/traces; define alert policies and on-call runbooks; track SLOs and drive root-cause prevention.
- Security basics: Apply least-privilege access, secrets management, and encryption in transit/at rest; collaborate on compliance requirements.
- Quality & tests: Write unit, integration, and data-quality tests (including SQL).
- Collaboration & docs: Communicate clearly; produce concise docs (data contracts, mappings, runbooks); partner with product/engineering/analytics.
What We're Looking For
- 7+ years in Data Engineering / Backend Engineering / DevOps.
- Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (Postgres, SQL Server, MySQL).
- Python for data/services; shell scripting (bash preferred; Powershell acceptable).
- Terraform and IaC fundamentals; CI/CD (version control workflows, automated tests, environment promotion).
- Cloud: Azure preferred (storage, compute, identity, monitoring) or very strong AWS background.
- Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays.
- Workflow orchestration: Prefect preferred; Airflow/Dagster acceptable.
- APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
- Observability: Logs/metrics/traces with actionable alerts/dashboards.
- Testing mindset: Tests for code and SQL; fixtures/test data; CI.
- Communication: Clear written/verbal; async and live collaboration.
- AI tooling: Ready to use Claude Code (and similar) while exercising judgment and writing your own code/tests.
Preferred qualifications
- Regulatory/compliance awareness (e.g., data protection, public-sector standards).
- Analytics experience (dim/fact modeling, BI consumption patterns).
- Full‑stack exposure (can prototype simple UI/ops views).
- Data platform architecture exposure (storage/layout, catalog/lineage, governance).
- Familiarity with streaming/message queues and columnar formats (e.g., Parquet).
- Experience with cloud monitoring stacks (Azure Monitor/App Insights, CloudWatch, Datadog, New Relic, etc.).
Engagement details
- Duration: 12+ months (extension/full-time possible).
- Schedule: Flexible, with overlap for key meetings across time zones.
- Compensation: Competitive, based on experience.
- Mode: Remote; available for virtual collaboration and on-call windows as agreed.
Benefits & perks
- Competitive contract compensation.
- Flexible remote work across time zones.
- Opportunity to work with modern data platform tooling.
- Professional growth with real production ownership.
- Exposure to Azure/AWS, Terraform, and orchestration frameworks.
Ready to join our team?
We're excited to hear from you! Click the button below to apply through LinkedIn and take the next step in your career with Black Dog Labs.
Note: Clicking "Apply on LinkedIn" will open LinkedIn in a new tab where you can complete your application. Make sure your LinkedIn profile is up-to-date before applying.