Principal Data Engineer

**About BayRock Labs**

At BayRock Labs, we pioneer innovative tech solutions that drive business transformation. As a leading product engineering firm based in Silicon Valley, we provide full-cycle product development, leveraging cutting-edge technologies in AI, ML, and data analytics. Our collaborative, inclusive culture fosters professional growth and work-life balance. Join us to work on ground-breaking projects and be part of a team that values excellence, integrity, and innovation. Together, let's redefine what's possible in technology.

We are looking for a highly skilled **Senior Data Engineer / Data Architect with deep hands‑on expertise in Azure, Databricks, PySpark, Python, SQL, and modern data warehousing**. This role is ideal for someone who has built large‑scale data platforms, designed enterprise‑grade architectures, and delivered high‑quality data products in cloud environments.

**Core Technical Expertise**

**Azure Cloud Platform**

• ADLS Gen2, ADF, Synapse, Azure SQL, Key Vault, App Services
• Cloud‑native architecture design, scalability, and cost optimization
• Security & governance: RBAC, Managed Identity, Private Endpoints

**Databricks Engineering**

• End‑to‑end development on Azure Databricks
• Lakehouse architecture implementation
• Delta Live Tables (DLT), Unity Catalog
• Performance tuning, cluster optimization
• CI/CD for Databricks workloads

**Big Data & Processing Frameworks**

• Apache Spark with advanced PySpark transformations
• Structured Streaming & batch data pipelines
• Delta Lake architecture & optimization

**Programming & Querying**

• Python for automation, data engineering, and orchestration
• PySpark for distributed data processing
• SQL for complex query optimization and analytics engineering

**Data Architecture & Engineering**

• Enterprise Data Warehouse (EDW) architecture
• Dimensional modeling (Star/Snowflake)
• Data Vault 2.0 modeling
• Metadata‑driven ingestion frameworks
• Change Data Capture (CDC) implementation
• Medallion architecture (Bronze/Silver/Gold)
• Data lineage, governance, cataloging
• Master Data Management (MDM)

**Data Products & Analytics Enablement**

• Design and delivery of scalable data products
• Business‑aligned semantic layer design
• KPI frameworks & enterprise reporting enablement
• Integration of ERP, SaaS, and operational systems
• Hybrid Lakehouse + EDW architectures

**DevOps & Engineering Practices**

• CI/CD pipelines (Azure DevOps, GitHub Actions, Bitbucket)
• Infrastructure as Code (Terraform, ARM templates)
• Automated testing (unit, integration, data quality)
• Monitoring, logging, and observability
• Agile/Scrum delivery

**Strategic & Leadership Impact**

• Principal‑level solution architecture design
• Cross‑functional stakeholder collaboration
• Technical roadmap planning and execution
• Guidance on data governance frameworks (GDPR, SOX)
• Cloud cost optimization strategies
• Migration from legacy EDWs (Teradata, Oracle, SQL Server) to Lakehouse

**Ideal Candidate Profile**

• 5–10+ years of hands‑on experience in data engineering or data architecture
• Strong expertise in Azure + Databricks ecosystem
• Proven experience building scalable, secure, high‑performance data platforms
• Ability to lead design discussions, influence architecture decisions, and mentor teams
• Strong problem‑solving mindset and passion for modern data engineering