Director of Data Engineering

Position Summary The Director of Data Engineering is responsible for building and operating the data engineering function within Ryan Specialty’s corporate data organization. This role is accountable for the silver and gold layer of the medallion data architecture thus transforming raw ingested data into clean, conformed, and business-ready data assets that serve as the foundation for downstream consumption. Beyond technical delivery, this role builds and scales a high-performing engineering team, establishes disciplined software development practices, and manages the contractor and external parties that augment internal capacity. An ideal candidate should be collaborative and dynamic and be able to work across the organization on a fast-paced and quickly evolving team. The Leader Will Have Five Primary Responsibilities Own and deliver the silver and gold layer of the medallion data architecture Establish and enforce a standardized software development lifecycle for all data engineering work Build, mentor, and develop a high-performing data engineering team Partner with IT on data platform infrastructure, tooling, and integration What will your job entail? Own the Databricks medallion architecture from silver forward: transformation patterns, data quality enforcement, business logic application, gold-layer curation, and pipeline orchestration Establish and enforce a standardized SDLC that all team members must follow: version control, code review, testing, CI/CD, and documentation Design and implement a data quality framework in partnership with the VP of Data Governance who defines it from a business perspective Lead Master Data Management (MDM) technical implementation: entity resolution pipeline design, match-and-merge workflow configuration, golden record integration into the gold layer, and performance optimization Serve as the primary interface between the data organization and the IT data platform team Translate engineering requirements into platform specification across Azure and Databricks for the IT data platform team to deliver Conduct structured code reviews that are explicitly developmental for data engineers Build a library of engineering standards, patterns, and examples Own the technical planning for contractor-staffed project delivery pods Design and maintain the employee and contractor onboarding process Provide ongoing technical oversight for all data engineers Own technical compliance enforcement in the Databricks environment: data lineage tracking via Unity Catalog, access audit logs, encryption verification, and contractor access governance Hire, mentor, develop, and assess a team of data engineers and contractors Position Qualifications Bachelor’s degree in computer science, information systems, engineering, or a related technical field 7+ years in data engineering with at least 3 years in leadership or management role Proven experience owning large-scale data engineering development in a cloud-native environment Deep expertise in Azure Databricks, including Unity Catalog, Delta Lake, Delta Live Tables, PySpark, Spark SQL, Databricks SQL, Jobs/Workflows, and cluster management Hands-on experience with data quality frameworks, including implementing quality gates, monitoring pipelines, and designing alerting systems Strong SQL skills with experience in large-scale dimensional modeling (star schema, slowly changing dimensions, multi-source fact table design) Experience with cloud data lake architectures on Azure (ADLS Gen2, Azure Data Factory, Azure DevOps, Azure Key Vault). AWS experience is acceptable if Databricks experience is strong Proficiency in Python for data engineering (PySpark, pandas, orchestration scripts, SDK integrations) Hands-on experience with CI/CD pipelines for data (Azure DevOps or equivalent); comfortable reviewing infrastructure-as-code, deployment scripts, and PR pipelines Demonstrated ability to push back on senior stakeholders and enforce process discipline Track record of building and scaling data engineering teams in a high-growth environment: establishing code review norms, architecture decision record, knowledge-sharing sessions, and technology onboarding programs Experience managing contractors and external development partners Experience in the insurance industry, particularly E&S lines preferred; or a strong willingness to learn Ability to understand and communicate with technical and business experts Unique Skill Requirements Intellectual curiosity with a strong drive to learn Critical thinker with an open mind to