Stephen Southern

Senior Solutions Architect · Databricks · United Kingdom

I have more than twenty years of experience in data and analytics, with a strong focus on data warehousing and big data solutions. During my career, I spent a considerable amount of time at the Department for Work and Pensions (DWP) Digital. As a passionate cloud-first advocate, I introduced new cloud-native technologies to enhance our effectiveness, thereby improving business processes and, critically, the outcomes for citizens.

Now, at Databricks, I am dedicated to fully unlocking the capabilities of the Data Intelligence Platform, supporting the development of data solutions that empower my clients to seamlessly analyse, process, and derive valuable insights from their data.

I aim to create a clear and valuable data experience, drawing on my broad knowledge and a growth mindset in the evolving digital and data fields.

Experience

Jan 2024 - Present

Senior Solutions Architect

Databricks (NHS)

In my role as a Senior Solutions Architect at Databricks, I work at the forefront of driving innovation in healthcare and life sciences. Leveraging Databricks Data Intelligence Platform cutting-edge capabilities, I strive to transform data analytics and AI applications, contributing to the acceleration of medical research and ultimately improving patient outcomes.

Jun 2023 - Jan 2024

Lead Technical Architect

DWP Digital (Universal Credit)

As Lead Technical Architect at DWP Digital, I created innovative solutions for Universal Credit, in line with DWP's vision. I focused on user needs, proactively implementing future strategies, and ensuring alignment with department goals. My expertise lay in crafting detailed solutions and driving forward successful designs, contributing to DWP Digital's overall success.

Jun 2020 - Jun 2023

Senior Technical Architect (Data Specialism)

DWP Digital (Health)

As Senior Technical Architect at DWP Digital, I played a key role in creating the Health Data & Analytics Platform. I led its development from concept to an approved architectural pattern, ensuring seamless functionality with advanced Azure data services like Azure Databricks. My dedication to advancing health data solutions brought strategic design expertise and technical leadership to DWP Digital.

Dec 2015 - Jun 2020

Data Science Platform Architect

DWP Digital (Data & Analytics)

As Data Science Platform Architect at DWP Digital, I provided technical leadership for a hybrid cloud infrastructure supporting data science across AWS, Azure, and Crown Hosting. I designed impactful public cloud workspaces, including the successful implementation of the "Cloud Analytics Workspace" on Microsoft Azure. I transformed conceptual ideas into a functional service, demonstrating the value of Data Science and contributing to DWP Digital's continuous advancement in this domain.

Mar 2011 - Dec 2015

Informatica Systems Manager/Architect

DWP Digital (Data & Analytics)

As Informatica Systems Manager/Architect at DWP Digital, specialising in Data & Analytics, I enhanced the Informatica ecosystem. I designed a High Availability Informatica GRID architecture, improving system resilience. I expanded the system architecture by integrating Informatica Data Quality and Test Data Masking for secure data management. My commitment to advancing Informatica capabilities is demonstrated through extensive experience in architecting robust data solutions at DWP Digital.

Jun 2007 - Mar 2011

Informatica System Manager

DWP Digital (Data & Analytics)

As Informatica System Manager at DWP Digital specialising in Data & Analytics, I oversaw and managed Informatica PowerCenter for ETL processes. My focus included seamless integration of business feeds into the Data Warehouse, controlling access, and permissions for ETL code. I proactively implemented structural changes, designed security models, and established best practices for Informatica, contributing to the integrity and security of the environment at DWP Digital.

Sep 2006 - Jun 2007

Extract Transform Load (ETL) Developer

DWP Digital (Data & Analytics)

As ETL Developer at DWP Digital, I focused on designing and delivering Informatica ETL code. Through collaboration and technical documentation, I gained a comprehensive understanding of business processes and source systems. Specialising in a strategic architectural approach, I made significant contributions to the Customer Information System (CIS) and played a pivotal role in establishing a single source of truth for customer information within the Data Warehouse business intelligence at DWP Digital.

Dec 2004 - Sep 2006

Extract Transform Load (ETL) Developer

DWP Digital (Data & Analytics)

As ETL Developer at DWP Digital, I played a significant role in developing Work & Pensions Data Warehouse releases 1 to 3, focusing on ETL for Pensions Transformation and Staffing information. I applied the Kimball Dimensional Model to structure the star schema. I led end-to-end processes, including architecture, design, and implementation. Notable achievements include spearheading the inaugural release of the Data Warehouse and strategically overhauling data store structures within DWP for improved efficiency. I also implemented a robust business intelligence service for the Pensions Transformation Programme, enhancing decision-making capabilities.

Jul 2003 - Dec 2004

Data Solution Developer

DWP Digital (Data & Analytics)

As a Data Solutions Developer at DWP Digital, I specialised in enhancing operational efficiency and accuracy of statistical reporting for key benefit programs. Working as part of a dynamic team producing Quarterly Statistical Reports for Income Support, Jobseekers Allowance, and Pension Credit. Utilised a 5% sample data, IFD bespoke ETL tool ASSIST, and SAS code for both published and web-based reports.

Skills

Key Skills and Knowledge
  • Database Technologies (SQL: Oracle, MySQL, PostgreSQL, SQL Server; NoSQL: MongoDB)
  • Big Data Technologies (Apache Hadoop, Spark, Kafka, HDFS, Databricks)
  • Cloud Computing (AWS, Azure, Google Cloud, Docker, Kubernetes)
  • Data Security (Encryption, GDPR, Role-based access control)
  • Data Integration (ETL: Informatica, Azure Data Factory APIs: REST, SOAP)
  • Data Analytics and Business Intelligence (Python, Power BI )
  • Problem-Solving (Root Cause Analysis, Troubleshooting, Process Optimisation)
  • Collaboration and Communication (Agile, Jira, Salesforce, Slack)
  • Adaptability (Learning new technologies quickly, Adjusting to dynamic environments)
  • Documentation (Technical writing, Git, Confluence)
  • Leadership (Team mentoring, Project management, Conflict resolution)

Awards & Certifications

Data Engineer Associate Data Engineer Associate Generative AI Fundamentals Platform Administrator Databricks Fundamentals Lakehouse Fundamentals

Podcast

Discover the unique story of my career through an AI-generated podcast, crafted using my LinkedIn profile as the foundation. Tune in to hear how my professional journey has unfolded, with key highlights, milestones, and insights brought to life by advanced AI technology. It's a fresh and engaging way to explore my work experience, skills, and achievements—directly from the information in my LinkedIn profile. Whether you're curious about my background or interested in seeing how AI can transform a career narrative, this podcast offers a fascinating listen!

Interests

In addition to my role as a solution architect, I love exploring the world, documenting family adventures through photography and video. A big fan of alternative music, my record collection spans from 80s indie classics and 90s Britpop to 00s Scandinavian electronica.

Along with my artist wife Sally, we live on the North East Coast, walking distance from the beach, which is great when you want to get away from code and screen and clear your head and thoughts. We have one amazing daughter Kitty, presently studying BioMed at Strathclyde University.