Data Engineering and Architecture Talent & Skills Snapshot

Based on ADLIB internal data and knowledge, we have created a UK Data Engineering and Architecture talent and skills overview as we head into 2024.

Sector Overview

The year 2023 was a challenging one for the UK data job market, especially for the tech industries and the contract market. Many businesses faced redundancies and hiring freezes due to the economic downturn and the pandemic. As a result, employers gained more power and control over the hiring process and became more selective of the candidates they chose.

However, this created an opportunity for the companies that were able to adapt and innovate. With the continued trend of modern data platform builds, dataOps, machine learning infrastructure and real-time streaming, the demand for highly skilled data engineering and architecture talent remained high. The most forward-thinking businesses invested in creating bespoke data platforms with modern open-source technologies, offering flexible and more scalable solutions than traditional ones. Data mesh architectures also became more popular, as they enabled a more decentralised and agile approach to data management.

A key point to highlight from our clients this year is the ongoing debate of building in-house versus plug-in and play solutions. Considerations of time, customisation, control, cost, talent skills etc. have pushed clients in different directions depending on their needs. From the leaders in the space – build where its core, integrate where it’s not.

Skills in demand

Key Hard Skillsets

Hard skillsets that were in demand for data engineering and architecture roles were mostly the same as last year: cloud experience on AWS (35%), Azure (35%) and GCP (10%), Python, SQL, and infra-as-code, containers, datalakes and lakehouses technologies. However, the competition for these roles was fierce, as many candidates had similar qualifications and experience.

Key Soft Skillsets

Key soft skills include effective listening, articulate speaking and active engagement. Data professionals should excel in collaborating with cross-functional teams, simplifying complex concepts for diverse audiences, and ensuring buy-in from both technical and non-technical stakeholders. They must be confident in voicing their opinions during meetings and contribute valuable perspectives. A strong focus on delivering customer value is essential, coupled with a proactive approach to understanding and meeting business needs and objectives.

How to attract and retain the best Data Engineering and Architecture talent in 2024?

The key is to offer flexibility and autonomy to your candidates, with a progressive tech stack, while embracing technological advancements in ML & AI.

More and more data professionals are looking for work arrangements that suit their preferences and lifestyles, such as fully remote or hybrid options. They also want to work on projects that are meaningful and challenging, and that allow them to use their creativity and skills. They want to have a say in the design and implementation of the data platforms and to have access to the latest tools and technologies. They want to work in a culture that values collaboration, learning and feedback.

If you can offer these benefits to your candidates, you will have a better chance of attracting and retaining the best data engineering and architecture talent in 2024.