Data Engineering, Platforms and Architecture Market Snapshot 2022

We recently completed a Data, Insight & Analytics Employment Guide using data collected via a detailed survey of industry professionals. We then combined the data with ADLIB’s extensive internal data and knowledge gained from operating within the Data sector to provide this snapshot of the current Data Engineering, Platforms and Architecture Market .

The trend of companies moving onto the cloud has increased dramatically following the COVID years, with companies forced to transition to remote working during lockdowns, the demand for data engineers and architects has risen significantly. What was already a very candidate driven market with a lot of contracting opportunities, has continued to follow that trend, with day rates and salaries increasing with the demand.

Our Consultants thoughts – Scott Rogers, Principal Consultant, Data Engineering & Architecture

The world has changed, prior to the pandemic most companies still had the bargaining power when hiring in data space; this is no longer the case. Data engineering and architecture candidates know this and fully understand their worth, so now the emphasis is on the company and hiring manager to sell opportunities to secure the talent.

Given this shift in power, candidates will leverage the need to achieve increased salaries (in-line with their worth) and gain more flexibility with working arrangements. The majority of data engineers and architects can do their job just as well, if not better, remotely. Arguments for this include time to focus, the ability to work when they want and managing their personal lives. All are valid reasons which increase candidate well-being, motivation and productivity. Companies that offer this autonomy receive engaged employees in return, those that don’t may struggle to compete.

Once candidates have data lake or real-time streaming experience, salaries on the permanent side inflate considerably as these skills are in high-demand in an already extremely candidate short market. Highly skilled candidates know that they have leverage so remote working options and remuneration are currently the best ways to secure them in the post-covid world. Key industries to source data engineering and architecture talent from are Technology, Banking, Insurance, Energy, Telco and Consultancy.

For more mature platforms, we are seeing an increase in demand for data engineers with machine learning infrastructure, deployment and CI/CD experience also. Companies should be aware that this means paying more to acquire the broader skillset, or, willing to flex on certain areas and upskill. This may not be ideal for hiring managers but it will create buy-in from prospective hires as they will learn further skills to develop their careers. Who wants to move to a new role and do the exact thing they were doing before anyway?

Top 5 key tech skills we see for data platform, engineering and architecture:

• Cloud experience (AWS, Azure, GCP)
• Modern programming language (Python/Scala/Java)
• Big Data storage (ADLS, BigQuery, EMR, AWS Datalake etc.)
• Streaming/Ingestion (Kafka, Kinesis, Pub/Sub, Eventhubs etc.)
• CI/CD and Containerisation (Azure DevOps, Jenkins, Gitlab,  Kubernetes, Docker etc.)