The Analytics Engineer is a relatively new role that emerged during the “Modern Data Stack” trend of the last 5-10 years and was ”formalized” by dbt Labs. In this article, I discuss some of the learnings, challenges, and benefits that I have experienced while adopting this role in my team.
TLDR: Analytics Engineers merge the analytical skills of a data analyst with the engineering mindset and practices of a software engineer, creating a hybrid profile. This can be very effective in keeping a data team agile, providing insights, and aiding in the decision-making process more effectively than traditional data analysts.
The Identity Crisis: More Than Just an Analyst (or Engineer)
One of the first hurdles analytics engineers face is an “identity crisis” within their organizations. The title might suggest a sort of chameleon that can seamlessly transition between deep data analysis and traditional software engineering tasks.
While it’s true that many analytics engineers are proficient in Python and other programming languages, their expertise in SQL and understanding of the data lifecycle and analytics best practices are what sets them apart.
So, what’s an Analytics Engineer?
At its core, the Analytics Engineer role is about bridging the gap between data and decision-making. Unlike data engineers, whose work primarily revolves around building the infrastructure and pipelines to move and process data, Analytics Engineers focus on creating testable and replicable data products capable of providing actionable insights to the business. They have both a deep knowledge of what data is available, and how the business operates – making them excellent thought partners and leaders in your business.
Analytics engineers adopt from software engineers the use of software development lifecycle (SDLC) techniques in data modeling and analysis. This includes version control, testing, documentation, and continuous integration/continuous deployment (CI/CD), which are not typically used in traditional analytics teams. By adopting these methodologies, analytics engineers make data transformations testable, scalable, and maintainable which translate into reliable and accessible data across the organization.
Where do they fit?
In small to medium-sized teams (3-8 people), this role fits perfectly alongside other data engineers, as the collaboration will be very tight and the business and technical knowledge can flow more freely. The same applies for big organization with distributed data teams.
In bigger organizations with established central data teams (20+ members), it might be more effective to treat this role as an internal evolution by adopting its best practices and tools aiming to provide agility and autonomy to its team members.
Data teams don’t operate in a vacuum, and as any change in an organization, it will take time to adapt. It will include a learning period and is crucial to have support from business stakeholders as it will change how they collaborate with the data team. This support includes engagement and accountability from the business to ensure they are leveraging the analytics engineers in their day-to-day, providing feedback and engaging with the data products they develop.
The Challenge
Expectation Gaps
The main challenge of this role is managing the expectations of business peers that are used to work with traditional Analytics and BI teams. These teams are viewed as service teams that deliver specific requests such as dashboards, one-off reports, and answer ad-hoc business questions. In contrast, this new role encourages that analyst not only responds to requests but also proactively propose and build data products that can be used directly by business peers without relying on the data team.
As this role evolves within the organization, it is important to maintain a delicate balance between promptly addressing business questions and developing reusable data products that are flexible enough to support multiple use cases or even teams. This can be particularly challenging if businesses expect to combine multiple roles and responsibilities into this one new role without the proper support.
The Benefits
- Facilitates a data-driven culture: by having an end-to-end understanding of the data and the business, analytics engineers are in the unique position to promote a data-driven culture by building data products that allow to discover opportunities and aid in the decision making process.
- Enhanced Data Reliability: by incorporating engineering best practices and their analytical mindset, they produce testable and replicable datasets that can be trusted and shared across the organization, reducing data silos and gaps.
- Team Agility: by not relying in data engineers to build data models and analysis pipelines, Analytics Engineers can work with increased autonomy and speed compared to traditional data analysts, as they now have the tools to build more complete data solutions.
Final Thoughts
It’s important to note, that the viability of the Analytics Engineer role is only possible with the adoption of modern tools. These tools simplify many of the data engineering tasks needed to have a proper data practice where data is accesible, trusted, secure, and can be easily leveraged to add business value. These tools are often grouped under the “Modern Data Stack” concept, with platforms like dbt, Mode, and Fivetran leading in their respective domains of data transformation, analytics, and integration.
This role clearly reflects the evolution and maturity of the data & analytics field over the last few years, shifting from separate teams of engineers and analysts to a more integrated and lean approach. We will need to see how the integration of ML (and particularly LLMs) in analytics tools revolutionize even further the roles in the data space. Time will tell.