Analytics Engineer is a relatively new title in the world of data that has recently exploded in popularity. A few years ago when it was first being discussed the idea embodied a role that many data analysts played but few had the terminology to describe. Now the role is considered a foundational part of every modern analytics team.
Role and responsibilities
The analytics engineer sits at the intersection of the skill sets of data scientists, analysts, and data engineers. They bring a formal and rigorous software engineering practice to the efforts of analysts and data scientists, and they bring an analytical and business-outcomes mindset to the efforts of data engineering. It’s their job to build tools and infrastructure to support the efforts of the analytics and data team as a whole.
The analytics engineer’s role is to:
- Write production-quality ELT code with an eye towards performance and maintainability
- Coach analysts and data scientists on software engineering best practices (e.g., building testing suites and CI pipelines)
- Build software tools that help data scientists and analysts work more efficiently (e.g., writing an internal R or Python tooling package for analysts to use)
- Collaborate with data engineers on infrastructure projects (where they advocate for and emphasize the business value of applications)
Where do Analytics Engineers come from?
This analytics engineer role is critical because the role of data teams has increased dramatically in scope and the amount of code they’re responsible for.
Data analysts are no longer expected to work just in excel, but rather to use sophisticated BI tools and write complicated queries. Data scientists have always taken pride in being not-particularly-good coders, but now people have realized that if we want our automated-decision-making code written by data scientists to work consistently, we need the code to be good and not just the algorithms.
Analytics engineers bring the discipline and rigor of software engineering to the world of analytics and data. Analytics engineers may be former data engineers who want to get closer to the insights and understand how the data are actually used, or they may be former data analysts or scientists who find that they enjoy the discipline that comes with software engineering even more than they enjoy exploring data.
What expertise do they have?
Analytics engineers are expected to be fluent in SQL and at least one scripting language (generally R or python). They are expected to have a strong knowledge of how cloud data warehouses operate (and how to optimize them for efficiency) and they also should have some familiarity with the type of work that data analysts and data scientists do: how BI tools work, how machine learning models and pipelines work, etc.
Analytics engineers should contribute to:
- Data modeling and data warehouse architecture
- Design of data intensive applications
- BI tool architecture
- Tooling for data scientists
In general, it’s the analytics engineer’s responsibility to use software best practices to help the data team operate more efficiently. That could mean removing duplicated code in the ELT pipeline, teaching analysts how to use version control for their queries, or writing software that allows data scientists to more easily A/B test their models.
How is an analytics engineer different from a data engineer?
We think of the analytics engineer role as being distinct from the data engineering role. However, like other software engineering roles, responsibilities might blur at the edges and the degree of overlap will vary across different organizations.
Data engineers are responsible for data infrastructure and plumbing code. This could be data-intensive tools like Kafka, or data processing tools like Hive or Spark. Data engineers tend to focus on getting raw bytes from point A to B as efficiently as possible, but don’t want to concern themselves with what the data might mean or what the insights are. In general, most companies need fewer data engineers than they did in the past. This is because there are now SaaS companies like Fivetran, Stitch, and Census that provide out-of-the-box solutions for many of the tasks that data engineers were traditionally responsible for. Many companies can get by just using contractors and consultants in the beginning, and they may only need one or two data engineers to “fill in the gaps” of what they can’t purchase from off-the-shelf solutions.
Analytics engineers, in contrast, should care about how the data are being used. This is the biggest difference between analytics engineers and data engineers — analytics engineers need to be excited to get their hands dirty working with analysts and data scientists. Whereas a data engineer’s job might stop once the raw data hit the warehouse or the csv file that’s delivered to a data scientist, an analytics engineer should follow the data all the way to its end use case and try to use software best practices to help make every step of that journey more productive and efficient.
Conclusion
If your company has data analysts or data scientists that write a lot of code, then you probably need an analytics engineer to help productionalize and maintain all of that code.
Unfortunately, because the role is so new, analytics engineers are in incredibly high demand right now, and can be difficult to hire. The good news is that here at Analytics Engineers Club we’re training the next generation of analytics engineers by helping current data analysts step up into the role. If you’re interested in learning more, subscribe to our email list or read more about the course contents here.
