• Challenges and Benefits of the Analytics Engineer Role
    The Analytics Engineer is a relatively new role that emerged during the “Modern Data Stack” trend of the last 5-10 years. In this article, I discuss some of the learnings, challenges, and benefits that I have experienced while adopting this role in my team.
  • Local Dev with Astro CLI and Remote Databases
    When working with Astronomer and Airflow for data processing and ETL tasks, it’s common to encounter scenarios where you need to interact with databases hosted within a private subnet in in your cloud provider. In this blog post, we’ll demonstrate how to set up local development for Astronomer and Airflow while connecting to a Redshift database hosted in a private AWS subnet. TLDR: We’ll achieve this by adding a new container to the Docker Compose file that contains the Astro CLI configuration and configure the Airflow Connection to use that container to reach to your remote DB. Prerequisites Before diving… Read more: Local Dev with Astro CLI and Remote Databases
  • Managing Python versions and environments in macOS
    I have been using Python for a while now, and managing dependencies and versions always has been a pain. macOS still comes with Python2.7, but most dev works have moved to Python3. This time around, I saw myself using Python more and more and decided to stop kicking the can down the road and “properly” set up Python (at least for my use cases). This is a quick setup summary on how I set up and manage my Python versions and environments using pyenv and pyenv-virtualenv. These tools are also compatible with various Linux distributions such as Debian, Ubuntu, Mint. Prerequisite:… Read more: Managing Python versions and environments in macOS
  • How to easily migrate your Brew packages in macOS
    Recently I decided to do a clean install on my personal Mac at the same time and start optimizing my dev setup and overall productivity. The first step towards this goal was to automate installing all the packages and apps I use daily. I have been using Homebrew to manage my macOS dependencies for a few years now and recently I discovered the awesomeness of Brewfiles to export and move dependencies to your new Mac and/or to track it in version control. This very short guide assumes that you already are using Brew to manage macOS packages. If not, you… Read more: How to easily migrate your Brew packages in macOS
  • From Software to Data Engineering
    I started to work as a data engineer when me and my team were tasked to automate some data sets and reports for the businesses/brands our team was supporting. The challenge was significant: these sites lacked any established data pipelines, relying solely on basic event capture. What was initially a three-month project ignited my passion for data and analytics. In this post, I’ll share my experiences and key learnings from the past year. What is Data Engineering? Data engineering is a specialized field within software engineering focused on designing systems that transfer and transform data. The goal is to make… Read more: From Software to Data Engineering