Snowflake – Maja Ferle

Another Certification Achieved: Snowflake Advanced DataEngineer

19 March, 20249 September, 2024 by Maja FerleNo Comments

The Snowflake SnowPro Advanced Data Engineer certification is considered tough. After taking the exam, I understand where the "tough" comes from. It's not so much about the difficulty of the exam as about the vastness of the range of topics covered. Sometimes, it's difficult to judge what the responsibilities of a data engineer are. On one extreme, a data engineer is nothing but a developer who receives the requirements and implements them in the pipeline. On the other extreme, a data engineer is expected to understand the platform, configure it, design the security, architecture, and automation, while also performing data analysis. In the real world, data engineers usually fall somewhere between the two extremes. The Advanced Data Engineer exam tests the full spectrum, which covers: Data Movement: ingest data from various formats and load into Snowflake, design data pipelines, build data sharing solutions Performance Optimization: configure the pipelines for the best performance and troubleshoot queries that perform poorly Storage and Data...

Snowflake Data Engineering

31 January, 202422 February, 2024 by Maja FerleNo Comments

Snowflake Data Engineering is my latest book in the making. It's available in the Manning Early Access Program (MEAP) where you get access to new chapters as I write them. In this book, you'll learn how to build data pipelines that ingest data from source systems and store the data in Snowflake. The chapters are organized so that you start by building a simple, basic data pipeline and add increasingly complex functionality. Some of the functionalities covered include ingesting structured and semi-structured data, setting up continuous ingestion with Snowpipe, transforming the data in Snowpark, augmenting your data with generative AI, optimizing performance and cost, designing robust data pipelines, incorporating CI/CD, and much more. Currently available chapters: Data Engineering with Snowflake Creating your First Data Pipeline Best Practices for Data Staging Transforming Data Continuous Data Ingestion Executing Code Natively with Snowpark Stay tuned for more information and updates as new chapters become available. Here is the link to the MEAP: https://www.manning.com/books/snowflake-data-engineering...

Getting Started with Data Engineering and ML using Snowpark for Python

7 October, 20236 November, 2023 by Maja FerleNo Comments

The Data Cloud World Tour is a series of Snowflake events across the globe that aim to share Snowflake’s latest innovations and customer use cases. This year, I attended the Data Cloud World Tour in Dubai and because my company In516ht was one of the sponsors, I was asked to deliver the hands-on workshop. The workshop was based on the Getting Started with Data Engineering and ML using Snowpark for Python quickstart. Attendees were given instructions to set up their laptops with the pre-requisite software, including Python with some additional packages (snowflake-snowpark-python, snowflake-ml-python, pandas), a git client to clone the starter repository and a free trial Snowflake account. The first part of the workshop covered Data Engineering with the following lessons: Configure the connection parameters and establish a connection from Snowpark Python to Snowflake Retrieve data from Snowflake tables that were set up initially into Snowpark data frames Perform exploratory data analysis on the data frames Use data frame methods to query...

Snowflake Summit 2023

10 August, 202330 October, 2024 by Maja FerleNo Comments

This was my second consecutive year at the Snowflake Summit in Las Vegas. While the hot topic last year was data mesh and all sessions about data mesh sold out, this year data mesh was like last year's snow. Now the running theme is Generative AI and LLMs. The good news is that attendees were able to pre-book sessions and many sessions were also recorded so that I didn't miss any of the buzz around these topics. What an exciting Summit it was! So many announcements! Here are some of my favorites: Dynamic Tables. No more streams and tasks. Just write your SQL and Snowflake takes care of the rest, in some ways similar to materialized views, but with less restrictions about the types of queries you can use. Native Applications. Write your application and distribute it via the Snowflake Marketplace. I built a simple app and wrote a blog post about it: Maintaining a Mapping Table with a Snowflake Native App. ...

Another Certification Added to my Stash: SnowPro Advanced Data Analyst

4 April, 202322 April, 2023 by Maja FerleNo Comments

The latest Snowflake advanced certification that was just released is SnowPro Advanced: Data Analyst. Out of all the advanced certifications offered by Snowflake, this one is closest to my professional experience and that's why I decided to tackle it as my first advanced Snowflake certification. Some topics on the exam were relatively easy for me since I have been doing data analysis for decades and SQL is second nature to me. I was able to answer SQL questions without much prior preparation. However, some topics on the exam are very Snowflake-specific that required careful review and study time. These include: Snowsight dashboards. I haven't been using them much, but I had to learn them for the exam. Snowsight dashboards can't compete in terms of features and functionality as compared to more mature tools such as PowerBI, Tableau, or Cognos, to name a few. But what they offer in their simplicity is the possibility to quickly visualize and summarize data, either for...

Role-based Access Control (RBAC) with Secondary Roles

11 October, 202222 April, 2023 by Maja FerleNo Comments

I've written about Role-based Access Control (RBAC) in Snowflake previously. One of the topics I discussed was how to resolve cross-database access. This has now been simplified since Snowflake introduced secondary roles. I wrote in more detail about how secondary roles can be used together with RBAC in this blog post on Medium....

Snowpark for Python First Impressions

28 June, 202222 April, 2023 by Maja FerleNo Comments

Just back from Snowflake Summit 2022 held in Las Vegas. There were so many new announcements about upcoming features in Snowflake, it's hard to keep track. One topic that I'm most excited about is Snowpark for Python. As soon as I got back from Las Vegas, I started digging into the details because I had a use case waiting to be tested. I described my approach and summarized my first impressions in a blog post on Medium....

What the Snowflake Community Means to Me—and My Career

10 May, 2022 by Maja FerleNo Comments

I was recently interviewed for the Snowflake blog where I discussed the benefits of being an active member of the Snowflake Community, how the community has helped me grow into my role as a Snowflake DataSuperhero and how it can be of use to anyone looking for hands-on Snowflake resources. With respect to the usefulness of the Snowflake community I was quoted as saying “Snowflake offers a lot of great technical documentation, but it’s useful that there’s so much material out there from the Snowflake community around personal case studies and how other people are implementing specific features.” Read the full article here....

Snowflake and Data Mesh

24 April, 202210 May, 2022 by Maja FerleNo Comments

More than ever, the ability to use data for decision-making is critical to company success. Despite this knowledge, companies are still not fully empowering their employees with easy access to the data they need. According to Zhamak Dehghani, the founder of Data Mesh, we must start thinking outside of the box because the traditional approach to managing and collecting data is not sufficient any longer. For decades, there has been a divide between operational and analytical data with ETL as the intermediary process to get data from operational systems into the analytical data warehouse. ETL, which has always been primarily in the hands of IT developers, is perceived as a bottleneck to delivering timely analytical data. Furthermore, dimensional data models are not well suited for machine learning models that have become essential. To overcome this, the data lake emerged around 2010. The idea of the data lake is to store vast amounts of semi-structured data in object stores to allow various consumers...

Snowflake account administration best practices

24 February, 202226 February, 2022 by Maja FerleNo Comments

Although it is well documented, it shouldn’t hurt to review a few of the basic Snowflake account administration best practices, as I often see in practice that they are not followed consistently. In my experience, there are so many IT administrators that are pressed for time that they don’t bother to create security roles and policies to be followed. To make it quick, they just use the ACCOUNTADMIN role on Snowflake (or an equivalent all powerful admin role on other systems) because it allows them to do anything and everything without having to think about security best practices. Using an admin role for day-to-day tasks is a bad idea Consider this true story. The administrator tries some functionality, and it works. The user (who doesn’t have admin privileges) tries, and it’s not working. The administrator doesn’t believe the user when they report that the functionality is not working, dismissing them as incompetent. This goes back and forth a few times, until someone gives...