Maja Ferle

Another Certification Achieved: Snowflake Advanced DataEngineer

19 March, 20249 September, 2024 by Maja FerleNo Comments

The Snowflake SnowPro Advanced Data Engineer certification is considered tough. After taking the exam, I understand where the "tough" comes from. It's not so much about the difficulty of the exam as about the vastness of the range of topics covered. Sometimes, it's difficult to judge what the responsibilities of a data engineer are. On one extreme, a data engineer is nothing but a developer who receives the requirements and implements them in the pipeline. On the other extreme, a data engineer is expected to understand the platform, configure it, design the security, architecture, and automation, while also performing data analysis. In the real world, data engineers usually fall somewhere between the two extremes. The Advanced Data Engineer exam tests the full spectrum, which covers: Data Movement: ingest data from various formats and load into Snowflake, design data pipelines, build data sharing solutions Performance Optimization: configure the pipelines for the best performance and troubleshoot queries that perform poorly Storage and Data...

DATA_FAIR, a Data Engineering and Data Science Conference

16 February, 202430 October, 2024 by Maja FerleNo Comments

I spent an incredible day at the DATA_FAIR, a conference dedicated to fostering an inclusive environment for knowledge exchange, networking and upskilling in data engineering and data science. It was a day packed with learning from my peers, meeting new like-minded individuals and exchanging experiences. The focus of the conference was on practical applications of data engineering technologies, current and emerging trends in ML and AI, followed by a round-table discussion about ethical data engineering. My contribution to the conference was to speak about "The Role of a Data Engineer in a Data Mesh Architecture". I explained the traditional data warehousing architecture, the challenges of this architecture that include long time to delivery, low flexibility, and dependence on the IT department for implementation. Because the ability to use data for decision-making is critical to company success, companies should empower their employees with easy access to the data they need. According to Zhamak Dehghani, the founder of Data Mesh, we must start thinking outside...

Snowflake Data Engineering

31 January, 202422 February, 2024 by Maja FerleNo Comments

Snowflake Data Engineering is my latest book in the making. It's available in the Manning Early Access Program (MEAP) where you get access to new chapters as I write them. In this book, you'll learn how to build data pipelines that ingest data from source systems and store the data in Snowflake. The chapters are organized so that you start by building a simple, basic data pipeline and add increasingly complex functionality. Some of the functionalities covered include ingesting structured and semi-structured data, setting up continuous ingestion with Snowpipe, transforming the data in Snowpark, augmenting your data with generative AI, optimizing performance and cost, designing robust data pipelines, incorporating CI/CD, and much more. Currently available chapters: Data Engineering with Snowflake Creating your First Data Pipeline Best Practices for Data Staging Transforming Data Continuous Data Ingestion Executing Code Natively with Snowpark Stay tuned for more information and updates as new chapters become available. Here is the link to the MEAP: https://www.manning.com/books/snowflake-data-engineering...

Getting Started with Data Engineering and ML using Snowpark for Python

7 October, 20236 November, 2023 by Maja FerleNo Comments

The Data Cloud World Tour is a series of Snowflake events across the globe that aim to share Snowflake’s latest innovations and customer use cases. This year, I attended the Data Cloud World Tour in Dubai and because my company In516ht was one of the sponsors, I was asked to deliver the hands-on workshop. The workshop was based on the Getting Started with Data Engineering and ML using Snowpark for Python quickstart. Attendees were given instructions to set up their laptops with the pre-requisite software, including Python with some additional packages (snowflake-snowpark-python, snowflake-ml-python, pandas), a git client to clone the starter repository and a free trial Snowflake account. The first part of the workshop covered Data Engineering with the following lessons: Configure the connection parameters and establish a connection from Snowpark Python to Snowflake Retrieve data from Snowflake tables that were set up initially into Snowpark data frames Perform exploratory data analysis on the data frames Use data frame methods to query...

Artificial Intelligence for Project Managers

29 September, 20235 November, 2023 by Maja FerleNo Comments

With the current wave of Generative AI opportunities permeating all aspects of work and personal life, I grabbed the chance to enroll in PMI's Generative AI Overview for Project Managers course. According to the course introduction, Generative AI will impact project management in various ways, including in improving project delivery success rates, benefits realization, societal impact of projects with global influence, and career advancements for project managers. The World Economic Forum predicts that 75% of companies might adopt some form of AI technologies by 2027. Time for project managers to get ready. The course illustrates many practical ways that Generative AI tools, such as ChatGPT, Bard, and other emerging tools can help project managers in various scenarios, such as: cost-benefit analysis developing a business case justification creating a project charter calculating earned value creating agile user stories prepare talking points to communicate with difficult stakeholders give advice on how to communicate more empathically It can also help project managers with repetitive tasks which...

Snowflake Summit 2023

10 August, 202330 October, 2024 by Maja FerleNo Comments

This was my second consecutive year at the Snowflake Summit in Las Vegas. While the hot topic last year was data mesh and all sessions about data mesh sold out, this year data mesh was like last year's snow. Now the running theme is Generative AI and LLMs. The good news is that attendees were able to pre-book sessions and many sessions were also recorded so that I didn't miss any of the buzz around these topics. What an exciting Summit it was! So many announcements! Here are some of my favorites: Dynamic Tables. No more streams and tasks. Just write your SQL and Snowflake takes care of the rest, in some ways similar to materialized views, but with less restrictions about the types of queries you can use. Native Applications. Write your application and distribute it via the Snowflake Marketplace. I built a simple app and wrote a blog post about it: Maintaining a Mapping Table with a Snowflake Native App. ...

Another Certification Added to my Stash: SnowPro Advanced Data Analyst

4 April, 202322 April, 2023 by Maja FerleNo Comments

The latest Snowflake advanced certification that was just released is SnowPro Advanced: Data Analyst. Out of all the advanced certifications offered by Snowflake, this one is closest to my professional experience and that's why I decided to tackle it as my first advanced Snowflake certification. Some topics on the exam were relatively easy for me since I have been doing data analysis for decades and SQL is second nature to me. I was able to answer SQL questions without much prior preparation. However, some topics on the exam are very Snowflake-specific that required careful review and study time. These include: Snowsight dashboards. I haven't been using them much, but I had to learn them for the exam. Snowsight dashboards can't compete in terms of features and functionality as compared to more mature tools such as PowerBI, Tableau, or Cognos, to name a few. But what they offer in their simplicity is the possibility to quickly visualize and summarize data, either for...

My Book is Available: SnowPro™ Core Certification Companion

31 March, 202322 April, 2023 by Maja FerleNo Comments

As anyone who has ever written a book might tell you, it's not an insignificant endeavor. Many evenings and weekends were spent outlining, writing, editing, tweaking and updating chapter after chapter. It was a rewarding experience and as anyone who has ever written a book might tell you after finishing it: I need a break but I'll be back with new ideas to get started on another one. The SnowPro™ Core Certification Companion Hands-on Preparation and Practice is a publication for Apress in their Certification Study Companion Series. The idea of this series of books is to help candidates prepare for various certification exams as covered in the series by breaking it down into sizeable chunks that can be studied in evenings after work. In addition to explaining the material in each chapter, exercises are provided so that the reader can practice what they have learned in a hands-on manner. One of the challenges of writing this book, about a new technology such...

Role-based Access Control (RBAC) with Secondary Roles

11 October, 202222 April, 2023 by Maja FerleNo Comments

I've written about Role-based Access Control (RBAC) in Snowflake previously. One of the topics I discussed was how to resolve cross-database access. This has now been simplified since Snowflake introduced secondary roles. I wrote in more detail about how secondary roles can be used together with RBAC in this blog post on Medium....

Snowpark for Python First Impressions

28 June, 202222 April, 2023 by Maja FerleNo Comments

Just back from Snowflake Summit 2022 held in Las Vegas. There were so many new announcements about upcoming features in Snowflake, it's hard to keep track. One topic that I'm most excited about is Snowpark for Python. As soon as I got back from Las Vegas, I started digging into the details because I had a use case waiting to be tested. I described my approach and summarized my first impressions in a blog post on Medium....