In the past five years, Snowflake has disrupted the database, data storage, and analytics industry by massively simplifying the entire data and analytics landscape. However, a new wave of innovation is horizon - Artificial Intelligence and AI Teammates.
This article won't discuss Generative AI, which is (shockingly) already old news, but how AI Teammates working with AI Agents can collaborate alongside your (human) Snowflake System Administrator to help maximize performance and optimize spending on your Snowflake system.
After reading this article, you’ll realize just how powerful this technology is and what a potentially massive game-changer it presents.
What is an Agentic AI?
Unless you've been living under a rock for the past two years, you'll be well acquainted with Chat-GPT (other AI Chatbots are available), but these are simply passive assistants that quietly wait for your questions and generate text.
That's not denigrate the impact of Large Language Models (LLMs), as a recent McKinsey Report indicates that generative AI will add up to $4 trillion to the global economy with up to 70% less employee work required. Indeed, Artificial Intelligence is well on the way to being integrated into almost every aspect of Data Warehousing.
However, unlike Chat GPT that passively waits for user requests, an AI Teammate works alongside existing staff to monitor query patterns, identify potential issues and offer actionable tuning suggestions. They act as an artificially intelligent advisor, perhaps using machine learning to refine recommendations based on administrator feedback.
AI teammates work autonomously to recommend real-time system configuration and processing adjustments, reacting to events and recommending remedial actions to improve Snowflake customer experience or optimize cost.
Whereas an AI Teammate provides a consulting role, communicating with System Admins in natural language, AI Agents work autonomously to handle the low-level monitoring, analyzing query patterns and resource usage. In many ways, AI Agents are the back-room workers, providing data to be analyzed and interpreted by the AI Teammates which in turn support the Snowflake Admins.
Taken together, these are often referred to as Agentic AI.
Key Features of “AI Teammates” and “Agentic AI”
It's all well and good using jargon like “AI Agents” - but what makes a solution “Agentic” as opposed to, (for example), a Python script that uses machine learning to identify and alert users of potential problems.
Here's a list of the key features an “Agentic AI” system.
Dynamic Problem Solving: Instead of relying on fixed scripts, in exactly the same way Chat-GPT can currently generate and execute python code, AI agents will generate source code and scripts to analyze data and solve problems . AI Teammates work just like a real person — assessing the situation, deciding what needs to be done, and then executing the plan.
Reduced Technical Skills: IT automation typically needs technical experts (DBAs and System Admins) to configure and maintain it through scripting. In contrast, Agentic AI allows anyone, including non-coders, to instruct the system in natural language. For example, a user could give the AI an objective like “reduce warehouse costs by reducing idle time” and the AI would figure out the best way to do that, without needing predefined scripting.
Autonomous Operation: Agentic AI doesn’t need constant supervision. Once given a goal, it will continuously monitor for opportunities to act and take the next steps without waiting for further instructions.
Real-World Interaction: These AI systems will connect with external tools and databases, pulling in real-time information, and even executing actions like sending emails or making configuration changes. For example, an AI Teammate tasked with reducing Snowflake Virtual Warehouse costs could automatically resize warehouses in real-time to maintain a balance between cost and performance.
Efficiency & Adaptability: These systems can handle complex workflows by breaking down tasks into smaller steps, calling different tools, and even calling upon AI Agents to handle specific tasks. As AI Teammates are largely autonomous, they are highly efficient needing little human interaction and constantly adapt to changing circumstances.
How can AI Teammates work on Snowflake?
AI Teammates can play a vital role in optimizing a complex Snowflake Data Warehouse deployment by automating performance tuning tasks. Here are some practical examples of how Agentic AI could be deployed:
Identifying Slow Queries: An AI Teammate could continuously monitor query elapsed times against an objective, prioritizing performance over cost. It would then suggest optimizations, such as improving partition pruning or moving a given workload to a different warehouse size.
Query Rewriting: If a query can be rewritten for better performance (e.g., using specific functions like LIMIT to reduce data scanning), the AI Teammate could automatically suggest or implement these changes after analyzing the query execution plan. Of course, this may need a “human in the loop” to verify, test, and deploy the changed code.
Virtual Warehouse Management
The screenshot below illustrates how AI Agents deployed by Altimate.AI automatically monitor and tune system configuration to identify potential cost savings and implement configuration changes to eliminate waste.
Warehouse Configuration: AI Teammates can continuously monitor virtual warehouse usage patterns and (for example), automatically adjust the AUTO_SUSPEND, WAREHOUSE SIZE or CLUSTER COUNT settings to optimize efficiency. For instance, if a warehouse remains idle longer than expected, the AI Teammate could reduce the auto-suspend timeout or automatically resize warehouses during off-peak hours to reduce costs.
New Feature deployment: Agentic AI could automatically detect and recommend potential improvements using new Snowflake features. For example, an AI Teammate tasked with maintaining strict performance targets could recommend deploying Snowflake Query Acceleration or Search Optimization Service to improve query performance. This reduces the need for System Administrators to constantly research, understand, and test new Snowflake features, reducing the dependency upon highly skilled (and expensive) resources.
Managing Snowflake Cluster Keys
Identifying Clustering Opportunities: Snowflake Cluster Keys can significantly impact Snowflake query performance, but deployment is more an art than a science. An AI Teammate could automatically analyze query patterns and data distribution to recommend adding, modifying, or removing cluster keys. For instance, if certain columns are frequently used in filter conditions but are not well clustered, the system could recommend creating cluster keys on those columns to speed up query performance.
Monitoring Clustering Efficiency: As query workloads change over time, once appropriate cluster keys may become less efficient over time. For example, if a data pipeline changes, adds frequent update operations, or changes the data cardinality, this could make clustering less cost-effective. An AI Teammate could monitor the situation and recommend or even automatically deploy changes to improve query performance and optimize cost.
Dynamic Workload Allocation
The diagram above illustrates the overall challenge faced by System Administrators. Despite constantly increasing system workloads, we need to balance the conflicting needs to maximize system throughput (for example loading and transforming data), with maximizing end-user query performance (for example, delivering query results to analytics), and optimizing spend. Agentic AI can help with this challenge by automating system monitoring and adjusting workloads as needed.
The diagram below illustrates a typical mixed workload frequently found on Snowflake MEDIUM size warehouses.
While it's not obvious, research into Snowflake Workload Segmentation demonstrates that the above workload would be more efficiently deployed over two or four warehouses, separating workloads by size and frequency. This has the potential to improve system performance and cost efficiency significantly.
Most customers deploy workload separation by team, which leads to poor utilization and high spending. AI Teammates constantly monitor jobs, recommending or even moving them to an appropriate warehouse size to achieve the best balance of throughput and cost efficiency.
For example, high-priority analytical workloads could be processed on larger warehouses while smaller, less important workloads could be shifted to more cost-effective, smaller warehouses configured for batch processing and prioritizing throughput and spend over individual query performance.
Proactive Snowflake Monitoring and Alerts
The screenshot below illustrates automated Snowflake cost monitoring across all sectors, including Warehouse computing, Storage, and Snowflake Serverless processing. AI agents can build upon basic monitoring to alert administrators proactively or take action.
Anomaly Detection: AI teammates can track the system’s performance and detect anomalies such as sudden spikes in query times, sudden increases in storage usage, or unexpected credit consumption.
Once detected, the system could either notify System Admins or attempt to automatically resolve the issue (e.g., by suggesting or implementing query optimizations).
In one case, a major Snowflake customer in the UK accidentally spent over $120,000 due to a simple manual configuration change that took a week to identify and resolve, which could have been automatically identified by an AI Teammate within seconds, potentially avoiding a significant bill.
End-to-End ELT Optimization
According to Snowflake, automated data transformation tasks account for around 80% of customer spending. However, tuning increasingly complex pipelines can be remarkably resource-intensive, often requiring an army of highly skilled data engineers.
The diagram below illustrates the challenge with a simple transformation pipeline with potentially conflicting delivery timelines. The challenge comes from optimizing data delivery within the required SLA, but avoiding unnecessary processing.
In one example, a Snowflake customer in Munich, Germany was able to reduce overall system cost by 50% simply by suspending real-time analytics during out of hours. In short, if nobody is consuming the data, there's no need to produce it.
AI Teammates can help in the following ways:
Transformations and Load Efficiency: AI Teammates can analyze transformation pipelines, looking for inefficiencies or bottlenecks. They can recommend breaking down complex SQL statements into smaller tasks or parallelizing operations to improve load and transformation speeds.
Dynamic Table Optimization: Snowflake's Dynamic Tables massively simplify and streamline transformation pipelines with opportunities to dynamically change processing frequency and warehouse deployment. AI Teammates can automatically monitor both workload size and data consumption and adjust the pipeline configuration to balance and optimize performance and cost.
The diagram below illustrates one potential tuning step in which the ETL pipeline can be automatically adjusted to match changing the frequency at which data must be refreshed. In this case, a critical data feed is refreshed every five minutes. Once deployed, an AI Teammate can automatically adjust the refresh frequency for example reducing the frequency during out-of-hours to optimize cost.
Summary & Conclusion:
AI Teammates in a Snowflake environment can act as intelligent assistants that continuously monitor and optimize various aspects of a Snowflake deployment.
They can provide actionable recommendations and potentially even implement changes themselves—whether it’s tuning queries, managing warehouse resources, or optimizing storage usage. This can lead to improved performance, reduced costs, and more efficient use of Snowflake’s powerful features, all with minimal manual intervention from human operators.
In conclusion, having spent nearly 10 years as a Senior Solution Architect much of that time working for Snowflake UK, I’ve a fair understanding of the huge challenges that Snowflake System Administrators face. I firmly believe that AI Teammates are indeed the secret to efficient Snowflake cost and performance management.
However, unlike the pessimistic view of Artificial Intelligence which warns about the machines taking over, the fact that AI Teammates work collaboratively with DBAs and Admins mean they’ll simply make an almost impossible task way more effective.
Deliver more performance and cost savings with Altimate AI's cutting-edge AI teammates. These intelligent teammates come pre-loaded with the insights discussed in this article, enabling you to implement our recommendations across millions of queries and tables effortlessly. Ready to see it in action? Request a recorded demo by just sending a chat message (here) and discover how AI teammates can transform your data teams.
