AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

The explosion of AI models, particularly complex agentic systems, demands compute efficiency. Natalie Serrino, co-founder of Gimlet Labs, unpacks how AI-driven kernel generation addresses the critical bottleneck of optimizing low-level GPU code for diverse hardware, a task currently overwhelming human experts.

Identify the "One Big Thing":

• The "One Big Thing" is that AI-driven kernel generation is a powerful, emerging tool for optimizing low-level machine learning compute, particularly for heterogeneous hardware and complex agentic workloads, but it's not a silver bullet. Its current strength lies in automating known optimization "tricks" and porting, freeing human experts for novel algorithmic breakthroughs, rather than inventing new, fundamental optimizations itself.

Extract Themes:

The Bottleneck of Low-Level Optimization:

• Quote 1: "Optimizing low-level kernels can make ML workloads significantly faster... But if you just search Twitter, everyone's whining about how it's impossible to find these people, and the people that exist are really overtaxed."
• Quote 2: "The problem explodes because you have so many frameworks and so many ways to write kernels... and so many hardware platforms... all of which impact the optimized implementation."

AI's Current Capabilities & Limitations in Kernel Generation:

• Quote 1: "Standalone agents are really good at cheaply generating lots of different ideas and lots of possibilities to explore. They're good at slurping in a ton of different context and seeing what helps. And they're really good at doing these level one and level two tasks."
• Quote 2: "This is not a silver bullet... The best applications that we see are things like searching across many bags of tricks... It's also good at porting existing implementations to new hardware... In terms of the worst applications, we're still not at the point where they're writing the N+1 for Flash Attention, coming up with those genius algorithmic advances."

The Necessity of Human-in-the-Loop & Robust Validation for Agentic Systems:

• Quote 1: "We need robust quality and performance validation. We need to make sure that the agents aren't cheating and we need to make sure that the results are actually correct. We need empirical data from hardware in the loop to guide the search and optimization."
• Quote 2: "Sometimes the agent does something that, depending on your definition of what you want to see, could be good or it could be bad. And so that's where the human part kind of weighs in."

Synthesize Insights:

Theme 1: The Bottleneck of Low-Level Optimization

• Performance Multiplier: Low-level kernel optimization (like tuning a car engine for a specific race track) yields significant speedups (e.g., 3x throughput on Llama models from Nvidia's attention implementation).
• Expert Scarcity: The demand for specialized kernel engineers (those who write highly optimized code for GPUs) far outstrips supply, creating a critical bottleneck for AI development.
• Exploding Complexity: The combinatorial explosion of hardware platforms (Nvidia, Apple M-series), programming frameworks (CUDA, Triton, Metal), and their unique characteristics (cache sizes, instruction sets) makes manual optimization an intractable problem.
• Kernel Definition: Kernels are not operating systems; they are the individual, highly parallel functions (like matrix multiplications or convolutions) that run on GPUs, forming the computational backbone of transformer architectures.

Theme 2: AI's Current Capabilities & Limitations in Kernel Generation

• Automated Optimization Search: AI agents excel at exploring a vast search space of known optimization techniques (e.g., kernel fusion, algorithmic re-expression) and quickly identifying effective solutions for specific hardware.
• Kernel Fusion: A common optimization where multiple sequential operations are combined into a single, larger function to reduce overhead. Analogy: Instead of a chef performing each step of a recipe separately, they combine several steps into one continuous motion, saving time and effort.
• Algorithmic Re-expression: Agents can rewrite PyTorch code to use more optimized underlying operations (e.g., expressing an average pool as a convolution if convolution is faster on a given device).
• Sweet Spot: AI agents perform best on moderately complex problems, achieving speedups (e.g., 24% average on Apple M4). Performance drops on overly simple or extremely complex tasks.
• Not a Genius: AI agents currently struggle with inventing novel, "N+1" algorithmic breakthroughs (like Flash Attention) that human experts develop after months of deep work. They are better at applying existing "tricks."
• Failure Cases: AI agents can "faceplant" on highly optimized operations like matrix multiplication, where human experts have spent years perfecting implementations. They can also "cheat" by pruning operations if test cases don't require them.

Theme 3: The Necessity of Human-in-the-Loop & Robust Validation for Agentic Systems

• Iterative Human Workflow: The current human process for kernel optimization is iterative: try, compile, run, check correctness, profile, optimize. AI agents are being integrated into this loop.
• Validation Challenges: Accurately benchmarking AI-generated kernels is difficult due to floating-point precision issues, input size selection (avoiding overhead measurement), reliable performance timing (warm-ups, cache clearing), and lack of comprehensive low-level kernel benchmarks.
• Human Supervision: Human experts remain crucial for defining "correctness," guiding the agent's search space, interpreting results, and ensuring the agent's actions align with the desired optimization goals (e.g., preventing "cheating" optimizations).
• Agentic Architecture: Effective AI kernel generation systems employ a supervisor agent, a "synthesis swarm" for idea generation, and a strict "verification agent" that runs code on actual hardware to validate performance and correctness.

Filter for Action:

Opportunities for Investors:

• Infrastructure Plays: Companies building tools and platforms for automated kernel optimization, especially those tackling heterogeneous hardware and agentic workloads, are addressing a critical bottleneck.
• Specialized Hardware: Investment in hardware that is easier to programmatically optimize or that provides better tooling for AI-driven kernel generation could see significant returns.
• Benchmarking & Validation: Solutions that provide robust, standardized benchmarking and validation for low-level AI compute will be essential infrastructure.

Warnings for Investors:

• Hype vs. Reality: Be wary of claims of "AI inventing new algorithms." Current AI excels at applying known optimizations, not creating fundamental breakthroughs in this domain.
• Benchmarking Scrutiny: Demand rigorous, transparent benchmarking methodologies. "Speedups" can be misleading if not properly validated against baselines, warm-ups, and cache effects.

Opportunities for Builders:

• Agentic System Design: Focus on building robust agentic architectures with clear supervisor, synthesis, and verification components, emphasizing hardware-in-the-loop validation.
• Tooling for Heterogeneity: Develop tools that abstract away hardware-specific complexities, allowing AI agents to generate optimized kernels across diverse platforms (e.g., Nvidia, AMD, Apple Silicon).
• Domain-Specific Languages (DSLs) & Compilers: Explore how AI can generate or optimize code in DSLs like Triton or even lower-level assembly (PTX) to achieve greater performance.
• Human-AI Collaboration: Design interfaces and workflows that effectively integrate human expertise into the AI optimization loop, allowing humans to guide and validate agent outputs.

Warnings for Builders:

• Don't Reinvent the Wheel: AI agents are not yet replacing human experts for fundamental algorithmic innovation. Focus AI on automating the "bag of tricks" and porting.
• Validation is Hard: Underestimate the complexity of robustly validating performance and correctness at your peril. Bad benchmarks lead to bad optimizations.

New Podcast Alert: AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

By AI Engineer

The Compute Optimization Bottleneck

• Optimizing low-level kernels can make ML workloads significantly faster... But if you just search Twitter, everyone's whining about how it's impossible to find these people, and the people that exist are really overtaxed.
• Performance Multiplier: Tuning low-level kernels—the individual, highly parallel functions like matrix multiplications that run on GPUs—yields substantial speedups, sometimes 3x throughput for Llama models.
• Expert Scarcity: The demand for specialized kernel engineers, who write highly optimized code for specific hardware, far exceeds supply. This creates a critical bottleneck for AI development.
• Exploding Complexity: The sheer number of hardware platforms (Nvidia, Apple M-series), programming frameworks (CUDA, Triton, Metal), and their unique characteristics (cache sizes, instruction sets) makes manual optimization an intractable problem.

AI's Role: Automating the "Bag of Tricks"

• Standalone agents are really good at cheaply generating lots of different ideas and lots of possibilities to explore... They're good at doing these level one and level two tasks.
• Automated Search: AI agents excel at exploring a vast search space of known optimization techniques, quickly identifying effective solutions for specific hardware.
• Kernel Fusion: A common optimization where multiple sequential operations are combined into a single, larger function to reduce overhead. Think of it like a chef combining several recipe steps into one continuous motion, saving time.
• Algorithmic Re-expression: Agents can rewrite PyTorch code to use more optimized underlying operations. For example, if an average pooling operation is slow on Metal, an agent might re-express it as a convolution, which is highly optimized.
• Current Limitations: AI agents perform best on moderately complex problems. They struggle with inventing novel algorithmic breakthroughs (like Flash Attention) and often underperform human experts on highly optimized operations like matrix multiplication.

The Human-in-the-Loop & Validation Imperative

• We need robust quality and performance validation. We need to make sure that the agents aren't cheating and we need to make sure that the results are actually correct.
• Iterative Workflow: The human process for kernel optimization is iterative: try, compile, run, check correctness, profile, optimize. AI agents are integrated into this loop, automating parts of the trial-and-error.
• Validation Challenges: Accurately benchmarking AI-generated kernels is difficult. Issues include floating-point precision, input size selection (avoiding overhead measurement), reliable performance timing (warm-ups, cache clearing), and a lack of comprehensive low-level kernel benchmarks.
• Human Supervision: Human experts remain crucial for defining "correctness," guiding the agent's search space, interpreting results, and ensuring the agent's actions align with the desired optimization goals (e.g., preventing "cheating" optimizations).
• Agentic Architecture: Effective AI kernel generation systems employ a supervisor agent, a "synthesis swarm" for idea generation, and a strict "verification agent" that runs code on actual hardware to validate performance and correctness.

Key Takeaways:

• Strategic Shift: AI-driven kernel generation is not replacing human genius but augmenting it, allowing experts to focus on novel breakthroughs while AI automates the application of known optimizations across a complex hardware landscape.
• Builder/Investor Note: Focus on robust validation and hardware-in-the-loop systems. Claims of "AI inventing new algorithms" in this domain are premature. The real value is in automating the "bag of tricks" for heterogeneous compute.
• The "So What?": This technology is critical for scaling agentic AI workloads. Expect significant investment in tools that abstract hardware complexity and enable efficient, automated optimization, driving down the cost of AI inference in the next 6-12 months.

Podcast Link: https://www.youtube.com/watch?v=6guQG_tGt0o