JetBrains + Weights & Biases: Establishing frameworks and best practices for enterprise AI agents

JetBrains + Weights & Biases: Establishing Frameworks and Best Practices for Enterprise AI Agents

Author: Vadim Briantov

Date: October 2023

Quick Insight: JetBrains, a two-decade veteran in developer tools, is navigating the complex frontier of AI agent development. Their experience reveals that even core AI algorithms demand rigorous observability and evaluation to function correctly, a critical problem Weights & Biases helps them solve.

💡 How are established developer tool companies approaching the new AI agent paradigm?
💡 What specific challenges arise when developing and deploying enterprise-grade AI agents?
💡 How does observability directly impact the correctness and efficiency of AI agent algorithms?

Top 3 Ideas

🏗️ Legacy Meets Frontier

"We are responsible for creating the best ideas in the world for more than 20 years and mostly we are making products for professional developers."

Developer Focus: JetBrains has a long history of building tools for professional developers. This background positions them uniquely to understand the practical needs and pain points in the emerging AI development space.
AI Agent Bet: JetBrains is making significant investments in AI agents, including their flagship Juni. This signals a strategic pivot for established tech companies into the AI agent paradigm, moving beyond simple models to complex, autonomous systems.
Observability Need: Building these advanced AI agents requires robust tools for monitoring and evaluation, a gap JetBrains addresses by using Weights & Biases. This highlights the critical need for specialized infrastructure as AI systems grow in complexity.

🏗️ Debugging the Black Box

"The most obvious outcome that we got from using uh weights and biases as observability provider for running our agent evaluation was uh figuring out that some of the core AI algorithm that we developed was was not working correctly."

Core Algorithm Flaws: JetBrains discovered fundamental issues in their core AI algorithms through W&B's observability. This demonstrates that even sophisticated AI components can have subtle bugs that only become apparent under rigorous evaluation.
Token Optimization: W&B helps JetBrains analyze token spend and optimize agent behavior. This directly translates to cost savings and improved efficiency, a crucial factor for scaling AI agent deployments.

🏗️ Strategic AI Coverage

"In JetBrains uh we make multiple bets on AI future and we have various products starting. We cannot predict the future... But I think that in JetBrains we are trying to cover to cover the whole spectrum of what the AI development could be."

Broad AI Strategy: JetBrains is pursuing a multi-product strategy to cover the entire spectrum of AI development. This indicates a proactive approach to an unpredictable future, hedging bets across various AI initiatives.
Transparency Value: JetBrains values transparency and responsive support from partners like W&B. This ensures they can quickly resolve issues and adapt their AI development pipeline, accelerating their ability to innovate.

Actionable Takeaways:

🌐 The Strategic Evolution: The rapid expansion of AI agents from research labs to enterprise production demands a corresponding maturation of development and operational tooling. This mirrors the evolution of traditional software engineering, where observability became non-negotiable for complex systems.
⚡ The Tactical Edge: Implement robust observability and evaluation frameworks from day one for any AI agent project. This prevents costly debugging cycles and ensures core algorithms function as intended, directly impacting performance and resource efficiency.
🎯 The Bottom Line: Reliable AI agent development hinges on transparent monitoring and evaluation. Prioritizing these capabilities now will determine which organizations can successfully deploy and scale their AI initiatives over the next 6-12 months.

Podcast Link: Click here to listen

My name is Vadim Briantov. I'm a technical lead at JetBrains. We have been responsible for creating the best ideas in the world for more than 20 years, mostly making products for professional developers.

Obviously, we are heavily into the AI game these days, and we also created Juni, which is our flagship AI Agent, as well as many other different AI initiatives that we are doing.

JetBrains is using Weights & Biases, specifically W&B, for observability of the AI Agents and evaluations that we are running. We use W&B to analyze what's happening in the Agent, how many tokens we spent, and how to optimize that.

The most obvious outcome that we got from using Weights & Biases as an observability provider for running our Agent evaluation was figuring out that some of the core AI algorithms that we developed were not working correctly.

By looking at the logs through the W&B console, we realized that there were some issues.

One of the key things is transparency and the ability to get the response anytime we want. We have our dedicated support team, all of them in our Slack channels, and whenever we need something, we can always get the right help and set up new products.

At JetBrains, we make multiple bets on the AI future, and we have various products starting.

We cannot predict the future, and there is a possibility that some other products might come in the future. But I think that at JetBrains, we are trying to cover the whole spectrum of what AI development could be.

JetBrains + Weights & Biases: Establishing frameworks and best practices for enterprise AI agents