Giving New Life to Unstructured Data with LLMs and Agents

This episode unpacks the journey of taming unstructured data with AI, revealing how Large Language Models (LLMs) and intelligent agents are poised to revolutionize enterprise automation and hinting at a future of decentralized, federated execution relevant to Crypto AI.

The Unstructured Data Challenge and Early Attempts

• The speaker begins by defining Robotic Process Automation (RPA): a technology that records human click-streams on a desktop to automate repetitive tasks. RPA struggles with unstructured data because its processes are brittle and break when data formats change.
• "RPA would be fully eaten by AI automation and the future is likely going to be more of decentralized federated execution," the speaker predicts, setting a transformative tone.
• The speaker, drawing from research at MIT around 2015, defines unstructured data as "anything that cannot be put into nice database tables where you can run SQL." This includes PDFs, images, and other heterogeneous data without a predefined schema.
• The core research question was how to answer questions when data isn't in a structured format. This led to the development of "Data Hub," a project capable of mounting file systems, databases, and application nodes to query diverse data sources.
• Early enterprise attempts to handle unstructured data for processes like loan applications or immigration decisions were rudimentary:
  • Templates: Extracting data based on fixed pixel coordinates (e.g., "10 pixels below and 10 pixels from the right" for a passport number), which is highly brittle.
  • Rule-based systems: Using keyword searches, which often break.
  • Early ML models: Training models with custom features for specific document types, a very difficult task.
• The speaker's initial research into program synthesis (computers writing code, specifically regular expressions, on the fly based on input-output examples) showed promise for structured parts of documents but struggled with variability.

The Transformer Revolution and a Breakthrough

• The 2017 Transformer paper (a novel neural network architecture that relies on a self-attention mechanism to process input data, becoming foundational for most modern LLMs) and the release of the BERT model were initially exciting.
• However, applying BERT directly to unstructured documents yielded "really, really bad results."
• The speaker's team had a breakthrough by encoding not just the word's position in a sentence but also its X and Y coordinates on the document page.
  • This approach, training a model called "installm" (similar to BERT) on 110 million documents with positional and coordinate encoding, produced "great results" because the attention mechanism could understand document layout.
  • The host notes this technique of encoding 2D positional data has become somewhat standard today.
• This innovation led to significant business success for the speaker's company between 2021 and 2022.

The Rise of LLMs and Compound AI Systems

• The launch of OpenAI's ChatGPT in November 2022, demonstrating the power of scale ("the bitter lesson held: size matters"), initially seemed like an existential threat.
• However, the speaker realized that LLMs, while powerful, require "a bunch of systems before and after" to be reliable for enterprise use, referencing a DataBricks paper on compound AI systems.
• The host shares a personal anecdote of using an LLM to organize a large collection of personal PDF files by first asking the LLM to generate a document hierarchy and then classify each document.
• Strategic Implication for Crypto AI: The need for compound AI systems suggests that decentralized AI solutions will also require robust, multi-component architectures rather than relying on a single monolithic model, especially for complex decision-making.

Enterprise Use Cases: Beyond Simple LLM Queries

• The speaker describes a common enterprise use case: processing a 100-page loan application packet, which can contain varied documents like bank statements and even irrelevant items like "cat's picture."
• Banks need to verify income and identity reliably, without errors.
• Challenges with naive LLM application:
  • Context window limitations: LLMs can only process a certain amount of text at once.
  • RAG (Retrieval Augmented Generation): A technique where relevant data chunks are retrieved from a vector database (a database optimized for storing and querying vector embeddings, often used in AI) and fed to an LLM. While RAG can improve precision, it might miss crucial information, impacting completeness.
  • Surprising errors: LLMs can make unexpected mistakes, like missing random cells in a table within a bank statement, which can alter outcomes significantly.
• The speaker emphasizes that "LLM is not all you need." A reliable solution involves:
  • Splitting the document packet into meaningful components.
  • Running specialized algorithms (e.g., table-to-text) for specific structures.
  • Classifying documents and extracting data against relevant schemas.
  • Validating extracted data and performing cross-validations (e.g., comparing W2 forms with pay stubs).
• The speaker's platform allows building these complex applications without coding, enabling processes like loan approvals in seconds instead of weeks.
• Another use case is intelligence analysis, where millions of documents are processed daily to detect threats. Instead of just dumping data into a RAG system, a more robust approach involves analyzing every page, extracting specific threat indicators, storing them in a database for SQL queries, and then performing deeper analysis on matches to ensure completeness.
• Actionable Insight for Researchers: The limitations of standalone LLMs highlight opportunities for research into hybrid systems that combine LLM strengths with symbolic reasoning, structured data extraction, and verification mechanisms, particularly relevant for high-stakes crypto applications.

Reliability, Predictability, and Human Oversight

• The speaker argues that AI systems don't need to be perfect, but they do need robust error handling and escalation paths to humans, similar to how human processes operate.
  • "AI is not supposed to work reliably 100% of the time. You have to build a system around it," the speaker states.
• The host raises the shifting enterprise mindset: instead of demanding absolute perfection from AI, some are comparing AI performance to well-trained humans and aiming for AI to be significantly better, acknowledging that even humans make errors.
• The speaker clarifies that predictability is more important than absolute accuracy for enterprises.
  • "People are fine with errors as long as errors are predictable. When errors are not predictable, that's where the problem is."
• Enterprises can accept 80-90% accuracy if they know which 20% needs review. This requires tooling and systems around AI to detect and explain errors.
• Strategic Implication for Crypto AI Investors: Projects focusing on explainable AI (XAI) and auditable AI systems are likely to gain more traction in regulated or high-value crypto use cases, as predictability and transparency are key for adoption.

The Future of Document Interaction and User Experience

• The speaker envisions a future where humans interact with dashboards summarizing AI-processed documents, only diving into details for items of interest, much like using Google search results. AI will reduce boilerplate and surface essentials.
• The host humorously notes the potential for a cycle: one AI generates a PDF from key points, and another AI reduces that PDF back to key points.
• An interesting use case from a bank in India: offering entire lending processes over WhatsApp. Customers upload documents conversationally, leading to a "fundamentally very different" customer experience.
• The speaker believes AI's biggest impact will be creating new classes of customer interaction, moving beyond AI as just an internal software technology. Processes like insurance claims and even immigration can become more interactive and real-time.
• Actionable Insight for Crypto AI Developers: Focus on user experience (UX) driven by conversational AI can unlock new applications in DeFi, NFT marketplaces, or DAO interactions, making complex crypto processes more accessible.

Barriers to Enterprise AI Adoption

• Enterprises are historically slow-moving, though the speaker notes they are moving "a little quicker in the AI revolution."
• Key concerns for enterprises boil down to:
  1. Data safety and security.
  2. Auditability and predictability.
• Enterprises need to explain AI decision-making processes, especially if errors occur, similar to how human errors are traced and explained. AI cannot be a "black box" for customer-centric use cases.
• Strategic Implication for Crypto AI: Addressing governance, data sovereignty (especially in decentralized contexts), and transparent, auditable decision-making will be crucial for broader adoption of AI in crypto ecosystems.

The Role of AI Agents: Compile-Time vs. Runtime

• The host introduces AI agents (software entities that can perceive their environment and act autonomously to achieve goals) as a hot, sometimes overused, term.
• The speaker draws a parallel to existing enterprise workflows: developers create defined workflows that run predictably.
• A problem with current agents is their potential for non-deterministic behavior: given the same goal and tools, an agent might choose different paths at different times, which enterprises dislike for runtime operations.
• The speaker proposes a distinction:
  • Compile-time agents: AI agents can generate the first draft of a workflow or code. Humans review, edit, and approve this artifact. "Cursor is a great example," the speaker notes, referring to an AI code editor.
  • Runtime execution: The human-approved, deterministic artifact runs in production, ensuring auditability and debuggability.
• "I do not believe that autonomous agent would be a runtime phenomena. However, there would be a build time or compile time phenomena," the speaker asserts.
• This approach leverages AI for development efficiency while maintaining control and predictability in operations.
• Actionable Insight for Crypto AI Researchers: The compile-time vs. runtime agent distinction offers a pragmatic framework for designing DAOs or other autonomous crypto systems. Agents could propose strategies or code, which are then ratified (compiled) by token holders before deterministic execution on-chain or via smart contracts.

Future Vision: Federated, Decentralized AI Execution

• The speaker believes AI will continue to improve, playing a significant role in compile-time tasks (building, reasoning).
• Regarding execution patterns, two views exist:
  1. AI simplifies data management by moving everything to one place.
  2. Data remains siloed, and AI becomes smart enough for multi-agent communication and error handling across these silos.
• The speaker's company is working on Federated AI Execution: a framework where an organization can define thousands of agents in a federated way. These agents can dynamically discover each other and communicate to achieve larger goals without central orchestration.
  • "The bet that we are taking is that AI will drive automation in a significant way. RPA would be fully eaten by AI automation and the future is likely going to be more of decentralized federated execution."
• This vision involves many open questions and challenges but points towards a future of decentralized automation frameworks.
• Strategic Implication for Crypto AI Investors: The concept of "federated, decentralized AI execution" aligns closely with the ethos of Web3 and decentralized systems. Investors should look for projects exploring these architectures, as they could form the backbone of future AI-powered decentralized applications and organizations.

Impact of AI Advances on Service Delivery and RPA Replacement

• Historically, the speaker's company focused on the unstructured data problem – getting data into a structured format for the next decision step, but not the decision itself.
• Other systems, often using RPA (Robotic Process Automation), handled subsequent steps. RPA records human actions on a desktop (opening browsers, inputting data, clicking buttons) to automate tasks. It's brittle, especially with unstructured data inputs, but can work if subsequent steps are consistent.
• The speaker argues that AI can now potentially operate these downstream systems as well, moving towards end-to-end AI automation that could replace RPA.
  • This assumes AI can effectively interact with various systems, possibly leveraging emerging protocols like MCP (Model Context Protocol), which allows dynamic discovery of capabilities and function calls, though MCP still has issues like authentication.
• A potential hack for permissions is identity pass-through, where an AI agent operates under a user's identity. The host raises a valid concern about granting agents full user capabilities.
• The speaker reiterates that such decisions (e.g., limiting an agent's permissions) should be made at compile-time by humans, ensuring the runtime behavior is controlled and predictable.
  • "This problem comes when an AI agent is making runtime decisions because then you have no control where things are going. Uh so the separation is critically important."

Concluding Thoughts on Enterprise AI Adoption

• The host likens the current AI wave to the dot-com boom, emphasizing the need for enterprises to adopt early despite complexities, or risk obsolescence.
• The speaker agrees, highlighting three key benefits of AI adoption for enterprises:
  1. Significant cost savings.
  2. Increased speed and efficiency.
  3. Fundamental changes to customer experience.
• The primary question for enterprises is not whether to adopt AI, but how to make it work effectively and reliably.

Conclusion: This episode underscores AI's transformative power in managing unstructured data and automating complex enterprise workflows, with a clear trajectory towards more intelligent, potentially decentralized systems. Crypto AI investors and researchers should monitor the evolution of compound AI systems, compile-time agency, and federated execution models as key indicators of future decentralized AI infrastructure.