SN75 Hippius :: Decentralized Cloud Storage on Bittensor :: Cheaper + Faster than Datacenters...

This episode unveils Hippius (SN75), a Bittensor subnet aiming to revolutionize decentralized storage by offering a faster, cheaper, and more transparent alternative to centralized giants, built upon its own dedicated Substrate blockchain.

The Genesis of Hippius: A Passion for Decentralized Storage

• Mark, in his introductory role, outlines his long-standing passion for decentralized storage, stemming from early crypto investments and his experience as a web developer. He highlights the critical need for robust storage solutions within the Bittensor ecosystem, especially for data-intensive subnets.
• He recounts his early, unsuccessful attempt to create a storage subnet, emphasizing the inherent difficulty of the task despite the clear end-user product need.
• The discussion touches upon the "FileTOW fiasco," an unspecified past event that underscores the challenges in implementing decentralized storage.
• Mark introduces Dubs, the Chief Architect at Talstats, praising his profound infrastructure knowledge. Dubs' expertise in Substrate (a modular framework for building custom blockchains) and decentralized storage from previous projects became pivotal.
• Mark notes Dubs' meticulous nature: "He just is one of those people who's like yeah I can do that and goes away and does it and often... does it really really well and takes quite a long time to do it but does it properly."
• A key concern addressed was the potential for subnets with their own blockchains to divert value from Bittensor's native TAO token. However, Mark also saw potential for enhanced complexity in incentive and reward mechanisms that the native chain might not initially support.

Dubs' Vision: Real Solutions for Real Problems

• Dubs introduces himself, emphasizing his passion for developing "real solutions for real problems" using Substrate and his commitment to decentralization. He sees Bittensor as a prime ecosystem for such solutions.
• His primary goal for Hippius is to offer a robust storage solution for Bittensor, addressing the reliance of current subnets on centralized storage like Amazon S3 and Google Cloud.
• Dubs states, "Basically you produce ADI [AI] with the centralized compute but the missing bits is the storage and we know that that data is the new gold."
• The Hippius team includes individuals like Radu (previously from Corsel) and Akin Shinaza, bringing diverse expertise from within and outside the Bittensor ecosystem.

Market Context and Hippius's Core Solution

• Dubs acknowledges the massive and growing demand for storage globally, driven by emerging technologies. Hippius aims to distribute storage worldwide, enabling the decentralization of AI models alongside their data.
• The solution focuses on transparency and the use of standard developer tools, particularly a Python SDK, to ensure ease of adoption.
• IPFS (InterPlanetary File System): Hippius leverages IPFS as its underlying storage technology. IPFS is a peer-to-peer network for storing and sharing data in a distributed file system, known for content addressing.
• S3 API Compatibility: Hippius offers an S3-compatible API. The S3 API is an interface for Amazon's Simple Storage Service, a widely adopted standard for cloud object storage, making integration easier for developers accustomed to it.
• Dubs highlights that while IPFS is proven, it can be "a pain to use." Hippius aims to solve this usability challenge by integrating it with a blockchain for enhanced management and trust.

Advantages of Hippius Decentralized Storage

• Cost: Significantly lower costs compared to centralized providers. Jacob (Novelty Search) and Dubs discuss how access to a broader, decentralized pool of storage capacity and miner intelligence in optimizing resources can drive down prices.
• Data Transfer Speed: Dubs argues that decentralized storage can be faster.
• Centralized providers, despite CDNs, can have bottlenecks. Hippius plans for decentralized gateways, allowing users to access resources from multiple points simultaneously.
• Erasure Coding: This technique is key to speed and resilience. Erasure coding involves splitting files into smaller, encoded chunks with redundancy, distributed across many miners. Downloads can then pull these chunks in parallel from the fastest available sources.
• Jacob questions this, noting S3's high speeds. Dubs clarifies that by splitting files (e.g., AI model gradients) and replicating them, Hippius can offer comparable upload speeds and potentially much higher download bandwidth by aggregating connections to multiple miners.
• Security & Autonomy:
• Security arises from the open network and constant "battle-testing" by miners attempting to find exploits, which Dubs views as a superior audit.
• Autonomy means the system doesn't depend on a single entity and aims to run without continuous intervention once mature.
• The system includes mechanisms to detect and prevent the pinning of harmful content (e.g., CSAM) by leveraging community reporting and AI detection, with such content hashes recorded on-chain to be ignored by validators and gateways.
• Scalability: Relies on the collective capacity of the Bittensor miner community rather than a single provider's infrastructure. Dubs praises the Bittensor mining community for their expertise and contributions to optimizing the solution.
• Dubs: "When the miners start to give insight to give help it really helps to the product because it's on their advantage because everything is more stable for them but it's more stable for us so it's more stable for the people using the products."

The Hippius Blockchain: Transparency and Governance

• Hippius utilizes its own independent Substrate-based blockchain. This chain is not intended to launch a new token but to provide a transparent, auditable ledger for all storage operations, incentives, and miner activity.
• Role of the Blockchain: It acts as the "brain," managing miner registration, data tracking, scoring, and incentive distribution. All data is on-chain, allowing anyone to run a node and verify its state.
• Bittensor Validator Integration: Bittensor validators run validator nodes on this Hippius sidechain. They observe the state of this chain (e.g., proof of storage, miner uptime) to inform their weight-setting decisions on the main Bittensor network.
• Dubs explains, "Any validator in Bittensor can run a node, uh run a validator node of the chain and and become a block producer of the chain."
• This architecture allows for more complex and granular incentive mechanisms than might be feasible directly on the main Bittensor chain.

Proof of Stored Data and Miner Uptime

• The core challenge for a storage subnet is verifying that miners are actually storing the files they claim to and are consistently online.
• Hippius uses IPFS's content addressing (CIDs, or hashes of content) for tracking. If a file changes, its CID changes, providing inherent versioning.
• Verification Process:
1. Miners register their unique IPFS node ID on the Hippius chain.
2. Validators don't just ask if a miner has a file. Instead, they request a specific, unpredictable block (small chunk) of that file. IPFS breaks files into these blocks.
3. Validators can calculate the expected hash of this specific block using the file's Merkle DAG (Directed Acyclic Graph) structure, without needing to store the entire file themselves. This is a key feature of IPFS.
• Jacob clarifies: "IPFS is a big part of this. It takes care of the Merkel DAG structure for these files."
• This method prevents a miner from faking storage or relying on a single file instance for multiple rewards. Uptime is also monitored by pinging the miner's registered peer ID.
• The system allows miners to retrieve data from other miners (peer-to-peer), which is inherent to IPFS and helps with data resilience and recovery.

Addressing UID Pressure and Scalability

• A significant innovation is how Hippius handles UID pressure. In Bittensor, each registered entity (UID) typically represents one operational unit. This can limit a miner's ability to scale their operation.
• Hippius uses a "cold key / hot key" model analogous to Bittensor's, implemented via a proxy pallet on its Substrate chain:
1. A miner registers their main Bittensor UID (linked to their Bittensor hotkey, which acts as a "cold key" on Hippius).
2. They can then create multiple "proxy" or "hot keys" on the Hippius chain, each representing a separate storage node/server in different locations.
3. All these nodes are linked back to the single main UID for incentive calculation, but their individual performance is tracked.
• This allows a single Bittensor UID to manage a distributed fleet of storage nodes, increasing decentralization and capacity without consuming excessive UIDs on the main Bittensor network.
• Mark emphasizes: "The difference with this is that each miner or each node essentially is in itself its own separate entity and therefore its performance is trackable, immutable, slashable, uh rewardable and scorable."
• This model could potentially be forked and adapted by other subnets facing similar scaling or operational distribution challenges.

Bridging and Economic Model

• A bridge to the Hippius network is planned to facilitate payments using TAO (Bittensor's native token).
• Users will interact with a marketplace on the Hippius chain, purchasing storage credits with TAO (or fiat, eventually).
• These credits are consumed as storage is used, and the TAO is then automatically distributed to network participants (miners, validators) and through a referral system.
• The team is considering Hyperlane for the bridge implementation, pending further developments in the Bittensor TAO ecosystem (e.g., potential for ERC20-like functionality).

Pricing Model

• Hippius aims for highly competitive pricing, stated at approximately $3 per terabyte per month.
• This pricing is not directly tied to current TAO emissions for the subnet but is benchmarked against other decentralized storage providers and the need to attract users.
• The goal is to offer a compelling alternative to centralized services like AWS S3 and Google Cloud Storage, which are currently used by many subnets.
• Jacob raises the point of comparing speeds alongside costs, especially against services like Cloudflare R2 (which offers free egress). Dubs acknowledges that further optimization for egress speed is a next step, with miners potentially being rewarded based on the bandwidth they provide.

Live Demo: Hippius Explorer and SDK/CLI

• Dubs showcases the Hippius network explorer, a web interface providing real-time data indexed from the Hippius blockchain.
• Features include: network statistics, individual miner details (location, storage capacity, uptime, files hosted), and a "CID tracker" to see where specific file replicas are stored across the network.
• This transparency is a core tenet: "Don't trust, verify."
• The explorer allows for analysis of network decentralization and identification of regions needing more storage providers.
• Hippius.com User Interface:
• Users log in with a mnemonic phrase (seed phrase) specific to their Hippius account.
• The interface allows uploading files, managing IPFS storage, and will include S3 bucket management.
• A teaser for future virtual machine (VM) capabilities is shown, suggesting Hippius aims to offer compute services beyond storage, potentially enabling users to run applications like Nextcloud directly on decentralized infrastructure.
• SDK and CLI Demo by Radu (Veggies):
• S3 Integration: Radu demonstrates using a standard Python S3 client (Minio) to interact with Hippius by simply changing the endpoint URL and using a Hippius sub-account seed as an API key. This highlights the ease of migration for existing S3 users.
• Python SDK & CLI:
• Installation via pip install hippius.
• The hippius CLI tool allows account login, file storage, downloads, and erasure coding.
• Erasure Coding Demo: Radu shows how a large file (1GB) can be erasure coded, splitting it into redundant chunks for robust storage and parallelized retrieval. This process is computationally intensive locally but ensures data integrity and availability.
• Direct large file uploads (without erasure coding for speed) are also possible.
• Jacob confirms he tested the SDK and achieved "incredible upload and download speeds."

Business Model and Validator Role: The Referral System

• Mog (Mark) explains the referral system, designed to incentivize building businesses on Hippius.
• Anyone can generate a revenue stream by referring customers or building applications that utilize Hippius storage.
• Validator Role: While validators are access points to the network (as in standard Bittensor), the referral system broadens value extraction. Validators participate in the network's rewards, but others can also earn by driving usage.
• Example: WordPress Plugin:
• Mog commissioned a WordPress plugin that allows WordPress sites to use Hippius for media storage.
• This plugin demonstrates the referral model: the plugin developers can earn a fee from the storage usage generated through their free plugin.
• They could also offer premium services, reselling Hippius storage credits in packages.
• This model encourages a broader ecosystem of developers and businesses to build on Hippius, extending its reach beyond the immediate Bittensor community.

Roadmap and Future Vision

• Completed: Initial launch, S3 integration.
• Upcoming:
• Desktop App: An open-source desktop application for secure, user-friendly file management and interaction with the Hippius chain.
• Fiat and TAO Integration: Streamlined payment options.
• High-Speed CDN and Gateways: Miners will run their own gateways to serve their local communities, enhancing download speeds.
• Database and Compute Services: Expanding beyond storage to offer decentralized databases and traditional compute (VMs).
• Dubs emphasizes that applications will be built by both the core team and third parties.
• The ultimate goal for Hippius is to provide a comprehensive, transparent, and decentralized infrastructure layer that can even host itself ("Hippius should run on Hippius.")
• For other Bittensor subnets, Hippius offers transparency via its blockchain (timestamping, ownership tracking) and a robust alternative to centralized storage.
• Dubs: "Make a product that's adaptable to Bittensor and not adapt a product to Bittensor to user product."

Conclusion

Hippius presents a technically robust and economically viable decentralized storage solution for Bittensor, potentially setting a new standard for infrastructure subnets. Crypto AI investors should monitor its adoption by other subnets and its cost/performance benchmarks, while researchers can explore its novel blockchain integration and UID management for broader decentralized systems.

Actionable Takeaways:

• Investors: Evaluate Hippius's potential to reduce operational costs for data-heavy AI applications on Bittensor. Track its progress in achieving superior speed and reliability compared to centralized and other decentralized storage options.
• Researchers: Analyze the Hippius sidechain architecture for its implications on subnet scalability, governance, and complex incentive design within decentralized AI networks. The approach to UID pressure is particularly noteworthy.