Hash Rate - Ep. 158 - Crunch Brings 11,000 AI Experts To Mine Bittensor

Hello everybody and welcome to hash rate. My guest today is Jean Harell of Crunch. Hello Jean. How you doing?

Hi Mark. I'm good. And you?

I'm doing pretty good. Just kicking my claw miner to keep the stupid thing up and running which we'll get into that later. But you are bringing brainiac human miners into the Bit Tensor ecosystem. What I've read is 11,000 machine learning engineers and 1,200 PhDs all are now able to mine TOAO. Is that correct?

Yeah, that's correct, Mark.

So, okay. So, tell me a little bit about just let's just go back and talk a little bit about Crunch because Crunch has been around for a little while and what sort of interested you in the Bit Tensor ecosystem and what are you doing now? So, just sort of walk me through the story briefly.

Yeah. I mean obviously this number took some times right we didn't get there overnight it's been four years that we're building on Crunch and just a bit of a backstory here I started building AI models in the insurance industry at first and when we exited that first business with my partner we decided to use the same kind of AI algorithm to launch a long short hedge fund so a very complex investment vehicle that would be using AI and mathematics to place all the orders of the phone and that was much much more difficult than anything we would have done in the insurance industry.

Because this problem is very very hard to solve extremely competitive like you compete against huge firm like Citadel Millennium two Sigas and you know other teams of hundreds of quants and as little as a little bit of information left anywhere on data is already exploited and is already mine by this big hedge fund and so very complex and we were really struggling.

So back then 2020 I was seeing a project from overseas called Numera that started to pay a crowd of data scientists to help them doing exactly what we were trying to do. And so I decided to follow on and I decided to start you know put some cryptobonties on our website and try to talk about it on Reddit and on LinkedIn etc. And it actually grew pretty fast like we ended up with couple of hundred or maybe a thousand people already interested let's say in the first couple of months and I experienced what probably many subnet owner experience today on bensoor which is people are going to beat you they're going to be much better at building AI algorithm than what you can internally and especially if you have a competitive framework and especially if you have the right incentives.

So pretty early I realized that that I was really relevant anymore in building this algorithm. I better just focus on finding more brains and yeah so Crunch really started out of our own limitation really like we were we just realized that there is people out there smarter than you and that applied to everyone on the world right there's always the smartest person the goal is just how can you reach that person and that's what became really my focus for the following years so we continued on that path for a year or two we reached 2,000 3,000 people helping us on the phone and along the way I started to receive interest from some big institutional.

So I'm talking about a large research lab of a sovereign wealth fund in UAE operating a very large quant team the one that I was actually competing against and they reached out and they were basically interested in tapping into into this crowd into this talent that we managed to gathered around around our idea and phase two of Crunch kind of started at that moment where we started to share the people that was sending algorithm into Crunch and that was 2023. 2024 moved on and we started to get more you know more interest and we ended up you know providing algorithm for the broad institute at Harvard and MIT and other big institutions you know that are actually extremely interested in finding this talent and trying to improve what they're doing internally because they already invest a lot of capex into this operation.

So what we do with Crunch is we really solve pretty well this talent aggregation you know where on the other side we're looking at beno that really near pretty well the capital formation and the decentralization aspect of it and I think the the capital formation is kind of a consequence of the proper decentralization right it works really well people got people didn't had to trust they could really see what was happening everything was well thought through and so Bitensa managed to attract a lot of capital and so today Crunch didn't manage to do that right we could not build this decentralization because we were so focused on talent and really trying to solve that part of the problem that we kind of let centralization aside and so for us recently I mean it's been maybe a year that I'm really looking into starting to mine into Penso and so maybe we're going to deep dive further into that but it's been a year that we were looking at what the subnet were doing and can we consider a subnet as kind of a new type of customer, right? A decentralized customer.

Can we consider mining as a profitable economic activity? And so that's what took us a bit of time to really gain trust into until more recently or where we really decided to move on where we see some subnet really maturing and and really fitting the kind of problem that Crunch can help solving basically.

Can I interrupt you for a second because I I have a number of questions, but I just want to make sure I clarify some of this first. So So you know, starting four years ago, you basically said, "I'm going to set up a competition for people basically a subnet before, you know, before we had subnets really, right? So without Bit Tensor, you set up a subnet and you basically put up crypto bounties for solving AI problems to be used in trading algorithms."

But you were basically crowdsourcing the best AI brains on earth and they showed up in force and with massive amounts of talent probably much greater talent and much it was probably a lot more effective than you thought it might be right and then I think every subnet owner has kind of seen this right so the the world is just full of really smart people weirdly you you wouldn't think so looking on Twitter but it is and if you just if you put like the little bounty up you know here's some cash come get it if you solve my problem, the world just sort of swarms on it, right?

So, you saw that you went, "Wow, this is awesome." And then you then you basically discovered Bit Tensor and you're like, "Oh, this solves, you know, we didn't decentralize, but this solves a lot of problems that we now don't have to solve because the architecture already exists." And for the for the people who are contributing to your subnet, they know deterministically that they will get rewarded if they solve the problem the best.

So that the sort of trust problem of how the judgment happens I guess goes away and I guess maybe there I guess because you were centralized there was some concern about that. But but now that if that concern existed it just goes away completely and so now you're like wow so I've got this sort of army of people I can maybe turn it onto the problem of mining rather than trading. Does that capture it correctly?

Yeah. I mean there is this vision of what do you do with this enormous amount of talent and that's really the question we were asking oursel and it's already connecting with further question when we talk about AI agent and what's going to be the position of human and and you know stranded talent and and liquidity of work and all this kind of of um of subject that nobody has answered these problem are not solved yet you know the humanity is really facing this kind of this kind of problem at the moment and so Crunch has is very institutional and very big customers right we talk about trillions of dollar type ofum with this kind of customer and so um this is these are customer that we're going to continue to work with we're going to continue to serve um you know um a big institutions but we also have for the first time I think realized that there is maybe you know some subnet within the potential ecosystem may become really big as well and so may become this like massive um potential customer for us right and so the only difference with um you know with a beatens of subnet is that you are paid by performance right whereas slightly different with more institutional customer.

I do think that the problem of decentralization we need to solve it at Crunch as well and so it's going to be slightly different it's just that we need to make sure and by the way we have we have a test net now it's on Solana because we don't have to launch our own chain and this kind of thing we wanted something very light but we have to enable this 11,000 people to track down you know the IP that they are providing right and they need to make sure that what we do on Benso is extracting you know um um um rewards that go straight back to them and so this is really what we want to implement here we're going to we are on test net now and we have this kind of layers that allows you know both subnets and data scientists from Crunch to track what is happening right so they'll be able to know that a model is being deployed that the model has been attached to a minor and now this minor is generating this amount of rewards and you know is receiving that amount of reward minus what we have been spending on compute in order to run the models that we've been gathering and so Crunch become this abstraction layer of decentralization and the complexity of decentralization while maintaining it and forcing it right I don't want a recentralized thing because otherwise you kind of lose trust like you say so we are this simplification layer now for enabling these guys to join benso a lot of people wanted to join for a while but we're kind of stopped by the cvolution let's say of the of the of the of the very strong coordin ation layer that is needed right that is one of the reason why the capitalization works so well it's needed but we can abstract it with Crunch and enable kind of a uniformization layers for these data scientists to go very quickly and to kind of assess the opportunity for them where they're going to be good let's say you are a bio researcher you know you're going to be good at this and this subnet or if you're more into time series you know financial prediction problems are going to be the right one if you're computer vision scientist, you're going to look at, you know, different type of subnets.

So, it allows this kind of uniformization and very simple interface. They don't have to relearn the new rule of another subnet. And, you know, it's the same submission, it's the same library while being decentralized. And that's that's really the vision there.

Well, so if anyone who knows me knows that for the last week and a half or so, I've been using an open claw, right, to vibe mine Bit Tensor. And um with some success and some failure, I would describe where where I am right now. I'm vibe mining Subnet 85 and 44, but 85 is sort of my lead fighter. And it's a very complex thing to mine. There are there are four instances on various infrastructure providers that are coordinating with one another to both compress and upscale video in Vid IO which is subnet 85. And I but I haven't been able to do it profitably yet.

It's basically this process of my claw sets things up and then we wait 24 hours basically for the validators to kind of you know do thumbs up or thumbs down us and I keep getting dregistered and kicked out of the system and then I send my claw and I'm like I'm like okay what happened and why did that happen and sometimes it's a port mismatch other times it's a process crash or whatever and we have to fix it and try again right so it's been this sort of iterative loop it's not basically the whole point of this is even with a blah even vibe mining even with a magical computer that does everything for me mining is hard and so if you're not and especially if you're not familiar with the bit tensor ecosystem so you have all these brains right these 11,000 machine learning engineers of which some of them are PhDs they're very good at what they do but they don't know bit tensor there is no sort of mining portal like you know how do I where do I even start how do I even get into this world right so so you are building you're sort of building that last mile track for those people and and you released this sort of subnet mining hub which I want to show that you let's talk about this real quick and I think this is how you're onboarding people.

So what am I looking at here? What is this?

So, so basically like you just said before Mark, we we you don't want to deploy a subnet on a on a simple problem that you already that is already solved, right? Otherwise there is no reason in deploying decentralized intelligence and you know hundreds of people trying to make your algorithm better. Now within this complex problem Crunch would not be a fit for everything because benso became so I mean implement such a strong coordination that you could coordinate anything right now I mean you could do anything on benso you don't have to necessarily do predictive statistics you know on you could provide data you could provide compute you could do many things right Crunch is really a community of data scientists and So for us we kind of try to come with a to the Crunch fit index that is ready for us.

I mean we're not talking about the quality of a subnet or anything right? It's only about u how much of a fit we believe it is according to some rational that are sometimes wrong. And by the way please if you are operating a subnet and you see some mistake in this in this dashboard please come to us. we try to you know to gather that information but there is some mistake eventually and so what you see the ML fit is and the Crunch fit score I mean the CFI is really kind of this mix of okay is it a predictive task does it need the cruncher and talents you know to be solved properly or it's more a capital or it's more access to hardware kind of problem if it needs talent it's an ML problem and finally it has interesting reward and daily reward which is sometimes kind of hard to understand.

So we try to help you know trying to quantify how much daily reward you have on the subnets and so when you have these three these three factors and you there is a strong recommendation for Crunch. So um this is also a call for subnet. When we have subnets that are you know that are you know interested in in like triggering our our interest they can click start node or actually some people from the community and that's why our our identity is distributed and decentralized like this dashboard is for some of our data scientists eventually you know looking at this dashboard and say okay I really love what a score is doing or what Zeus is doing and I would like to you know to you know to crunch myself on it and so when they click Start node we really handle everything.

You don't have to touch the blockchain. You won't have to even set wallet. Cruncher have already wallet in their in their space and they will be able to launch a node and that's basically what we try to do because um we we are going to you know we going as core team into some subnets but we also want it to be open you know and and if some cruncher want to help us scale and do things faster please be our guest.

Yeah. Okay. So let's let's go over this. Right. This is awesome by the way. I love this. This is the first time I've seen all this information for mining kind of collected in one place. Um, so there there's there's your specific metrics, but I love this here, this daily dollars. So, you know, Barry Silbert posted something this week where he said, you know, Bitensor Bitensor is handing out a hundred million a year to miners, right? Which which that it sounds crazy, right? But that is in fact what the chain is producing, right? Just like Bitcoin hands out a certain number of Bitcoin every year to their miners, Bit Tensor does to subnet miners as well, right?

So, what does that break down into in sort of daily take? Well, Synth is handing out 3.2K a day, right? Down here, Score is handing out about 20K a day. Shoots is handing out almost 60K a day. Holy crap, right? So, um but then there's the, you know, how well does this fit a h a human minor, right? Some of these subnets aren't really human minable or it doesn't really matter that much because they ask you to supply compute or they ask you or bitcast is you know make a video on YouTube right not really machine learning generally speaking right so not a good not a good fit so this is um so the CFI of 88 I guess this is the best fit Numinous um but subnet 50 I I see all these start nodes so and then I see this mine So, if I click on if I click on mine, what does that do?

If you click on mine, you're going to be redirected to the Crunch Hub. And this is where you can mine without having to even start a wallet. So, this is really the strength of Crunch. Here you have PhD data scientists, people from TRA data science that never touch crypto and actually hate it. You know, they see that they'll be like, "Oh, you know, I like AI, I like ML." but they be kind of you know going away when they see like there is crypto incentives and etc. So if you go up and you click on launch a model um you basically need something like 3 minutes to uh for Crunch to create your wallet and let you start mining since so in three minute you're capable of mining s and okay you need to create an account we need your email and etc but in three minutes you are like all set up and so this is really the power of Crunch and I've seen some tweet last week where I was seeing some people saying like I I I don't know if you've seen this tweet And it's like I wanted to to mine sin for a while and I'm finally capable of doing it.

And so that's really what we want. And at the same time I think there is this Crunch is going to be a bit more patternalistic with the data scientist world where we are assessing who is performing or not and then we will automatically take care of the trading of the mining slots and etc. And so the data scientists on our side don't have to put this skin in the game at the beginning that also is a big barrier for a lot of people to participate right some students some researcher don't have $300 to start mining and for them it's a big amount of money and especially in certain part of the world and so Crunch is abstracting that by okay submit on Crunch we're going to look at everyone's performance we monitor when a model have a positive mining ROI and when you do then we automatically open a minor and try to, you know, try to push your model there. And maybe you get dregistered or not, but this is this basically where we want to go.

So you make it so so basically all the crap that I'm dealing with with my claw like you just that that just goes away. So So let's do a test with your cloud. This week uh basically yesterday we released um skills and code skilled for Crunch. And so this time your agent will just have to deal with only the data science part and submitting. So you remove maybe all the complexity related to trading the mining slots and you know and interacting directly with beno is abstracted you can submit and then we directly submit it uh for you behind and that's I think it will be easier now that's something that is very core for Crunch like human I do believe that human are still going to be very important to solve these key these high stake problems.

And why? Because LLMs are basically designed to to and this model they are designed to kind of converge to the norm and they end up you know coming up with solutions that are mostly already known or already used and when you run a competitive subnet there's this is already absorbed in about a week with the right incentives all the solution out there are already played and are already tried and everything have been tried already. So this fact is already this is already absorbed. So the new benchmark is the latest frontier model right that's the benchmark of all humans and especially for data scientists.

Now the people that are relevant are the people that can add this non-consensus intelligence on top of models right we can you come with a new idea you come with a new way to operate cloud and you come with a new way to you know to orchestrate or design your models that is going to outperform all the other benchmark and this is what the institutions are ready to pay a lot of money and this is also what can bring a lot of money when you're mining on bensor as well right because if you like overperform everyone we've seen the kind of reward that you that you can get a shot at. And I believe that this is going to be accelerated overall.

Because um this the price of the next unit of intelligence above benchmark is going to become very clear like in the past it was complicated because intelligence was illquid right you hire someone you take a PhD in your team you you you know you train him he grows and they build some models and you don't know what the other organization are doing and how performant they are you know and it's difficult to position yourself and and your employees. whereas in this kind of very liquid talent marketplace where intelligence is completely fluid you you know directly like who is the best models right and so that's why there is this benchmark everywhere because human I'm like are like what what should I use to be average intelligence right should I use grock should I use cl should I use chpt what is great with this competitive framework is that in this abundance of noise we have the signal very clear now because everybody's competing against each other.

So you know what's noise under the benchmark and you know what's signal above the benchmark. And if your signal is on a high stake problem that has a lot of value, there is a huge economic upside to grab.

No, I mean using my claw like I have a little pull down menu on the left where I can use the latest claude or I can scale back to like Kimmy, right? the sort of the the junkier models are kind of down here and they cost a lot less to run but they my my experience with them is they are they are marketkedly stupider right like you're trying to solve a problem on a subnet so or anywhere like this kind of problem are complex to solve if you don't scale your model and everyone else is already using the frontier you are behind the benchmark yeah no I feel it in my bones when I don't have cloud running like it's it's it's like I it's just everything's slower and it doesn't find the answer usually But if I put cloud up there, it's pretty fast and it generally finds the answer really, you know, pretty quickly.

Now, humanity still have two edge right now. one edge is the fact that they steal a lot of domain knowledge and experience you know knowledge is the sum of skills and experience right so that's I don't remember which who say that but um humans still have you know this experience bucket that you have in some jobs right so if you try to do computer vision now and you try to compete even with the best model against someone that has been doing that since the 90s it's going to be difficult right I'm thinking about Yanlukan for example since the '9s has been working on this kind of things. So you're going to be if you take him plus a machine is going to be outperforming you for sure and so the machine is just the new benchmark and someone that knows nothing about that field should have the performance of the highest models and so that asked a lot of other question which is okay how do I access to this latest level of intelligence and what's the price so it means that potentially if I cannot pay enough tokens because I mean you've used in cloud now it's quite expensive right so yes it means that I mean if I go further than the context of collective intelligence and competitions and in everything in life.

It means that people that cannot afford that you know these tokens are going to maybe perform you know less in a world where uh the latest models in the benchmark. So I also believe there is tons of value in this decentralized AI vision where we try to make sure that there is no single entity that is owning the stake or owning the pricing scheme behind that and we try to keep you know open source alternative that close to the edge possible and I think is doing great as well in that in that regard for sure.

No agreed. So let me let me just clarify some stuff about Crunch because I just want to complete the loop here. So, if I'm in your community of um of data scientists, of machine learning engineers, of PhDs, and I use the interface that I we just saw, how am I getting my money? Like, do I do I give you a credit? Do I give you like a bank account? Do you send me stable coins? Like, how how do I actually get my do?

So, right now, we send stables. We send stables. So, USDC. So when you come on the platform we automatically we use privy to automatically create you a wallet. So when you go in your personal space you find the wallet keys and you can change it with your own like so let's say you are someone that knows how to use web 3 and know how to use a wallet you can change it and and you know put your own wallet there but by default we just abstract that. So we create your wallet you won't have to worry about that and just the day you want to send this money somewhere else you can. So now it's quite easy because it's USDC only.

So for the moment we just we create a wallet, it's in USDC. It's fine. Now that we integrating and we opening to other protocol, we may integrate a new tokens. So we in the middle of that, you know, trying to make sure that people will be able to also receive um you know um subnet tokens or this kind of things. So we really want to take you know make sure so what's important to keep is the alignment between um the data scientists and the crunchers and the project they are mining on. It's super important because this is what motivates them. I mean all of them have very good job. They're making a lot of money. They don't really need you know money. They do that because of the potential of sites that there is out there.

So if we just like cut it and say take the reward and then sell it and give them USDC, it's kind of not interesting anymore because you don't get a chance to get that upside or maybe they can go and buy it back. But most of them are not, you know, they don't have this frequency that you may have in web 3. So that doesn't work. So we want to make sure that we can really redistribute you know this incentive and keep this alignment between you know I create more value uh for that tokens to become better and you know for my intelligence let's say to be um um um more and more valuable in the future for every unit I provide and so yeah that's a technical challenge that we are in the middle of uh it's good that we don't you know now we are in the warm-up phase with the first subnet that we that we integrated We've seen so we really like you know plugging the last wire making sure that things works and you know properly flow between the different parties and as soon as this is fixed we'll have a clear a clear vision on how the flus are passing and how do we make sure they don't have to fall back into the web three let's say I want to say bad UIX because yeah there is a very bad UI to web 3 unfortunately there is I mean I think I mean just going back I mean I don't want to bag on Bit Tensor specifically because it's all of web 3 but yeah it's very difficult for new people like very smart people even to figure out how to mine and and even people who are familiar with Bit Tensor like like I am like it's still confusing for me right even using a claw right so man even sometime it's difficult to stake on some protocol and you're like that should be the easiest thing in the world and you just want to stake money somewhere and again benur is not that bad actually compared to probably a lot of a lot of the project I've seen along with the the past five years things are getting better but also you have to there is this ent level of complexity that has some virtue because it allows this decentralization right you cannot simplify thing I'm a I love simplification I love making things as simple as possible but there is a limit to that and so when you are with this level of decentralization you there is complexity because there is a balance of powers is because there's all that kind of game theoryic environment that is very complex that needs that has many moving parts.

And and so the the you cannot make things as simple as you would like if you want to keep that level of uh of decentralization whereas on now side we can kind of you know create abstraction abstract a lot of a lot of things that are not yeah I mean I mean I the way I think of what you've done here is you've created like a device driver for very smart humans to plug into Bit Tensor right like there's a lot of stuff that those smart even those smart homes couldn't probably get through quickly or easily or maybe not at all partly some of them just don't like crypto so they just would never you know on their own go and figure out wallets or something right so and I can say like the the complexity is not a problem for bots right now like cloud bots can go and read that and maybe help you sensitize and make things simpler but humans don't have a lot of time like you know our time our you know when especially now like we have one or two days and you're like okay over the weekend I want to launch that you know and you kind of put your goals into it and so you that's short amount of time to read a lot of documentation.

So um yeah we try to make that box as small as possible so human don't have to can just have all the information as soon as possible then leverage eventually agent for the one that wants to in order to deploy a strategy that will be like okay I'm going to use structural break on that type of time series or I'm going to use that type of g or that types of algorithm that I like um I just need to have clo you know u um coded but they need to understand the problem and get to Okay, how I am going to solve it? This is how you extract human alpha. The rest is machine alpha. But the human alpha is okay. Oh, actually that would be cool to solve it that way and just okay, let's ask logo to do it. But these guys have a lot of experience, right? They they've been it's their job to build models. So human device driver to extract human alpha. Love that. Yeah, that's what you've done.

And human is insanely valuable. Like I I don't think I think there's a lot of people living in our space who don't understand how insanely valuable what you have just done is for all of Bitensor. Like it's just like these things could not really connect easily before and now it's just blink plug and play. You've invented Windows like it's like just the impact of that is going to be enormous. And thanks Mark because I've been the way the beat and so community has really like opened the arm and really I was capable of meeting a lot of people and meeting a lot of subnets and subnets coming to me understanding and helping me to understand things better fix stuff. It's been like a unique experience and and you know we've been like we built on actually before USDC we were paying at the very beginning when even our first customer when our first customer was a hedge like our hedge fun so we could not really pay a lot of reward we started with uh with an Ethereum token uh you know and and at that moment you know the gas fee just went to the roof at the same time so we were like well we don't have money to pay rewards but we end up not having money to even pay the transaction fee to send like rewards cards that have no value.

Uh, so we ended up, you know, moving back to, uh, to USDC because by the time we realized this gas fee problem, we could actually pay USDC uh, reward because we had a bit of a bit of runway and a bit of business on the other side. And and yeah, it's uh, I mean it's it's been a and you know, it's a journey, right? like this entrepreneurship is a journey and and solving this problem is is not yet done. We are at the very beginning of an amazing journey which is I think the fundamental question we can we can answer here is what is going to be the relationship between human and uh between human and and these machines and how we going to work with LLMs right in the future and everybody's asking themselves this question now even me myself every day I'm like where where are we going to still be relevant I do believe that in decentralized AI in Crunch and ben so there going to be opportunity to be relevant And it's going to be a lot of money there.

And that if you can be relevant there, then you're going to be an extremely wealthy person and you're going to be able to live in this space where you don't need a job, right? You will be providing intelligence to several opportunities and be free and be you know uh also protected by this very strong decentralization mechanism that will be uh you know kind of an anchor to your financial safety for the rest of your days. You know you just maybe you provide a model one day and you stay it generate revenue for the next 20 years and maybe you give that wallet with that models to your to your children's you know that's going to be yeah you know um I love the fact that you are allowing or will allow in the future um your participants to keep subnet alpha tokens right because it's sort of like getting paid in stock right and if you if you convert that immediately to to USDC then I I miss out on the upside. Like if I was there in the early days of ridges or whatever and it goes and it just balloons up and goes ballistic and I miss out on that, I'm not going to be happy about that, right?

So yeah, unfortunately it's not unfortunately it's not. That would be even even better and even, you know, crazier to be able to do that. But um to reach that kind of scale today would be very very difficult. So it's kind of a proxy of that. It's a far proxy. I don't think I don't think even the value may be correlated you know because um it's different word but if the perceived value of what you do um