Bittensor Cofounder Explains What Makes a Great Subnet

Basically all the subnet is at the kernel. Throw everything out. It's a benchmark, which is a way of evaluating submissions.

You just need to figure out how to produce a very, very, very precise and aligned benchmark. You don't need a website. You don't need a product. You don't need revenue.

The first thing you start on is, okay, how good is this benchmark? Can people overfit on it? Is it aligned with the problem that I'm solving? All else is irrelevant until that has been solved.

The commodity that's going to be produced will be obvious implicitly from whatever that benchmark and how that benchmark is defined.

The first thing is to define the benchmark, like how do you measure what the miners are doing? Then define what is the format of the submission that the miners produce.

That could be they submit to a Hugging Face model, or they're an endpoint that receives queries and is an RPC endpoint, or it's a model running on Shoots, or it's an Agent that they write into a GitHub in a Python file that I run inside of a Docker file, or it's a Docker file itself.

It could also be trades submitted on a centralized exchange, or data collected from the web and put onto an S3 bucket. Those are all reasonable formats for the submission type.

So once you know what the submission type is, you need to figure out how you're going to evaluate it. Actually, both of those have to happen at the same time.

Once you have those two things, you just go write up the two small Python files that you need to do those two things. One is the validator and one is the miner.

One is, okay, we'll write your code and then run this Python code to submit it to GitHub. And then the validator is, okay, pull all of the commits from the chain and run this evaluation and then use that as a weight.

That's all a potential subnet is. And then you can get much more fancy about the crytosis of the incentive curve and winner take all versus winner take all and OSS versus OSS and the complications of that, but that's the gist, that's the kernel, and everything else is irrelevant.

So what makes a good subnet is if the incentive is perfectly aligned with improving against the thing that they want to improve on the subnet. You know, reducing cost on storage performance of the Agent.

There's what you want, a good Agent that for coding, and then there's how you define what you want, and those might not be the same thing. So you need to focus on that alignment first.

So how do we actually measure a good coding Agent? That might be a set of benchmarks. That's the first thing that makes a good subnet.

But that's not the only thing that will drive alignment of the incentives because there's also, for instance, how much variance there is in the estimation of an Agent against that benchmark.

So if there's a lot of variance, if there's a lot of randomness involved in the evaluation, so we have a benchmark, it's aligned with our goal, but the Agent, when you run the Agent evaluation, the variance is very high, that reduces or it makes less sharp the incentive landscape, right? Makes the incentive curve less steep around the point of maximum improvement.

Going back one step, it's actually very difficult to make the benchmark in alignment with the end goal because of something called overfitting.

So a subnet needs to focus on the alignment with the goal but also make it so the miners are not overfitting, which means that they're not just getting better at the benchmark but not at the end goal, which was a problem that for instance Ridges had for a while because they're using Swebench Pro, which is a data set of maybe a benchmark of a couple hundred repos, maybe it's a couple thousand repos.

But the Agents, the minor submissions, were robust enough, large enough that it was possible to inject the specific solutions to those thousand or so problems without actually getting better at software engineering.

So they needed to come up with what was called a synthetic benchmark, a way of taking not taking those thousand repos and turning them into 10,000 or taking all repos on GitHub and turning them into 100 trillion different problems that could be solved that were all in alignment with the original goal. So that's an ongoing process for them.

There's also things like copy resistance, right? So if it's possible to just copy the previous, the current top score and get the same score as them, then that erodes incentive because you're no longer putting 100% towards the top guy.

There's going to be erod it's going to erode some maybe randomly towards the guy that copies them. So you want to reduce the possibility that just a copy solution will outperform the top solution on the network, which comes down to measuring the variance of the evaluation over the benchmark and making sure that to become the next top miner you have to beat them by that much, which is the variance amount.

Alternatively, we talk about UID pressure, which is when the subnet is not a winner take all subnet, but it's something where you can contribute X amount and you get paid for X amount.

As an example would be like a scraping subnet. If you can, you know, your minor scrapes 10,000, I scrape 100,000. And so, you know, I'm going to make 10 times as much incentive as you.

Maybe after normalization, something like this will have UID pressure. If you cap the total number of submissions that a single UID can have, UID pressure means that let's say that you can a single minor can only submit a thousand or 10,000 things scraped from the web.

Then miners will be incentivized to take up all of the 256 slots on the subnet. If you don't have a cap, then they don't need to have multiple UIDs. And so they'll just push as many scrapes per UID.

So that they actually in practice there'll only be like maybe five, 5, 10, 15, 20 miners that need slots on the subnet and the rest will just the registration fee will go to zero. That's a good sign.

It's a good practice to start with open-source competitions where the miners actually contribute. They submit the full coded solution to their problems for infrastructure for scraping for training etc etc so that you can see explicitly where you are pushing your miners to optimize.

Blackbox would be alternative where you just query an endpoint and then the endpoint returns a solution. That's good for subnets that might want to protect their minor IP.

But I do think that subnets should go for OSS first because it actually reduces the learning curve on a lot of subets that were blackbox miners would come to the network and just have no idea how to improve because they can't see what's going on.

Not only that, but the subnet teams can't see what's going on. They don't know what the miners are doing. So they don't even know if they're being manipulated or overfit or if they're converging the miners towards true intelligence in the direction that they want.

So OSS is a great place to start. Winner take all is a great place to start.

A good subnet also integrates with other subnets on Bittensor. I think that's something I've been pushing for a lot. If you need to host models, host them on Shoots.

If you need compute, use Basilica, Kadargon, or if you need data, use Dataverse. If you need your Agents to be able to access the internet, let them use 22. If they need to store files, give them Hibious buckets.

There's a major benefit of this is just that those teams will help you because you're using the technology you're possibly paying for it. And more than that, the cost is probably lower because Bittensor commodities tend to be cheaper.

So that means that there's less cell pressure on your subnet because the miners have to pay less or the validators have to pay less to run the infrastructure. The less cell pressure in your subnet means you can have more flow which means you have more emission which is all in the interest of doing well in Bittensor.

There are levels to what is important in a subnet. The first one is transparency. It's important, extremely important that everybody can come in and see that you cannot have cheated.

Your subnet, if it's possible to cheat, people will accuse you of cheating and no one will know the difference and that will cause that will take up your time whether or not you cheated or not. So you need to figure out a way of making it auditable.

Decentralization is a second level of auditability. Figure out auditability first and then figure out how to do the auditability in a decentralized way.

You can have a centralized subnet on Bittensor. There's a number of subnets that are pretty centralized. Shoots as an example. Aphine is actually an example. Swarm is an example.

There's one validator that's doing the evaluations, but they're posting the results to a bucket that can be run that other people can validate those that those results are actually produced by the validator that you're running.

If I stop writing the validator code that I said I'm running, people will see that those results don't line up with what I'm posting on the chain and the other validators can come into play and move away from your stake. Also, the miners can choose to leave. They can be like I don't want to work in this network and investors can choose to choose to sell because they actually there's actually consensus about the fact that this is not lining up.

So transparency is key and that means that the function that you go from evaluation of the minor to the weights, it needs to be able to be run by any individual in the network. The more transparent the better and that's just about good enough.

Decentralization can be costly and not always needed. Primarily people use the decentralization so they can have more vantage points, they can collect more data across more of the validators.

I think transparency is key for just in terms of how people should approach building a subnet in individualistically. The miners are always going to try to screw with you and your setup is always going to be being exploited, at least at first.

There's no reason to hide from that because it happens to everyone and it will help you develop solutions because if you're transparent about the types of exploits that are happening, you will be able to elicit help from your community because a lot of time the miners that are not exploiting in that particular way will help you fix the exploit because it's totally in their incentive and they'll feel like you're being honest with the community and that's always it's good to build that goodwill rather than hiding the fact that something bad has happened.

The funny mistakes that people make is that they go miners wouldn't do that. And I think that's quite funny after five years. Miners will definitely do that.

What do you mean that they'll try to overfit on the benchmark? Oh god, no. Yeah. No, of course they'll do that. That's the first mistake.

The second mistake is that they think they need to have the thing written and perfect right away. It doesn't. I'm a big fan of pushing very quickly to main and testing, but using Docker Watchtower and Git hooks to make it so that the validators update just on the fly as you push new changes to the incentive mechanism on main.

Don't dial it on test net for months. Just go straight to main, get your subnet going, start launching, testing, and evaluating and trying to understand exactly what's going on.

A lot of people shy away from actually looking at what's going on with the network. Like go and actually dog food what the miners are doing. Go look at the Agent, run the Agent, go query the models yourself and get these outputs because this is how you know you don't hide from the reality and fail fast is really important in Bittensor.