Blockchain experts provide contrasts and predictions on consensus protocols

Bitcoin has heralded a new age in consensus protocols which has already spawned alternatives based on proof of stake, reputation and a promise of new variations and hybrids. It's exciting to think this is just the first generation of open blockchains. The future could see a mesh of interoperable chains, block trees, hybrids, and value transfer across the Internet of Things, enhanced by powerful machine learning algorithms overlaying all this automation.

Consensus protocols date back to the late 1980s as people needed to keep distributed systems in synch when replicating some state. These were essentially closed consensus systems, where participants may not necessarily trust one another but know who each other are. Bitcoin was the first that really allowed anybody to join while somehow the system remained protected from the malicious activities of the participants.

The Bitcoin protocol is probabilistic; Nakamoto consensus actually weakened strict consensus just a tiny bit. Bitcoin uses a mathematical trick to ensure that when the system changes state and the blockchain reorganises, the probability of a transaction changing underneath you drops down exponentially as it gets older. This is why people who use Bitcoin wait for some number of confirmations; the chance of something changing is exponentially smaller as time goes on. The computational load needed to reverse this state is the basis of Bitcoin's security.

Security

Core Bitcoin developer Peter Todd said: "In the case of proof of work, then yeah, I know someone burned a ton of energy and I can prove it, or after my transaction has had x number of confirmations, I know exactly how much energy it's going to cost, within some bounds, to go reverse that. We have pretty good ideas of how much energy it takes to do SHA 256 computation, in part because there is a reward to reveal publicly how good you are at it."

Within a closed or trusted environment the threat of nodes within a network being compromised can be dealt with up front using a range of human level techniques and assurances, such as the legal system, fancy technology, auditing, whatever, said Todd. Starting with an open system there is essentially zero cost to the potential attacker to sign a digital signature; if someone gets a copy of keys then to create a new version of history costs nothing.

"In the case of proof of work, then yeah, I know someone burned a ton of energy and I can prove it, or after my transaction has had x number of confirmations, I know exactly how much energy it's going to cost, within some bounds, to go reverse that."

Peter Todd

Proof of stake provides an alternative whereby blocks are not as computationally onerous to mine, but if a validator on the network behaves badly or unpredictably they are penalised. In this case Ethereum's validators would forfeit funds from a security deposit.

Vlad Zamfir of Ethereum explained: "Casper is a security deposit-based protocol. There's incentivisation of all types of behaviour that aren't on the principle blockchain. The way to think about it is that everything will be included in blocks, even if it's orphaned, so that it will influence the incentives. There is a version of Casper that uses blocktrees instead of blockchains, but we are not going to go with that for the first release most likely except for in this non-direct way of just including things in order to have an effect on incentivistion."

Todd said the proof of stake functionality of Ethereum will certainly work but warned this has nothing to do with security. "The question is: does it get to a point where the security gets broken? Realistically if they accept some centralisation, they can probably actually get away with this. In much the same way that, if you look at the domain name system in detail, it's pretty dodgy what it's actually based on. You can have seven guys who did some crazy key signing ceremony; in the context of society that works reasonably well. I think the thing Ethereum should worry about is what happens if governments start threatening it, or what happens if you get developers turning on each other. What happens if you have equally strong incentives to compromise it?"

Reputation

Evolving from the world of closed consensus come systems based on reputation. These are usually structured around groups of quorums of closed participants with some overlay protocol to keep the quorums in synch with each other. The reputation approach, it is argued, can be easily manipulated. Security maximalists say reputation takes us further away from provable security and much more into hand-waving land.

"Even in the Bitcoin world, you see this consolidation of mining power where everybody is essentially trusting this handful of five guys that run all the mining power. But you don't get to choose them so they just choose themselves by just having a lot of mining power."

Jed McCaleb

Jed McCaleb, co-founder and CTO Stellar Development Foundation, said: "I know where they are coming from. Ideally they would like to have this totally trustless system where nobody has to know who anybody else is in the system. But the reality is that money is basically based on faith. You believe that this one institution will accept this credit. I think that it's inevitable that there is some level of trust or reputation that has to happen.

"Even in the Bitcoin world, you see this consolidation of mining power where everybody is essentially trusting this handful of five guys that run all the mining power. But you don't get to choose them so they just choose themselves by just having a lot of mining power. It's really not a better situation than being able to say: Ok, as long as these reputable institutions that don't have any relationship with each other, all think that this is the correct chain, then I can also think that this is the correct chain; these guys have all validated that this is what has happened in the world.

"Then as long as the people you are picking to listen to are distinct enough, like some banks, some non-profits, some universities, some hacker somewhere – these people are not going to collude. As long as you choose your set of people widely enough then I think you are way safer."

Zamfir of Ethereum said there are different types of reputation that you could use. These are either going to be subjective, as in people have their own idea of what everyone's reputations are, or they will be objective and based on numbers.

He said: "If they are subjective then there is no way for the protocol to punish bad behaviour. If they are objective then there is, but the price of the reputation is basically going to have to be measured in time – repeatedly doing good behaviour in order to get that good reputation. So you are going to have basically the long con problem. If it's relatively cheap for someone to build a good reputation then the punishment that protocol can lay on them when they behave badly isn't that high.

"The difficult thing about reputation systems is that it's really super-hard to measure their security. You can't say: what is the price of this reputation unless that reputation is transferable and you can buy it on the market, in which case it's not really reputation any more, it's just a deposit."

Emin Gun Sirer, an associate professor from Cornell University and expert on consensus protocols, agreed that reputation-based systems are always complicated because of the attack where somebody builds up a good reputation and cashes out. He said: "We see it in the real world all the time. I say this having worked on reputation systems for peer-to-peer file sharing. But that doesn't mean they are dead in the water; there is still lots of good uses for them." He pointed out that reputation is deeply embedded in the current financial system; applying for a mortgage or a credit card employs a reputation based system. "So we are going to see the same thing happen with blockchains," he said.

"Machine learning algorithms are very complex and depending on the inputs they have seen they might conclude different things."

Emin Gun Sirer

Machine learning

If large and complex peer networks seem to be on the cusp of exponential growth, so does the world of big data, machine learning and artificial intelligence. As finance becomes more automated thanks to blockchains, smart contracts and the Internet of Things, AI could perform all sorts of tasks. However, at their lowest levels consensus protocols are all about keeping rules and guarantees in place; predictability, simplicity and a deterministic nature are essential to consensus algorithms, noted Gun Sirer.

"Machine learning algorithms are very complex and depending on the inputs they have seen they might conclude different things," he said.

"Where I see them potentially playing an enormous role is when there is a layer on top, where typically there are decisions to be made. For example, if you are a merchant and somebody wants to give you a payment, do you trust this person to get away with the goods with very few numbers of confirmation. Is this a worthy transaction; or maybe somebody wants to buy something from you with escrow or whatever, will that payment be reversed on you.

"This is something that we know credit card vendors actually use. That's essentially a machine learning algorithms making a financial decision to perhaps block a card or block a payment because this is out of the ordinary. That will happen to a much greater extent when we have blockchains because blockchains help us automate everything."

Zamfir said it was possible a validator on the Ethereum network could do their own machine learning to try and predict which block will be valid, but that performing actions that do not help finalise consensus would be penalised. He said there could be scope for smarter algorithms at protocol level provided these operated transparently. "I would say definitely in the strategy space i.e. how people choose to interact with the protocol. I think there will probably end up being machine learning there, unless we can find solutions which say this is the best way to interact with the protocol, which I think will be a little bit hard given the network dynamics.

"The main reason why we don't want to put machine learning – like black box machine learning – inside the protocol is that basically the more free variables there are that affect the behaviour of the protocol, the less we can say about the guarantees the protocol will have."

Vlad Zamfir

"There might also be inference – I guess rather than black box, call it clearer box machine learning – where, for example, the protocol might dynamically adjust certain variables as a function of what's going on in the system, and that is, in the most general sense, machine learning. For example, the difficulty we are targeting with Bitcoin is in a sense due to inference; they are trying to like infer the hash rate in order to change the difficulty.

"The main reason why we don't want to put machine learning – like black box machine learning – inside the protocol is that basically the more free variables there are that affect the behaviour of the protocol, the less we can say about the guarantees the protocol will have."

Hybrid systems

Peter Todd said most of his research work has actually been on something akin to hybrid systems. He said the basic security property of a proof of work blockchain is something everyone can share; like a primitive that all these different applications can use.

"If you go push all the actual logic for how do you interpret that data to the client side, it's completely ok if you know the Bank of England's blockchain thing for moving their pounds around happens to co-exist on the same system as the Central bank of Russia's thing for moving rubles around. They can interpret the rules they care about and ignore the other data.

"Similarly even if you have a chain whose consensus is signed by say a group of three banks or something, they can always take that state and say it's proved through check-pointing in another blockchain. It lets you know if those trusted parties have been compromised. If I expect these three people to sign it and I also expect to see a summary of that state in the Bitcoin blockchain, well if those three people get compromised they are now signing two different ledgers at once. They'll see it on the Bitcoin blockchain because they can't control the Bitcoin blockchain, they can only publish to it. That gets you the best of both worlds."

McCaleb predicted some level of sharding will come into play as it gets harder to reach consensus on all the transactions in very large networks. "You need some way of breaking that up and making it more tractable, where maybe all the transactions in North America are settled by one consensus algorithm and maybe there's like an overarching one that settles between all the different ones or something like that," he said.

"This is the thing you have with databases when they get super-big you basically just have to shard them. You have certain parts of the data handled by certain different databases, and I think there will be a similar thing for consensus. At some point you have to limit the set of computers that are coming to agreement or the set of stuff they are trying agree upon otherwise it just won't scale."

He said innovations will take forms like the Lightning Network where tons of transactions are done at a remove from the consensus mechanism. "I think these public distributed databases will only become ways to settle disputes or settle larger things at the end of the day, not all these tiny little transactions. You don't need to broadcast publicly to the world that you bought this coffee, right."