Problematics

Introduction

DApps, block explorers and regular EOS users need to query the blockchain data on a regular basis to fulfill their information needs. This fact, quite trivial when thought of in the context of other blockchains, is creating a perplexed set of issues in the context of EOS due to certain unique properties differentiating this chain from others.

About 80% of all queries to the EOS blockchain are made towards the most recent block history (around most recent 20% of the blockchain length) while the remaining queries are distributed randomly throughout the rest of the chain history. DApps are the beating heart of the EOS ecosystem and arguably are the purpose of its existence. This is best illustrated by the fact that the top 10 dApps make more than 1.25 million transactions daily. The figure below shows the distribution of transactions among these dApps. To put it in perspective, today 10 largest EOS dApps account for roughly 44% of all the mainnet transactions.

In what follows, we will elaborate on the issues (current and potential) that derive from the aforementioned. For this purpose, we will analyze separately the technological issues that are the root cause of the problem and economic inefficiencies that exacerbate the state of affairs. Let us begin with the latter.

Economic dimension

While the community has mainly focused on the concerns present in the EOS governance layer design, the issues mounting on the infrastructural layer have remained largely overlooked. As we will demonstrate below, today, this layer hides the greatest and most hazardous source of centralization in the EOS ecosystem, while being absolutely vital to the functioning of the system as a whole.

In this part, we will explain how the remarkable transaction throughput efficiency, one of the major strengths differentiating EOS from other third-gen blockchains has also become its major weakness.

The costs

The cost of running full-history node has been estimated by a large EOS BP, EOS42, to run over US$ 30,000 now. These costs are substantial even for EOS42 that receives an average daily reward of 700 EOS. The reality is that running an FHN is a challenging task, both in terms of skills and human hours required, and, especially, business-wise in terms of costs involved. Furthermore, BPs who are arguably best equipped to run such nodes do not actually need to do so to perform their daily routine of producing blocks. And, neither are they rewarded for running such a node. In effect, there are only six entities who support the entire EOS mainnet by running a full-history API node.

Only two utilize the same database technology, while only EOS Sweden is an active block producer.

We cannot stress enough that these BPs take upon their shoulders the entire network traffic today and bear all the costs for doing so, without getting any reward except for good publicity that is yet limited to a very narrow cluster of people who understand the problematics described above.

Before we proceed to examine other dimensions of this adverce state of affairs, let us conclude with defining FHN Running Cost as the total economic cost of all the resources that are used by an entity running and maintaining a FHN in the foreseeable future. From the discussion above, it becomes evident that it is a function of the transactions per second though the network, number of actions per transaction, and action sizes. They grow with the popularity of dApps and the EOS ecosystem as a whole, and importantly, the number of active FHNs in the network.

The benefits

Above we have examined the cost dimension of running the FHN and have illustrated in bold outlines the importance of having a reliable set of nodes connected to the network, at all times providing data feeds to dApps, thereby addressing their existential need to query the chain history. In this part, we will augment the picture by describing the “benefit” dimension of running a FHN or, sadly, as we will demonstrate, the absence thereof.

Indeed, today, there are no quantitatively measurable benefits for running a FHN. Assuming it is a BP who runs the FHN, the only such benefit is the unobserved hypothetical positive relation between maintenance of one such node and the number of additional votes received (perhaps caused by good publicity and increased loyalty of dApps that rely on the FHN).

Now, let us define the FHN Running Utility as the total economic revenue that accrues to an entity running a FHN as a result of its activity. It is clear that there is a multitude of factors the FHN utility is a function of, including but not limited to: positive marketing effects and any fees that this entity charges for its activities. In the context of this new definition, we would like to reiterate that (a) the marketing gains are limited due to the lack of the public attention to the topic of blockchain data economics (probably owing to the technical complexity thereof) and, (b) that as of yet, there are no monetary gains accruing to those running FHNs (apart from the close source dFuse solution offered by EOS Canada).

The perspective

From the discussion above it is clear that, while the benefits of running a FHN are sparse the costs are significant and growing with every next block - i.e. the cost grows with the blockchain size. Furthermore, as a result of intricate hardware scaling economics, the FHN Running Costs will grow faster than in a linear fashion, with the blockchain size, as more queries and larger blockchain will necessitate the development of high throughput and high load services that will result in additional overhead costs, that would grow along with task complexity.

EOS blockchain statistics shows that, overall, the EOS blockchain size grows exponentially with time. This is somewhat odd state of affairs, given that the EOS block size is limited to include a maximum of 1000 transactions. The explanation lies in the fact that a single EOS transaction can include more than one action. As noted, more complex smart contracts create the need for more complex transactions implying more actions and larger average unit transaction size. The Bitcoin blockchain is a bit over 10 years old, while its size is over 15 times smaller than that of EOS, where the first block was verified 292 days ago. Furthermore, compared to the Ethereum blockchain, that of EOS is yet, over 20 times larger, while being over 9 times younger. Assuming the constant growth rate exhibited so far and extrapolating using the Bitcoin blockchain age, one sees that in roughly 10 years the size would be over 63 TB or 260 times larger than Bitcoin’s.

An additional, somewhat unobvious, feature of the FHN Running Cost function, is the hidden jump component that sits with the number of active FHNs argument. Indeed, when one FHN exits the system for political, technological or a business reason (or, more likely, a combination thereof), the load that has rested with this FHN shifts to those that remain. An important observation to factor into our discussion is that ceteris paribus there is a positive relation between the popularity of a given FHN and its likelihood to exit the business: given that there is no monetary reward for running a FHN, the better and more well-know the solution is, the larger is its share of the network traffic and therefore the larger the cost.

On the other hand, the benefits of offering this vital service to the community that remains largely ignorant of this problem are vague. Worsening the matter further is the fact that even the marketing benefits accruing for running a FHN are fuzzy, while they presumably materialize in more votes and therefore larger BP revenue, the price of EOS is not correlated with the blockchain size. On the other hand, as we have shown earlier, the costs definitely are. More importantly, there is no free market based system to figure out the equilibrium value of FHN service provision. It’s somewhat ironic that probably the most free-market-ideologically-charged community has no market for one of the most vital services that upholds its very existence and long-term prosperity.

Technological Dimension

As slightly noted earlier, there is no industry-wide standard for running and maintaining the FHN. Effectively, the majority of BPs that run such nodes are using different software to store blockchain data and, as a result all have heterogeneous APIs.

Naturally, different database systems and APIs create an environment where inability to quickly switch between blockchain data providers is common among dApps: indeed, over 90% of dApps1 are connected to a maximum one of the six active FHN service providers. As a result, there is a lack of any kind of a failover mechanism for the vast majority of dApps in the event of API malfunction or shutdown.

Given absence of any failover on the API access level, adding insult to an injury is the positive feedback loop, that exists between the hypothetical failure of any single FHN and the likelihood of failure of its peers. Indeed, if one of six giants upholding the EOS sky today is defeated, the rest will have to assume additional traffic that would naturally shift to them. In other words, the rest of the full-history API providers would have to bear additional load further increasing their cost of maintenance, magnifying significantly their FHN Running Costs, thereby, increasing the Chain-Wide Failure Intensity.

To wrap up the discussion about the problems that plague the economics of EOS data, we need to take a brief look into the future. Let us assume that EOSIO becomes the true undisputed leader in driving mass-adoption of blockchain technologies. While, obviously, being an extremely good future to live in for everyone heavily invested in the EOS ecosystem, this scenario exacerbates the aforementioned issues by several orders of magnitude bringing it to a different level of complexity to resolve. Assuming truly mass adoption, we are talking about hundreds of terabytes of actions to be held accessible to millions of geographically dispersed queries a second. We are not talking about thousands of dollars worth of monthly running costs, we are talking about millions. And, certainly, when ultra-low latencies for mission-critical applications are concerned, we are not talking about simple hardware scaling roadmaps, we are talking about the existential call for innovation. These issues while being much less known to a wider community are in fact no less vital to the prosperity of EOSIO - e.g. the notorious RAM cost and horizontal scaling concerns.

Summary

  1. The important dimension of blockchain data economics is currently being largely overlooked by the wider EOS community. This state of affairs is somewhat due to the unique features of EOSIO that differentiate it significantly from other distributed systems software.
  2. The speed of transactions and large throughput that set EOSIO aside from other blockchains are the major pillars upholding the claim that EOS is the system that is best fit for building dApps ready for mass adoption.
  3. Maintaining such speed and throughput capacity as we move forward sets different demands for hardware and software of those supplying the network with computational resources.
  4. While issues with economics of, for example RAM, NET and CPU, are widely covered by EOS focused media, the emerging problem of servicing the network traffic querying the blockchain data is not widely known to the community.
  5. So-called full-history API nodes are getting exponentially more expensive to run and maintain, while there is no free-market based mechanism for determining the equilibrium reward to those providing this vital service. As a result, today there are only 6 entities providing this service and upholding the chain-wide query traffic of the entire EOS mainnet.
  6. DApps and other services running on the EOS mainnet need to query the historical data contained therein. The vast majority of queries are addressed to the recent block history. Despite a relatively small number of queries that address the entire chain history, they are as important for orderly dApp (and other services) performance.
  7. The infrastructure and costs involved in maintaining a full-history API node that allows for such queries make it prohibitively expensive for regular users to run. Simultaneously, the block producers that are arguably best equipped to maintain such nodes, do not need to do so in order to fulfill their BP duties. As a result there are only six entities that provide this vital service.
  8. The EOS mainnet token price that acts as the only relatively direct reward for running a full-history API node is not appreciating in line with blockchain size growth.
  9. On the contrary, the cost of running such node grows exponentially with time and linearly with the blockchain size. Furthermore, if any of the existing service providers decide to seize this line of operations for technological, business or other reasons, the weight of the network load resting on the shoulders of remaining solutions will abruptly increase.
  10. The absence of chainwide standard prohibits the orderly development of a unified API solution and therefore complicates the creation of a formidable failover infrastructure by those using such solutions.
  11. The future holds even more challenges as exponential growth of blockchain size that is inexorable, assuming mass adoption, will require the development of much more expensive and fault-tolerant infrastructure.

Absence of market for these resources impedes the efficient price formation and distorts incentives. A classic free-rider problem in relation to a public good arises. If we are to start building a resilient ecosystem incentivising resource provision and fair reward for the resources provided, we need to create a market for blockchain data provision. Tokenomics is the tool of the 21st century most fit for such ambitious undertaking.