Skip to main content

Measuring Impact

Impact Metrics​

Quantifying the impact Dev Tools Guild members have on the Ethereum development and wider ecosystem is crucial to establishing trust in the work of the Guild and ensuring the sum of its parts is greater than any individual project's impact.

Unlike core protocol development which ultimately results in updates to the Ethereum clients running the Ethereum mainnet, there is no co-ordinated onchain metrics that can be used to cohesively measure onchain impact of most developer tooling (Solidity and Vyper contract deployments being the exception that comes to mind).

Historically, more traditional quantifiable metrics have been used to demonstrate the impact of developer tooling projects in Ethereum. However, these don’t necessarily provide a real understanding of the impact these projects have had. As per Goodhart’s Law ("When a measure becomes a target, it ceases to be a good measure”), metrics can be gamed, and especially when one sees the size of communities on X, Discord or Telegram for projects with planned token launches, it’s clear how much incentives can be misaligned in web3 to favour short-term gains over providing long-term positive impact by projects.

It’s vital that the Dev Tools Guild does not fall into this trap and outlining ways in which impact can be measured from an impact-driven perspective is key.

Classical metrics do have their place still, but these should be considered an early filter for projects, these include metrics such as:

  • Project age
  • GitHub stars
  • Contributors
  • Closed PRs
  • Closed issues
  • Releases
  • Discord/Telegram/Gitter/Matrix group members
  • Lifetime downloads
  • Monthly downloads
  • Year-on-year change in downloads
  • Team size

Impact metrics are less tangible and could include metrics such as:

  • Projects or companies using tool
  • Annual developer survey - asking developers about their tooling stack, languages, etc.
  • Interview (or at least review) top 100 project teams for their dev tools stack (also a great opportunity to encourage their sponsorship)
  • Minimal published audits to demonstrate funding is going towards FOSS tool(s)development and maintenance, and not other commercial aspects of the receiving organization
  • Projects rewarded by airdrops may be a helpful metric for determining the impact of projects. For instance, the following airdrops have provided some useful information on OSS they have benefited from:
  • Industry reports may be helpful such as Electric Capital Developer Report
  • Ethereum Stack Overflow activity — example
  • Partnership with projects such as https://openq.dev/ that provide deep analysis on open-source projects. 
  • In library instrumentation such as using the RPC User-Agent header where RPC providers aggregate metrics on client interactions

Reporting​

Establishing accountability mechanisms will be crucial alongside impact metrics to demonstrate the value of the Guild’s activities.

Whilst impact metrics will be used to demonstrate the reach of member projects and how they benefit the Ethereum community, reporting will be important to demonstrate how funds being donated to the Guild are being utilised by member projects.

Unlike Ethereum clients, dev tools do not evolve in lockstep with the Ethereum mainnet (although some upgrades do require updates to tooling). Hence, for the purposes of transparency by members and for members and funders, it’s important that there is visibility into the activity of members.

Whilst FOSS is developed in the open, heading to project GitHub repos, social accounts and blog posts to see updates from members will be time-consuming without curation. Hence with regular reporting, there is a straightforward mechanism to keep up with multiple teams in one place.

The reporting template is not intended to be onerous for teams to fill out but should capture enough information to be useful.

A bi-annual cadence will be used for reporting by members, using the reporting template. Inspiration for this are outlined below.

These reports will be supplemented by project update sessions between members which will cover complimentary or additional material to demonstrate progress by projects.