Is anyone using graph databases or sparql for these sorts of use cases? sparql is so powerful but not sure if there is enough metadata with ethereum for it to be useful.
We see them for smaller transaction extracts, but most vendors here are too expensive for most teams at the scale you are likely thinking. Instead, we see log, kv, dwh, etc systems -- think elastic, Clickhouse, databricks, and the like. Some teams will stage daily batches into a graph DB, but having an expensive second system of records masquerading as a compute engine gets frustrating.
Separating scalable transaction storage from graph compute opens a lot. The new Google Spanner Graph launch is interesting here, and for OSS, we have been working on GFQL so you can bounce between vectorized Python dataframe mode on small graphs to 1B row GPU batches for bigger graphs.
I initially tried using Memgraph but faced stability issues and a lack of flow control tools during traversal, making it impossible to load one month’s worth of Ethereum blockchain data. The current solution, however, handles 10 times more data hosted on the same machine.
ClickHouse is an excellent database, provided you don't need to traverse graphs. Graph traversal requires many queries, and frequent disk interactions can significantly degrade performance.
BTC is the next obvious step.
The most challenging thing is to find relative off-chain data. It's still in progress.
The feature I'm working on is finding paths between 2 wallets. However, I was forced to remove it because it's not stable enough and consumes too many resources.
Is anyone using graph databases or sparql for these sorts of use cases? sparql is so powerful but not sure if there is enough metadata with ethereum for it to be useful.
We see them for smaller transaction extracts, but most vendors here are too expensive for most teams at the scale you are likely thinking. Instead, we see log, kv, dwh, etc systems -- think elastic, Clickhouse, databricks, and the like. Some teams will stage daily batches into a graph DB, but having an expensive second system of records masquerading as a compute engine gets frustrating.
Separating scalable transaction storage from graph compute opens a lot. The new Google Spanner Graph launch is interesting here, and for OSS, we have been working on GFQL so you can bounce between vectorized Python dataframe mode on small graphs to 1B row GPU batches for bigger graphs.
I initially tried using Memgraph but faced stability issues and a lack of flow control tools during traversal, making it impossible to load one month’s worth of Ethereum blockchain data. The current solution, however, handles 10 times more data hosted on the same machine.
ClickHouse is an excellent database, provided you don't need to traverse graphs. Graph traversal requires many queries, and frequent disk interactions can significantly degrade performance.
I've seen in-house deployments.
Tangentially related but https://trueblocks.io/ is a great project if you have personal use-cases.
Is there a main usage goal in mind?
Some thoughts are:
1. when i see other addresses i want to explore those as well, perhaps see how it relates.
2. add time ordered replay
Are there plans to integrate with other blockchain platforms or data sources, or to add new features and capabilities to the tool?
BTC is the next obvious step. The most challenging thing is to find relative off-chain data. It's still in progress.
The feature I'm working on is finding paths between 2 wallets. However, I was forced to remove it because it's not stable enough and consumes too many resources.
For an early prototype it's very cool and fast AF - good job!