Data Freshness
Understand the frequency of updates for various data types on Dune.
At Dune, data is continually refreshed through a multi-stage process to ensure accuracy and reliability. The three main stages of data updates are: Raw Data, Decoded Data, and Curated Data. The update frequency for each type of data varies based on several factors.
Raw Data
Raw data consists of tables such as ethereum.blocks or ethereum.transactions that are directly ingested from the blockchain. The frequency of raw data updates is primarily influenced by the blockchain’s frequency of producing blocks and its likelihood of a reorg.
Blockchain Dependence
- Chains like Optimism, Arbitrum, and Avalanche: These blockchains exhibit lower chances of undergoing reorgs. As a result, data ingestion occurs more swiftly since the stability of these chains allows for quicker validation of block data.
- Chains like Polygon and Ethereum: Given their higher propensity for reorgs, there is a deliberate delay in data ingestion. This delay ensures that only stable and reorg-resistant blocks are considered, preventing the ingestion of potentially unstable or reversible transactions.
We do our best to ensure that the data is as fresh as possible while maintaining the integrity of the data. You can find the data freshness of the raw data in the table below.
Decoded Data
Decoded data refers to tables such as uniswap_v3_ethereum.Factory_evt_PoolCreated or tokens_ethereum.erc20 that are derived from the raw data. This stage of data processing involves transforming raw blockchain data into more accessible and interpretable formats.
The decoding of data is performed in real-time and is contingent on the volume of data being processed. The typical timeline for decoding is as follows:
- Time Frame: Decoding generally occurs within 15 to 60 seconds following the ingestion of raw data.
- Dependency on Data Volume: The specific duration within this range depends on the current data throughput and the complexity of the decoding operations required.
You can find the data freshness of the decoded data in the table below.
Curated Data
Curated data tables on Dune are tables like dex.trades or nft.trades that are derived from raw and decoded data. They take away the complexity of working with raw or even decoded data and are ready to be used for analysis.
The data pipelines are defined and documented in this GitHub repo.
Curated tables are usually scheduled to run once every hour. The runtime of the data models can vary depending on the volume of transactions on any given day, therefore curated datasets are sometimes slower than the usual 1-hour target to update.
Community Data
Several notable 3rd party data providers are uploading their data into Dune. These data providers have their own update schedules and are not controlled by Dune. The data freshness of these providers can be found in the table below.
Data Freshness
Understand the frequency of updates for various data types on Dune.
At Dune, data is continually refreshed through a multi-stage process to ensure accuracy and reliability. The three main stages of data updates are: Raw Data, Decoded Data, and Curated Data. The update frequency for each type of data varies based on several factors.
Raw Data
Raw data consists of tables such as ethereum.blocks or ethereum.transactions that are directly ingested from the blockchain. The frequency of raw data updates is primarily influenced by the blockchain’s frequency of producing blocks and its likelihood of a reorg.
Blockchain Dependence
- Chains like Optimism, Arbitrum, and Avalanche: These blockchains exhibit lower chances of undergoing reorgs. As a result, data ingestion occurs more swiftly since the stability of these chains allows for quicker validation of block data.
- Chains like Polygon and Ethereum: Given their higher propensity for reorgs, there is a deliberate delay in data ingestion. This delay ensures that only stable and reorg-resistant blocks are considered, preventing the ingestion of potentially unstable or reversible transactions.
We do our best to ensure that the data is as fresh as possible while maintaining the integrity of the data. You can find the data freshness of the raw data in the table below.
Decoded Data
Decoded data refers to tables such as uniswap_v3_ethereum.Factory_evt_PoolCreated or tokens_ethereum.erc20 that are derived from the raw data. This stage of data processing involves transforming raw blockchain data into more accessible and interpretable formats.
The decoding of data is performed in real-time and is contingent on the volume of data being processed. The typical timeline for decoding is as follows:
- Time Frame: Decoding generally occurs within 15 to 60 seconds following the ingestion of raw data.
- Dependency on Data Volume: The specific duration within this range depends on the current data throughput and the complexity of the decoding operations required.
You can find the data freshness of the decoded data in the table below.
Curated Data
Curated data tables on Dune are tables like dex.trades or nft.trades that are derived from raw and decoded data. They take away the complexity of working with raw or even decoded data and are ready to be used for analysis.
The data pipelines are defined and documented in this GitHub repo.
Curated tables are usually scheduled to run once every hour. The runtime of the data models can vary depending on the volume of transactions on any given day, therefore curated datasets are sometimes slower than the usual 1-hour target to update.
Community Data
Several notable 3rd party data providers are uploading their data into Dune. These data providers have their own update schedules and are not controlled by Dune. The data freshness of these providers can be found in the table below.