![]() ![]() Some operations support retry by re-applying metadata changes and committing, under well-defined conditions. If the snapshot on which an update is based is no longer current, the writer must retry the update based on the new current version. Once a writer has created an update, it commits by swapping the table’s metadata file pointer from the base version to the new version. Writers create table metadata files optimistically, assuming that the current version will not be changed before the writer’s commit. Readers use the snapshot that was current when they load the table metadata and are not affected by changes until they refresh and pick up a new metadata location. Optimistic ConcurrencyĪn atomic swap of one table metadata file for another provides the basis for serializable isolation. These stats are used to avoid reading manifests that are not required for an operation. Each manifest list stores metadata about manifests, including partition stats and data file counts. The manifests that make up a snapshot are stored in a manifest list file. Manifests can track data files with any subset of a table and are not associated with partitions. Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. The data in a snapshot is the union of all files in its manifests. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table.ĭata files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file’s partition data, and its metrics. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. Table state is maintained in metadata files. This allows writers to create data files in-place and only adds files to the table in an explicit commit. This table format tracks individual data files in a table instead of directories. Both read-optimized and write-optimized formats will be available.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |