Support setting of schema-creation timestamp [WIP] #5260
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
johnkerl
force-pushed
the
kerl/schema-stamp
branch
3 times, most recently
from
b2db23f to
2278cf2
Compare
johnkerl
force-pushed
the
kerl/schema-stamp
branch
from
937c47e to
3fd2a2b
Compare
ihnorton
requested review from
teo-tsirpanis
and removed request for
teo-tsirpanis
teo-tsirpanis
added a commit
that referenced
this pull request
Apr 4, 2025
This work builds on the work originally done by @johnkerl in #5260 which introduced the `tiledb_array_schema_alloc_at_timestamp` to the C API. As iterated from the previous PR, this removes the CPP API addition and adds unit testing for creating a schema at specific timestamps and further schema evolution at specific timestamps for adding and dropping attributes, extending enumerations, and resizing the current domain. The motivation for the PR is outlined in more detail [here](https://app.shortcut.com/tiledb-inc/story/47660/core-c-c-api-support-to-set-initial-array-schema-timestamp#activity-53864). TileDB-SOMA has had a longstanding [support](single-cell-data/TileDB-SOMA#1001) to create `SOMAArray`s at a given `tiledb_timestamp`. However, there is no way to set the initial `ArraySchema` file timestamp, and it is always written to the current timestamp. This limitation causes issues for performing schema evolutions at a timestamp, specifically with [extending enumerations](single-cell-data/TileDB-SOMA#2879) and [resizing arrays](https://app.shortcut.com/tiledb-inc/story/61254/python-array-resize-and-time-travel-not-working-correctly). To discourage misuse of this feature, we have limited its availability to the C API and added a warning to emphasize that is to be used with caution. --- TYPE: FEATURE | C_API DESC: Support setting of schema-creation timestamp --------- Co-authored-by: John Kerl <kerl.john.r@gmail.com> Co-authored-by: Theodore Tsirpanis <teo@tsirpanis.gr> Co-authored-by: Beka Davis <31743465+bekadavis9@users.noreply.github.com> Co-authored-by: Ypatia Tsavliri <ypatia@tiledb.com> Co-authored-by: Isaiah Norton <isaiah@tiledb.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Long description to be typed up.
See also:
cellxgene_census_builder//test_builder.pyfailing single-cell-data/TileDB-SOMA#2920ArraySchemaEvolutionto specified timestamp single-cell-data/TileDB-SOMA#2895Clean and tested against tiledbsoma acceptance cases as a patch on 2.25.0; iterating on clean patch against tip of
dev.Unit-test cases upcoming.
TYPE: FEATURE
DESC: to be typed up