Repository navigation
feat(standard-tests): check the cursor state of declarative sources in test_basic_read - #1270
Conversation
…n test_basic_read test_basic_read now checks the cursor state the read already emits for a ConcurrentDeclarativeSource: checkpoints that a resume would skip records from, a final state behind the newest record, per-partition state shape, parent state, partition coverage and a round trip through the cursor. Cursor values and check errors are reported as CursorStateWarning. `validate_state_messages: false` turns the checks off.
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/per-partition-state-checks#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/per-partition-state-checksPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
…ion-state-checks # Conflicts: # airbyte_cdk/test/standard_tests/source_base.py
…fail in a future release
…or state checks - A state past the cursor's end (a record dated in the future) no longer turns the checkpoint check off: a cursor with no slice resumes from the end of its last state slice, less the lookback window. - A record below the start of the slice that returned it counts from that start, since a resumed sync that requests the slice again reads it again. Streams with `is_client_side_incremental` still compare the record's own value. - `test_basic_read` installs the hooks only on a read whose cursor state it checks. - A crash while walking a stream's records or states stops only that stream's checkpoint check; the final-state, shape, parent, coverage and round-trip checks still run. - Tests pin the suite wiring (read duration, reference source, hook removal after a failed read) and run the checks without network access.
Daryna Ishchenko (darynaishchenko)
left a comment
There was a problem hiding this comment.
Requesting changes for one soundness issue in the checkpoint model (inline, 🔴), plus three medium notes (🟠).
What's done well: no second read, resume starts computed with the CDK's own cursor code (gaps, lookback, clamping all come out right), failure messages carry no values, and the tests are real end-to-end reads. I collected and ran the 71 tests, re-applied mutations #19, #72 and #59 (all killed), and the survivors #14, #18 and #88 are equivalent. Reverting the throttle to the literal 600 fails two tests, so the production refactor is covered and behaviour-identical. No runtime, Builder, SDM or manifest-server impact; feat: matches the 7.36/7.40/7.42 precedent.
Rollout: Check: destination-motherduck is red on #1266–#1273 as well, so it's infra. Merge order with airbytehq/airbyte#91061 is as described.
…h the stream its state selects - Build a `StateDelegatingStream` from each state message to get that checkpoint's resume start, instead of from the final state. A state past `api_retention_period` reruns `full_refresh_stream` in full, so it flags nothing. Builds are capped at 200 per stream and no longer put their cleared state on the read's queue. - Skip the round-trip partition count when the build from the final state cleared it. - Compare partition keys per parent `partition_field`, so a router with several parents that use different fields passes. - Skip the checkpoint check, with an advisory `CHECK_ERRORS` line, when a stream's records cannot be matched to the read's slices or the read ran on another source instance, instead of checking each record from its own value. - Drop the final-state cursor template, which only `StateDelegatingStream` needed. - Tests: monkeypatch the checks module by its dotted path instead of importing it twice.
|
/prerelease
|
Baz (bazarnov)
left a comment
There was a problem hiding this comment.
Thanks for this, Anatolii. It is careful work, and the two dry runs make the rollout easy to trust.
What I checked at de7d1c5d, merged locally with current main (clean merge):
- The 83 new tests pass locally (3 runs in a row). The
concurrent_partition_cursor.pychange keeps the 600 s default, and reverting it to the literal fails 2 tests. - The fixes from the previous review hold. The retention case (
late_days=5,step: P1D,api_retention_period: P30D) reports nothing, a two-parent router with differentpartition_fields passes, and unmatched slices only add an advisory note. Mutations 89, 90, 93, 95 and 98 are killed. - The throttle lift writes the cursor instance, not the class. Its counters and the interval write are under one lock on the worker threads, and
uninstallruns infinally. - Problem texts carry only stream, field and key names, formats, counts, the read duration and exception type names.
- On
airbytemaster, 211 manifest-only connectors with abasic_readentry get these checks on the next release: 214, minus chargebee, monday and stripe, which opt out in every entry. The stripe opt-out from airbytehq/airbyte#91061 is in its onlybasic_readentry.
Inline, by priority:
with_final_statecan edit an earlier, equal state message. That made the first run of the requiredPytest (Fast)check fail on this head (a re-run passed). A one-line fix and a test are inline; I'd land this before merge.- The per-partition
StateDelegatingStreambound has no test (the row 101 claim does not hold). - Two more mutations on enforced checks survive.
- Two wording fixes in
cdk-migrations.md.
None of these changes what a connector sees, so I'm approving. Check: destination-motherduck fails here at the secret fetch (its secret has no enabled version), as it does on other open PRs such as #1267. Before merge: darynaishchenko's change request still needs her re-review, and main requires every review thread to be resolved. The rotating-token connectors that were never run (greenhouse, linear, quickbooks, salesloft, strava, trustpilot, typeform) get their first real check after release; greenhouse's job_posts hits the enforced ListPartitionRouter coverage check.
…ee unpinned cursor-state guards `with_final_state` edited the first state message equal to the final one, so a concurrent read that repeats the final state left it unedited and the test flaked. It now edits the last equal message. New tests pin three guards that no test failed without: a per-partition StateDelegatingStream takes its resume bound from `incremental_stream`, a `states` entry whose cursor lacks the cursor field fails the shape check, and a `parent_state` missing an incremental parent fails the parent-state check. The migration note now lists all three final-state failures and says that a suite whose read runs on another source instance skips the checkpoint check.
|
I couldn't reproduce the greenhouse |
Daryna is currently OOO and won't be able to review the PR again.
Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/18410
What
test_basic_readnow checks the cursor state that its read already emits, for aConcurrentDeclarativeSource. It adds no extra read. The declarative factory builds the same cursor for a full refresh read as for an incremental sync, so this state is the one an incremental sync would emit.Enforced checks (the test fails):
is_client_side_incremental, which drops records by value). A state past the cursor's end (a record dated in the future) bounds the resume at the end of its last slice, less the lookback window. AStateDelegatingStreamis built from each state message to find the stream a resume would use; a state pastapi_retention_periodrerunsfull_refresh_streamin full and flags nothing. A stream whose records cannot be matched to the read's slices is not checked (advisory note instead)ConcurrentCursorstream has no cursor value, one the cursor cannot parse, or one behind the newest record at the precision of the cursor'sdatetime_formatuse_global_cursornot true although the manifest setsglobal_substream_cursor, astatesentry without a partition or cursor value, entries of the same parent stream with different partition keys (each parent of a multi-parent router may use its ownpartition_field), a globalstatewithout a cursor value, alookback_windowthat is not a whole number of seconds between 0 and the read's durationincremental_dependencyhas noparent_statefor its parentListPartitionRouterhas nostatesentryAdvisory checks (
CursorStateWarning, the test passes): records without a cursor value or with one the cursor cannot parse, and checks that could not run. Both found nothing in the fleet run. They stay advisory because the first depends on sandbox data (updated_at: nullis normal) and the second points to a gap in the checker, not in the connector.validate_state_messages: falsein abasic_readentry turns these checks off, along with the state message check from 7.42.0. Such a config, and a config not listed underbasic_read, reads without the hooks and with the production state throttle.ConcurrentPerPartitionCursorgets aSTATE_EMIT_INTERVAL_SECONDSclass attribute, so the suite can lift its 600 s state throttle for the first per-partition closes and check checkpoints there too. Failure messages give counts and field names, never record or state values, because CI logs are public.Fleet dry run
184 connectors (177 manifest-only, 7 Python), against
airbytehq/airbytemasterab1b47bce77, with this branch (d2a489d9) installed. 103 scenarios reached the checks; the rest already fail or skiptest_basic_readon master (mostly expired credentials). No check crashed, and no test passed without the hooks but failed with them. 79 per-partition streams in 11 connectors ran with the lifted throttle.StateDelegatingStreamto/events, which returns only the last 30 daysCompared with the first dry run (CDK at the first head of this PR, master
fa95775cb59):updatedAtis beforeupdated_at_start, and the checker now counts a record from the start of its own sliceyearWeekvalues are coarser than the read windows; same reasonplaylist_*_a2streams return no records, and the same without the hooks; the cursor checks found nothingThe GA4 run in the first dry run also exposed a real bug in that connector:
yearWeek202605was parsed as 2026-05-01, which moves the state months ahead of the data in the first weeks of every year.Not verified locally; these connectors run the checks in CI on release:
step, so the lifted checkpoint check applies: gitlab, pinterest, zendesk-support, zoom. Two useincremental_dependency: gong, pipedrive. Two use aListPartitionRouter: greenhouse, trustpilot.fuel_entriesstream alone read more than 55,000 records in 9 minutes without the hooks. mntn and rocket-chat hang on retries (HTTP 500 and dropped connections; an SSL error on the sandbox host). posthog hangs on dropped connections inpersons.Slowest check: 5.2 s on a 277 s read (source-hubspot).
The review fixes in
de7d1c5donly remove flags (retention fallbacks, multi-parent keys, unmatched slices), so this run still holds. Stripe's default config sets noapi_retention_period, so it stays flagged.Merge order
CI installs the CDK unpinned, and a CDK release ships every PR merged to
main, so this PR stays in draft until this connector PR merges:Before this PR is marked ready, a pre-release of this head runs connector CI on the 34 connectors the dry run did not cover (above), starting with gitlab, pinterest, zendesk-support and zoom, where the lifted checkpoint check applies.
lever-hiring and GA4 now pass without an opt-out, so airbytehq/airbyte#91056 (lever-hiring opt-out) and the opt-out in airbytehq/airbyte#91057 no longer gate this PR. The GA4
yearWeek/yearMonthparsing fix in #91057 stands on its own.Conflicts with open CDK PRs
git merge-treeof this head against each PR's head, leaving out conflicts they already have withmain:concurrent_partition_cursor.py: both rewrite the throttle line of_throttle_state_message. Whichever lands second must passself.STATE_EMIT_INTERVAL_SECONDSintostate_emission_is_due, or the suite can no longer lift the throttle (test_throttle_lift_emits_more_per_partition_states_within_budgetcatches it).source_base.py(thetest_basic_readdocstring and read call) andcdk-migrations.md.cdk-migrations.md.No order is required; the conflicts are textual and the second PR to merge resolves them.
Testing
unit_tests/test/test_cursor_state_checks.py: 87 tests;unit_tests/test: 1225 passed. The checks run without network access in the tests (_no_http).deepcopyof a state that is loaded only once, and>=vs>when pushing equal bounds onto a monotone stack), then 19 for the review fixes (rows 70-88: 18 killed, 1 equivalent). Two guards that survived were removed rather than kept: a redundantcheck_checkpoints = False(the stream has no slices left to check) and a parse guard for a slice start the cursor formatted itself. Each was applied alone to a scratch copy and run against the test file. Rows 11, 12, 25, 26, 35, 36 and 62 were re-run against the current code. Rows 89-104 cover the second review round; rows 30 and 31 no longer apply, because the final-state cursor template they mutated was removed (onlyStateDelegatingStreamneeded it, and it now builds the stream per state message). In the third review round, row 101 was corrected and rows 105-107 added: two more guards that no test failed without, and the test helper that edits the final state.Mutations
test_other_sources_are_not_checkedtest_basic_read_leaves_no_hooks_on_the_sourcetest_basic_read_leaves_no_hooks_on_the_sourcetest_checkpoint_check_does_not_flag_a_per_partition_stream_without_steptest_checkpoint_check_flags_a_per_partition_stream_with_step,test_throttle_lift_emits_more_per_partition_states_within_budget,test_lifted_closes_thin_out_after_the_first_onestest_checkpoint_check_does_not_flag_a_per_partition_stream_without_steptest_throttle_lift_emits_more_per_partition_states_within_budget,test_lifted_closes_thin_out_after_the_first_onestest_throttle_lift_emits_more_per_partition_states_within_budget,test_lifted_closes_thin_out_after_the_first_onestest_throttle_lift_emits_more_per_partition_states_within_budgettest_throttle_lift_emits_more_per_partition_states_within_budgettest_lifted_closes_thin_out_after_the_first_onestest_checkpoint_check_flags_a_per_partition_stream_with_step,test_per_partition_records_below_their_slice_start_are_not_flagged,test_records_below_their_slice_start_are_not_flagged,test_unmatched_partitions_skip_the_per_partition_checkpoint_checktest_checkpoint_check_flags_a_per_partition_stream_with_steptest_checkpoint_bound_comes_from_the_resumed_cursor,test_checkpoint_bound_of_a_state_delegating_stream_comes_from_its_incremental_streamtest_bounds_past_the_cap_use_the_bound_of_an_earlier_statetest_bounds_past_the_cap_never_flag_moretest_a_state_that_moves_back_does_not_lower_the_boundtest_a_record_at_the_resume_start_is_not_flaggedtest_checkpoint_check_does_not_flag_overlapping_slices[api_ignores_until],test_checkpoint_check_does_not_flag_overlapping_slices[no_end_option],test_checkpoint_check_compares_whole_records_without_a_primary_keytest_checkpoint_check_does_not_flag_overlapping_slices[api_ignores_until],test_checkpoint_check_does_not_flag_overlapping_slices[no_end_option],test_checkpoint_check_compares_whole_records_without_a_primary_keytest_a_state_that_moves_back_does_not_lower_the_bound,test_a_copy_after_the_first_skipping_state_does_not_exempt_a_recordtest_a_copy_after_the_first_skipping_state_does_not_exempt_a_recordtest_checkpoint_check_compares_whole_records_without_a_primary_keytest_records_that_do_not_match_the_read_are_checked_by_their_own_valuetest_unmatched_partitions_skip_the_per_partition_checkpoint_checktest_checkpoint_check_does_not_flag_a_per_partition_stream_without_step,test_suites_whose_read_builds_another_source_get_the_concurrent_cursor_part,test_final_state_check_skips_per_partition_streamstest_checkpoint_check_does_not_flag_a_per_partition_stream_without_step,test_final_state_check_skips_per_partition_streams,test_lookback_window_up_to_the_read_duration_rounded_up_passestest_custom_cursor_subclasses_are_skippedtest_checkpoint_bound_of_a_state_delegating_stream_comes_from_its_incremental_streamtest_a_rebuilt_stream_without_a_concurrent_cursor_keeps_the_read_cursortest_a_final_state_the_source_cannot_load_fails,test_a_build_that_sends_http_failstest_a_build_that_sends_http_failstest_a_build_that_sends_http_failstest_a_check_that_crashes_is_reported_as_advisory[add_record],test_a_crash_in_the_checkpoint_walk_keeps_the_other_checks[_resume_start]test_a_check_that_crashes_is_reported_as_advisory[add_state],test_a_crash_in_the_checkpoint_walk_keeps_the_other_checks[_add_checkpoint]test_a_check_that_crashes_is_reported_as_advisory[add_checkpoint_report]test_final_state_is_compared_at_the_state_precisiontest_checkpoint_check_flags_records_past_their_slice,test_checkpoint_check_passes_a_correct_api,test_checkpoint_check_does_not_flag_overlapping_slices[api_ignores_until]test_final_state_behind_the_newest_record_failstest_final_state_without_the_cursor_field_failstest_final_state_the_cursor_cannot_parse_failstest_checkpoint_check_flags_a_per_partition_stream_with_step,test_checkpoint_check_does_not_flag_a_per_partition_stream_without_step,test_suites_whose_read_builds_another_source_get_the_concurrent_cursor_parttest_missing_and_unparseable_cursor_values_are_advisory,test_global_state_is_not_required_when_no_record_has_a_cursor_valuetest_missing_and_unparseable_cursor_values_are_advisorytest_missing_and_unparseable_cursor_values_are_advisory,test_basic_read_warns_on_an_advisory_cursor_state_problemtest_global_substream_cursor_must_be_usedtest_malformed_per_partition_state_fails[entry_without_partition]test_malformed_per_partition_state_fails[entry_without_cursor]test_malformed_per_partition_state_fails[entry_without_partition]test_malformed_per_partition_state_fails[different_partition_keys]test_global_state_must_hold_the_cursortest_global_state_is_not_required_when_no_record_has_a_cursor_valuetest_malformed_per_partition_state_fails[bool]test_malformed_per_partition_state_fails[lookback_longer_than_the_read],test_lookback_window_up_to_the_read_duration_rounded_up_passestest_malformed_per_partition_state_fails[negative]test_lookback_window_up_to_the_read_duration_rounded_up_passestest_missing_parent_state_fails,test_parent_state_is_required_behind_a_grouping_routertest_parent_state_is_not_required_without_incremental_dependencytest_parent_state_is_required_behind_a_grouping_routertest_listed_partition_without_state_failstest_partition_coverage_is_only_checked_for_a_list_router,test_suites_whose_read_builds_another_source_get_the_concurrent_cursor_parttest_a_cursor_that_drops_partitions_on_load_failstest_a_final_state_the_source_cannot_load_fails,test_a_build_that_sends_http_failstest_basic_read_fails_on_an_enforced_cursor_state_problemtest_basic_read_warns_on_an_advisory_cursor_state_problemtest_basic_read_fails_on_an_enforced_cursor_state_problemtest_basic_read_leaves_no_hooks_on_the_sourcetest_basic_read_fails_on_an_enforced_cursor_state_problem,test_basic_read_warns_on_an_advisory_cursor_state_problemtest_a_future_dated_record_does_not_silence_the_checkpoint_check[0],[5],test_a_resume_past_the_end_starts_a_lookback_window_before_the_statetest_a_resume_past_the_end_starts_a_lookback_window_before_the_statetest_records_below_their_slice_start_are_not_flagged,test_per_partition_records_below_their_slice_start_are_not_flaggedis_client_side_incrementaltest_records_below_their_slice_start_are_flagged_under_client_side_incremental,test_client_side_incremental_comes_from_the_incremental_stream_of_a_state_delegating_streamtest_client_side_incremental_comes_from_the_incremental_stream_of_a_state_delegating_streamtest_checkpoint_check_flags_records_past_their_slicetest_records_below_their_slice_start_are_not_flagged,test_per_partition_records_below_their_slice_start_are_not_flagged,test_basic_read_bounds_the_lookback_window_by_the_read_durationtest_records_below_their_slice_start_are_not_flagged,test_per_partition_records_below_their_slice_start_are_not_flaggedtest_a_crash_in_the_checkpoint_walk_keeps_the_other_checks[_resume_start],[_add_checkpoint]test_a_crash_in_the_checkpoint_walk_keeps_the_other_checks[_add_checkpoint]test_basic_read_installs_no_hooks_when_it_does_not_check_the_cursor_state[opted_out],[not_a_basic_read_config]test_basic_read_installs_no_hooks_when_it_does_not_check_the_cursor_state[opted_out]reference_source=Nonetest_basic_read_checks_a_suite_whose_read_builds_another_source,test_basic_read_warns_when_the_reference_source_cannot_buildread_seconds = 0.0test_basic_read_bounds_the_lookback_window_by_the_read_duration,test_basic_read_checks_a_suite_whose_read_builds_another_sourcetest_basic_read_removes_its_hooks_when_the_read_failsglobal_substream_cursorflags droppedtest_basic_read_checks_a_suite_whose_read_builds_another_sourcetest_basic_read_warns_when_the_reference_source_cannot_build>=StateDelegatingStream: bound from the final-state stream (no per-checkpoint build)test_checkpoint_bound_of_a_state_delegating_stream_follows_its_retention_period[states_past_retention-*],test_rebuilding_a_state_delegating_stream_emits_nothing_on_the_read_queue,test_a_resume_without_a_concurrent_cursor_reads_in_full,test_rebuilds_of_a_state_delegating_stream_past_the_cap_never_flag_more...follows_its_retention_period[states_past_retention-per_partition]test_rebuilding_a_state_delegating_stream_emits_nothing_on_the_read_queuetest_rebuilds_of_a_state_delegating_stream_past_the_cap_never_flag_moretest_partition_keys_are_compared_per_parent[one_key_set_per_parent]test_malformed_per_partition_state_fails[different_partition_keys],test_partition_keys_are_compared_per_parent[no_parent_field]test_partition_keys_are_compared_per_parent[different_keys_for_one_parent]test_records_that_do_not_match_the_read_skip_the_checkpoint_check,test_suites_whose_read_builds_another_source_skip_the_checkpoint_checktest_suites_whose_read_builds_another_source_skip_the_checkpoint_checkStateDelegatingStreambounds shared by equal partition statestest_equal_partition_states_in_different_state_messages_resume_apartfull_refresh_stream) instead of the built onetest_per_partition_state_delegating_bound_comes_from_its_incremental_stream(survived until that test: the retention tests use the same definition for both sub-streams)test_a_resume_without_a_concurrent_cursor_reads_in_fullStateDelegatingStreamboundstest_bounds_past_the_cap_never_flag_more[state_delegating]cursorlacks the cursor field passestest_partition_entry_whose_cursor_lacks_the_cursor_field_failsparent_statewithout an incremental parent passestest_parent_state_without_the_incremental_parent_failswith_final_stateedits the first state message equal to the final onetest_with_final_state_edits_the_last_state_messageNot checked
Streams without a
ConcurrentCursororConcurrentPerPartitionCursor(no cursor,FinalStateCursor, custom or file-based cursors) are skipped, never failed. Sources that are not aConcurrentDeclarativeSourceare not checked. The Docker read (test_docker_image_build_and_read) is not checked either, because the checks need the source object in-process to rebuild cursors. This deviates from the issue, which expects the Docker read to get theConcurrentCursorchecks.🤖 Generated with Claude Code