[zahorniak]

Summary

extractJobs and extractWorkflows used api.NewApiExtractor, which reads all rows of the raw table on every run. This PR changes them to api.NewStatefulApiExtractor, the same as the GitHub, GitLab, Jira and YouTrack extractors:

• An incremental run reads only the raw rows added since the previous run.
• A full sync reads all raw rows, as before.
• The extraction logic and the raw params do not change.

A job or workflow can have more than one raw row (the "unfinished details" collectors add a new row when its status changes). The rows are read in id order, so the newest row wins, as before.

On the first run after the upgrade, the state manager starts from the collector state of the same raw table, so the extractor does not do a full sync again.

Tests

• New TestCircleciJobExtractorIncremental and TestCircleciWorkflowExtractorIncremental: a full run, then an incremental run after a new raw row is added, then a full sync. They check that the incremental run reads only the new row and that the newest row wins.
• All CircleCI e2e tests pass on MySQL 8 and PostgreSQL 14. The two new tests fail without the change.

Result on real data (one instance, 2 CircleCI projects, last nightly run, before #9309 is deployed):

Raw table	All rows (read by full extractor)	Rows since the last collector run (read by incremental extractor)
_raw_circleci_api_jobs, project A	7,971,777	132,507
_raw_circleci_api_jobs, project B	2,451,612	37,209
_raw_circleci_api_workflows, project A	722,514	10,836
_raw_circleci_api_workflows, project B	361,614	4,782

The raw params in _devlake_collector_latest_state are the same as the extractor params, so the first run after the upgrade is incremental.

Does this close any open issues?

Closes #9305

Other Information

Follows #9309 (#9304). With both changes, an incremental run in our instance reads about 24k new job rows, not the full raw table (about 10.4M rows).

[klesh]

LGTM
Thanks for your contribution.