Skip to content

docs: add adr for the sched historical import script - #5205

Open
joanagmaia wants to merge 2 commits into
mainfrom
docs/issue-944-sched-import-adr
Open

joanagmaia wants to merge 2 commits into
mainfrom
docs/issue-944-sched-import-adr

Conversation

@joanagmaia

Copy link
Copy Markdown
Contributor

Summary

Adds ADR-0037, the technical approach for importing past Sched events into CDP, for review by a teammate before any code is written. LF events leave Sched in 2027, so instead of an ongoing connector we import the history of LF's 112 Sched events once, with code that can be re-run later. The product spec is here.

Changes

  • docs/adr/0037-sched-historical-import-script.md (status: proposed):
    • A one-time, re-runnable crowd.dev script sends activities to the data sink the same way the Snowflake connectors do, so they follow the existing path to Tinybird and Snowflake.
    • A reviewed mapping file in the repo lists each event and its CDP project, with no secrets.
    • Per-event API keys live in Oracle Vault, one secret per event, read at run time by the job's own identity and deleted after the import. This is the first Vault use in crowd.dev.
    • Source ids include the event and the channel is the subdomain, so re-runs update activities instead of duplicating them.
    • Rejected alternatives: loading into Snowflake first, running a v2 connector once, and keys in a mounted Kubernetes or AWS secret.
  • docs/adr/README.md: index row for ADR-0037.

Numbered 0037 because #5185 already uses 0036.

Organizations and the product sign-off (key type, activity types and scores, registration time, project per event) are still open with product; the ADR notes them and covers the technical approach only. On approval, the reviewer goes into Deciders and the status moves to accepted.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Performance improvement
  • Chore / dependency update
  • Documentation

Related issue

Closes linuxfoundation/lfx-cdp-insights-ops#944

Refs linuxfoundation/lfx-cdp-insights-ops#944

Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>
Copilot AI balanced review requested due to automatic review settings November 4, 2026 15:55
@cursor

cursor Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Documentation-only; no runtime or schema changes until a follow-up implementation PR.

Overview
Adds ADR-0037 (proposed) documenting how to backfill LF’s ~112 Sched events before the 2027 move to Sessionize, without building a long-lived Sched connector.

The chosen approach is a one-time, re-runnable crowd.dev script that pulls each event from the Sched API and ingests activities through DataSinkWorkerEmitter.createAndProcessActivityResult (same path as Snowflake connectors), with event→project mapping in a reviewed repo file and per-event API keys in Oracle Vault (sched-<subdomain>), read at runtime and removed after import. The ADR also records rejected options (Snowflake-first load, one-off v2 connector, mounted K8s/AWS secrets), idempotent source ids/channel design for re-runs, and operational risks (rate limits, auth-in-200 bodies, logging).

docs/adr/README.md gains an index row for ADR-0037.

Reviewed by Cursor Bugbot for commit affa8e2. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This documentation-only PR appears safe to merge.

Summary

Adds proposed ADR-0037 for importing historical Sched events with a re-runnable script.

  • A new ADR records how CDP could import Sched event history.

No actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Mapping["Reviewed event-to-project mapping"] --> Script["Sched import script"]
  Vault["Oracle Vault: event keys"] --> Script
  Sched["Sched API"] --> Script
  Script --> Sink["Existing data sink"]
  Sink --> Tinybird["Tinybird"]
  Sink --> Snowflake["Snowflake"]
Loading

Reviews (2) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

CDP’s first OCI Vault integration requires CloudOps/security validation of the unspecified workload-identity and access-policy design, secret provisioning, and credential deletion before the ADR can be accepted.

No open issues 🎉

What changed in this PR

Documents a proposed approach for importing LF’s historical Sched events into CDP before the move to Sessionize in 2027. This PR adds documentation only; implementation and product decisions remain pending.

Changes:

  • Proposes a one-time, re-runnable script using the existing data-sink ingestion pipeline.
  • Defines event-to-project mapping, per-event Oracle Vault secrets, and safeguards for repeat runs and API limits.
  • Records alternatives, operational risks, and open product questions.
File Description
docs/​adr/​README.md Adds the index entry for proposed ADR-0037.
docs/​adr/​0037-sched-historical-import-script.md Documents the historical import approach, alternatives, constraints, and unresolved decisions.

Review effort: Balanced

…import-adr

Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>

# Conflicts:
#	docs/adr/README.md
Copilot AI balanced review requested due to automatic review settings November 4, 2026 16:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

CDP’s first OCI Vault integration requires human CDP/CloudOps validation of workload identity, access policies, and credential lifecycle before its architecture can be accepted.

No open issues 🎉

Review effort: Balanced

@joanagmaia
joanagmaia requested a review from mbani01 November 4, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants