refactor(data-grid)!: rebuild the macOS data grid on one model and native AppKit

.claude/rules/data-grid.md

@@ -1,21 +1,68 @@
 ---
 paths:
-  - "apple/TablePro/Views/Results/**/*"
+  - "apple/TablePro/Views/DataGrid/**/*"
   - "apple/TablePro/Core/DataGrid/**/*"
+  - "apple/TablePro/Views/Results/**/*"
+  - "apple/TableProTests/Views/DataGrid/**/*"
+  - "apple/TableProTests/Core/DataGrid/**/*"
   - "apple/TableProTests/Views/Results/**/*"
 ---
 
 # Data grid
 
-- **Selection indices are display positions**, not `TableRows.rows` indices. Resolve them through `DisplayRowMapping` (or `TableViewCoordinator.displayRow(at:)` and `tableRowsIndex(forDisplayRow:)`) before reading or changing a row; they match only while no sort or value filter applies.
-- **Cells are drawn, not mounted.** `tableView(_:viewFor:row:)` returns nil for data columns and `DataGridRowView` draws the visible cells, and every column stays attached. Never hide off-screen columns with `NSTableColumn.isHidden`: each write costs O(attached columns).
-- **Find data columns through `DataGridColumnPool.presentsColumn` and `firstPresentedColumnIndex`**, never a fixed index into `tableColumns` (the row-number column and pool slots sit there too).
-- **The header draws all of its own chrome** through `SortableHeaderChrome`: the cell never calls `super.draw`, and `highlightedTableColumn` stays unset, because AppKit paints a fixed-height band that does not fit the taller header. Sort state goes through `SortableHeaderView.applySortState(_:schema:)`, which also sets each header cell's accessibility sort direction.
-- **A themed header draws its own cells.** `NSTableHeaderView.draw` paints AppKit's band over the header fill before the cells, so while `dataGrid.headerBackground` is themed `SortableHeaderView` draws each column's header cell itself, the dragged one included; an unthemed header keeps `super.draw`.
-- **Row selection is `NSTableRowView`'s own** unless the theme overrides `gridSelection` or `gridInactiveSelection`, so the default theme follows Accent and focus. The gutter and a selected column heading read `DataGridBodyChrome.selectionColor(emphasized:scope:)` so they match the row. A theme change repaints visible rows; it never reloads data.
-- **A translucent theme selection is flattened once.** The row, the gutter and a heading sit on three different bases, so `DataGridBodyChrome.themedSelectionColor(emphasized:scope:)` lays the selection over the grid background and all three paint that opaque color with any alpha.
-- **A range outline stays apart from its fill.** Natively a multi-cell range is outlined in the selection color at 70%. Under a themed `dataGrid.selection` that outline would vanish into the opaque fill, so `GridSelectionOverlay.rangeBorderColor(_:)` takes the active cell's border color instead (`dataGrid.focusBorder`, else Accent) at 70%.
-- **The grid reads its colors through its `ThemeScope`.** `DataGridView(themeScope:)` hands it to `TableViewCoordinator`, and every grid color read goes through `coordinator.themeScope`, `KeyHandlingTableView.themeScope` and `SortableHeaderView.themeScope` included. `ThemeEngine.shared` supplies only the fonts and the cache counters that follow a system color change. A new scope repaints in place with no reload. Read `scope.resolved` once per draw, not once per cell. Scopes: `.claude/rules/theme.md`.
-- **`gridStyleMask` stays empty; `DataGridBodyChrome` draws column separators and the stripes.** AppKit's vertical grid lines add one subview per column and make every layout pass quadratic. Take separator positions from `presentsColumn` and `rect(ofColumn:)`.
-- **Scroll the grid with `NSView.scroll(_:)`**, never `clipView.scroll(to:)` plus `reflectScrolledClipView`, which leaves the header behind.
-- **Accessibility goes through `DataGridCellAccessibilityView`**, mounted for on-screen rows once `DataGridAccessibility.isActive`; `NSTableView` builds its cell tree from views only. Keep that path working whenever row drawing changes.
+Three layers, knowledge pointing one way: an owner's adapter, then `DataGridModel` (`Core/DataGrid`), then the AppKit view (`Views/DataGrid`). The value editors and viewers stay in `Views/Results`.
+
+## Model
+
+- **`DataGridModel` holds the grid's state** and outlives every mount: column set, row order, selection, sort, value filter, column layout, display formats, find and the edit session. Its owner holds it strongly; the data tab keeps one per tab on `MainContentCoordinator`.
+- **An owner talks to the model, never to the view.** It implements `DataGridDataSource` and `DataGridActions` in an adapter class and calls `reloadRows()` when its rows were replaced or `apply(_ delta:)` when they changed. The model keeps the selection by identity and sends the mounted view one `DataGridChange`.
+- **Rows are `RowID`, columns are `DataGridColumnID`.** `DataGridRowOrder` maps display rows, and `DataGridColumnSet` is the only converter between data index, display position and pane. A bare `Int` is a display position, valid only inside one call: capture the IDs before a popover, a sheet or an `await`.
+- **Every edit is one `model.commit(_:actionName:)`** of `DataGridCellEdit`s, taken by IDs when the gesture began. A command over many cells is one call, so one undo step.
+- **The selection is `model.selection`, changed through `GridSelectionReducer`**, a pure engine in `Core/DataGrid/Engine` with its own tests. The tables' `selectedRowIndexes` mirror the rows it spans; a change AppKit makes itself (Up, Down, type-select) goes through `rowsSelectedNatively` first.
+- **A selection says what it is in `GridSelection.Mode`**: a click selects the row with the cell cursor on the clicked cell (`.cursor`), a drag, Shift or Command gesture makes a cell range (`.cells`), and row numbers or Select All pick rows (`.rows`). Copy, Delete and the row commands read the mode, never the rectangle's shape.
+- **What a grid offers is `configuration.commands`.** Menus and validation read it, because a defaulted protocol method cannot be detected at runtime.
+- **State persists through `model.store`.** Only the data tab of a real, non-temporary table has `layoutScope` `.table`; every other grid is `.ephemeral` and never writes a table's saved layout or display formats.
+
+## Panes
+
+- **Two tables, one grid.** The main pane's `DataGridTableView` holds every column, the row numbers first. The frozen pane is a second table laid over the main pane's leading edge, showing the row numbers and the frozen columns. Keys, find, copy and VoiceOver work on the main table.
+- **The main table is the only first responder.** The frozen table refuses it and hands its presses to the controller. `DataGridContainerView.gridHasFocus` is the one focus flag: both panes' row views take `isEmphasized` from it, and so does the active-cell border.
+- **Row height, row count, reloads, inserts and removes reach both panes in one call**, so their document heights never differ.
+- **Vertical scroll stays in step through bounds-change notifications**: each pane's clip moves the other with `clipView.scroll(to:)` and `reflectScrolledClipView`, only when its y differs. Anywhere else scroll with `NSView.scroll(_:)`: moving a clip's x that way leaves the header behind.
+- **The frozen pane forwards every wheel event to the main scroll view**, so one momentum engine moves both.
+- **Frozen columns take at most two thirds of the grid** (`DataGridMetrics.maximumFrozenShare`). Columns past that scroll, and Freeze Up to This Column is refused in the menu.
+- **`DataGridTableView.scrollToVisible(_:)` widens the rect leftward by the frozen width**, so a revealed column never parks under the frozen pane.
+- **A column moves only within its side of the freeze line.** A freeze or unfreeze rebuilds both panes' column lists from the set, writing width and position explicitly: remove-then-add alone strands a column at the end and resets its width.
+
+## Cells
+
+- **Cells are drawn, not mounted.** `DataGridRowView` draws its pane's cells through `DataGridCellRenderer`. `tableView(_:viewFor:row:)` returns nil except for the cell being edited and the accessibility stand-ins: a view per cell costs 45 ms a frame at 1,000 rows by 200 columns against 1.5 ms drawn.
+- **A change repaints, it never reloads.** It reaches pixels as `needsDisplay` on the row views it names. `reloadData(forRowIndexes:columnIndexes:)` only mounts or unmounts a cell view, since reloading an edited row ends the edit. A full `reloadData()` drops the table selection without telling anyone, so the controller writes it back.
+- **Every column stays attached.** Never hide an off-screen column with `NSTableColumn.isHidden`: each write costs O(attached columns). Columns come from each pane's pool, keyed by `DataGridColumnID`, so a re-run of the same result does not flicker (#951).
+- **Resolve a table column through `DataGridPane.slot(atTableIndex:columns:)` and `tableIndex(forDisplayPosition:columns:)`**, never a fixed index into `tableColumns`: the row-number column sits first and the frozen pane holds a subset.
+- **`gridStyleMask` stays empty.** AppKit's vertical grid lines keep a subview per column (518 ms a layout at 500 columns). `DataGridBodyChrome` draws the separators in each row, and the stripes and separators past the last row.
+- **Selection is the system's.** `drawSelection(in:)` fills the whole row in a row selection and the selected columns' span in a cell range: `selectedContentBackgroundColor` when emphasized, else `unemphasizedSelectedContentBackgroundColor`. The active cell is an inner stroke in `keyboardFocusIndicatorColor`, 2 pt with focus and 1 pt without, in the selected text color over an emphasized fill. No outline goes round a range.
+- **Never draw the active cell with the focus ring API.** From a row view the ring loses its top and bottom; on an `NSTableView` it draws nothing.
+- **Colors come from `model.themeScope`**, read once per draw. A theme, Accent or Increase Contrast change repaints the visible rows and never reloads data. `ThemeEngine.shared` supplies only the fonts. No theme slot colors the header, the selection, selected text or the active-cell border (`.claude/rules/theme.md`).
+- **The row view draws the column a header drag is moving**, at the header's `draggedDistance`: drawn cells have no views for AppKit to float.
+
+## Header
+
+- **Stock `NSTableHeaderView` and `NSTableHeaderCell`.** `DataGridHeaderView` overrides only `menu(for:)` and `viewWillDraw`, which repaints the body on each step of a column drag. AppKit draws the primary sort arrow; the controller sets each header cell's `accessibilitySortDirection`, which AppKit leaves unknown.
+- **The model's sort is the source.** `tableView.sortDescriptors` is written from `store.sortState` under `isApplyingSortDescriptors`, because an assignment inside the callback fires it again. A plain click keeps only the clicked key and flips it on a repeat; an Option-click keeps the key AppKit appended (`HeaderSortCycle`). Shift-click reaches no sort callback, and Command-click selects the column.
+- **Header text is the column name.** The type, the comment and, while several keys sort, the sort order go in `headerToolTip`; an active value filter adds a funnel symbol attachment to the title, in the header text color.
+
+## Keyboard, mouse and menus
+
+- **Behavior comes from responder actions, never key codes.** `keyDown` sends Return, Enter, Tab, Backtab, Delete, Forward Delete, Left, Right, and Shift-Up or Shift-Down over a cell range through `interpretKeyEvents`. Everything else goes to `super`, which keeps AppKit's row movement and type-select.
+- **A modifier test masks `.capsLock` and `.function`**: compare within `[.command, .shift, .option, .control]` only. Forward Delete carries `.function`.
+- **Copy, Paste, Delete, Select All, Find, Copy Special, Set Value and zoom are actions the grid answers itself**, with `NSMenuItemValidation`, so they work in every window. The context menu's Paste takes the keyboard's path. A window's own handler runs only while the grid is not focused.
+- **The context menu is built from `clickedRow` and `clickedColumn`** in `menuNeedsUpdate(_:)`. It acts on the selection when the clicked cell is inside it, else on the clicked cell or row; the selection never moves.
+- **A press in either pane resolves to one `GridCoord`** through the container and the column set. A drag autoscrolls the main pane and crosses panes.
+
+## Accessibility
+
+- **AppKit builds the AXCell tree from cell views only**, so while `DataGridAccessibility.isActive` the main table mounts a `DataGridAccessibilityCellView` (identifier `data-grid-cell`) for each on-screen cell. It draws nothing and takes no clicks. Keep that path working whenever row drawing changes.
+- **A label reads "Row N, column M: value"**: N is the row number on screen, page offset included, and M the display position. UI tests find cells by it (`.claude/rules/ui-tests.md`, The grid).
+- **The frozen pane is hidden from accessibility.** A frozen column's stand-in in the main table reports the frame the frozen pane shows it at.
+- **Only the viewport mounts.** Mounting for every row starved the main thread behind a 1,000-row page (#2392); rows scrolled into view remount.

.claude/rules/theme.md

@@ -19,16 +19,17 @@ paths:
 
 - `TableProTheme` is Foundation only, so the iOS app can adopt it; AppKit and SwiftUI code stays in `apple/TablePro/Theme`.
 - **`ThemeSlot` is the one registry.** A slot exists only while a view reads it: removing its last reader removes the slot.
-- `isOverridden(_:)` means the slot draws a theme color, set or derived: ask it before swapping a native drawing path (row selection, caret, header). Only the theme editor asks `hasExplicitColor(_:)`.
-- **Adding a slot bumps `ThemeSchema.current`.** The decoder reads every schema from 3 up and refuses a newer one or an unknown key, rather than load a file without colors its author wrote. The bump renames the sync content kind (Sync).
+- `isOverridden(_:)` means the slot draws a theme color, set or derived: ask it before swapping a native drawing path (a code table's selection, the caret). Only the theme editor asks `hasExplicitColor(_:)`.
+- **Adding a slot bumps `ThemeSchema.current`.** `ThemeFileFormat` reads only the current schema: an older or newer one, or an unknown key, is refused rather than loaded without colors its author wrote, so a bump stops every older file from loading. The bump renames the sync content kind (Sync).
+- **Removing a slot never bumps the schema.** Its key joins `ThemeFileFormat.retiredKeys`, which the decoder drops, so a file that still carries it loads without it.
 - **An imported file is untrusted.** `ThemeFileImporter` checks the size before it reads, mints a fresh id every time, and never follows a VS Code `include` out of the file's folder.
 
 ## Reading colors
 
 - **A scoped carrier reads through its `ThemeScope`, never `ThemeEngine.shared`.** Only the theme editor preview uses `.fixed(_:)`, and a new definition is a new scope.
 - `palette.color(_:otherwise:)` returns a new object on every call: never compare it to skip a write. Bake `resolved` into a `CGColor` only inside the drawing view's appearance (`resolved.cgColor(_:otherwise:in:)`).
 - **Hold every AppKit carrier** (`ThemeTextViewStyle`, `ThemeTableChrome`, never on a source list) **as a stored property.** Its subscription and row views hold it weakly, so one held in a local stops at the first theme change.
-- `NSTableHeaderView` paints over anything beneath it, so `ThemeTableChrome` swaps in `ThemeTableHeaderView` while the header is themed.
+- **Header and selection are always the system's.** No slot colors a table header, a records row's selection or its selected text, or the data grid's active-cell border, so they follow Accent, focus and Increase Contrast. `ThemeTableChrome` never replaces a header; only a code table recolors its selection, with `editor.selection`.
 - **A theme change has to reach the view.** SwiftUI caches `Color(nsColor:)` by object identity, so a view reading the engine holds `@ObservedObject private var themeEngine = ThemeEngine.shared`; an AppKit view that bakes a color re-bakes on `AppEvents.shared.themeChanged`. `withAlphaComponent` on a system color outside `NSColor(name:dynamicProvider:)` freezes it.
 - **No theme color on selectable-row text**: it does not turn white on the emphasized selection. Never put `.themeForeground` on a container of selectable rows; `ThemeTableChrome.recordsTextSlot(...)` picks a records cell's slot.
 

.claude/rules/ui-tests.md

@@ -52,7 +52,9 @@ The UI plan holds end-to-end flows only. Swift Testing rules, isolation and shar
 ## The grid
 
 - Every grid read and click goes through `FixtureGrid` (`cell`, `value`, `waitForValue`, `clickCell`, `editCell`, `clickHeader`) or `gridPoint(in:of:dy:)`.
-- `FixtureGrid` matches a cell by the identifier `DataGridCellAccessibilityView`, the leaf that carries the value, not the AXCell wrapper AppKit puts around it.
+- `dataGrid(in:)` matches the main table, `data-grid`, which publishes every column. The frozen pane laid over its leading edge, `data-grid-frozen`, is hidden from accessibility.
+- `FixtureGrid` matches a cell by the identifier `data-grid-cell`, the leaf that carries the value, not the AXCell wrapper AppKit puts around it, and by the label `Row r, column c: `. Both count from 1: `r` is the row number on screen, page offset included, and `c` the column's display position, so hiding or moving a column renumbers the ones after it.
+- A plain `clickHeader` sorts by that column alone and flips it on the next click; `clickHeader(_:modifiers: .option)` adds it as another key. A right-click opens the header or cell menu and never moves the selection.
 - `FixtureFile.rows` and `FixtureFile.execute` read and write the fixture database directly, to set up or check data without the UI.
 - Never call `click()` on a grid element. Rows and cells read as obscured by their columns, a row element has an empty frame so an offset click lands on the window's close button, and a header never becomes hittable. The app side is in `.claude/rules/data-grid.md`.
 - A point offset from the grid measures from its top edge, above the header. Take `dy` from a row's frame (`row.frame.midY - grid.frame.minY`); a small constant hits the header and opens the column menu.

apple/Packages/TableProCore/Sources/TableProTheme/Import/VSCodeThemeImporter.swift

@@ -167,7 +167,7 @@ private extension ThemeDefinition {
             switch slot.fallback {
             case .system, .defaultSyntax:
                 continue
-            case .slot, .derived, .paired:
+            case .slot, .derived:
                 var unset = result
                 unset.overrides[slot] = nil
                 if unset.value(for: slot) == result.value(for: slot) {

apple/Packages/TableProCore/Sources/TableProTheme/Import/VSCodeThemeMapping.swift

@@ -22,9 +22,7 @@ private extension VSCodeColorSource {
 }
 
 internal extension ThemeSlot {
-    // An empty list leaves the slot to its fallback. So does a source list that stops short: the
-    // alternate row, the header, the header text, the bools and the selection text are derived by
-    // `fallback` from the background, text and selection, which is what VS Code paints there too.
+    // A list that stops short leaves the slot to `fallback`, which derives the stripe and the bools the way VS Code paints them.
     var vscodeSources: [VSCodeColorSource] {
         switch self {
         case .windowBackdrop:
@@ -84,20 +82,8 @@ internal extension ThemeSlot {
             return [.color("editor.foreground")]
         case .gridAlternateRow:
             return [.color("keybindingTable.rowsBackground"), .color("tree.tableOddRowsBackground")]
-        case .gridHeaderBackground:
-            return [.color("keybindingTable.headerBackground"), .color("tree.tableOddRowsBackground")]
-        case .gridHeaderText:
-            return [.color("foreground")]
         case .gridLine:
             return [.color("tree.tableColumnsBorder")]
-        case .gridSelection:
-            return [.color("list.activeSelectionBackground")]
-        case .gridSelectionText:
-            return [.color("list.activeSelectionForeground")]
-        case .gridInactiveSelection:
-            return [.color("list.inactiveSelectionBackground")]
-        case .gridFocusBorder:
-            return [.color("list.focusOutline"), .color("focusBorder")]
         case .gridNullValue:
             // The grid draws NULL as placeholder text. `descriptionForeground` equals the text color in
             // One Dark Pro, so NULL would not read as absent.