Skip to content

docs: add interactive Python runtime architecture GitHub Pages - #1949

Open
Varad Meru (vrdmr) wants to merge 7 commits into
devfrom
vrdmr-docs-architecture-pages
Open

Varad Meru (vrdmr) wants to merge 7 commits into
devfrom
vrdmr-docs-architecture-pages

Conversation

@vrdmr

@vrdmr Varad Meru (vrdmr) commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes # N/A

Adds an interactive, commit-driven Architecture Atlas GitHub Pages site (docs/site/) and multi-repository code-graph extractor (eng/docs/) for the Azure Functions Python runtime:

  • Cross-Repository Ecosystem Map: Links live contracts, files, and AST/Rust/Proto symbols across Azure/azure-functions-host (.NET Script Host runtime), Azure/azure-functions-language-worker-protobuf (Host ↔ Worker wire contract), Azure/azure-functions-python-worker, Azure/azure-functions-python-library, and Azure/azure-functions-python-extensions.
  • Three Worker Generations:
    • Python <= 3.12 (workers/azure_functions_worker): Monolithic worker architecture.
    • Python 3.13 - 3.14 (workers/proxy_worker + runtimes/{v1,v2}): Decoupled thin transport proxy + PyPI-shippable runtimes.
    • Python 3.15+ (workers/r2p2 + PyO3 + runtimes/{v1,v2}): Native Rust tonic/prost transport shell embedding CPython (GIL) via PyO3, highlighting both the zero-protobuf hot Datum tuple path (native_invocation.py) and the control/extensions path (protos_adapter.py).
  • Seven Lifecycle Phases per Generation: Dedicated phase-specific Compound Topology diagrams (including bypassed nodes and reverse Worker -> Host RpcLog routing), Multi-Actor Sequence Flows, 3-Era Phase Comparisons, and deterministic Mermaid exports for:
    1. Worker Initialization (StartStream + WorkerInitRequest)
    2. Worker Specialization (FunctionEnvironmentReloadRequest — Consumption placeholder mode)
    3. Worker Metadata & Indexing (FunctionsMetadataRequest)
    4. Worker Functions Load (FunctionLoadRequest — V1 primary)
    5. Worker Invocation (InvocationRequest)
    6. Worker Status (WorkerStatusRequest health/scale probes)
    7. Worker Logs (RpcLog reverse telemetry stream)
  • Python 3.14 Unit Test Teardown Fix (workers/tests/unittest_proxy/test_dependency.py): Adds an autouse preserve_sys_path fixture so sys.path mutations do not break pytest-cov teardown (Path.as_uri() -> import http.client) on Python 3.14.
  • CI Path Exclusions (eng/ci/{public-build,emulator-tests,integration-tests,official-build}.yml): Excludes docs/*, eng/docs/*, and .github/workflows/architecture-pages.yml so docs-only PRs and commits skip the full worker test matrices.
  • Automated GitHub Pages CI/CD (.github/workflows/architecture-pages.yml): Validates extractor unit tests and multi-repo builds on PRs, and publishes to https://azure.github.io/azure-functions-python-worker/ on pushes to dev, daily cross-repo syncs, and manual dispatches. (Note: ensure Settings → Pages → Source is set to GitHub Actions once on the repository).

Validation

  • python3 -m unittest discover -s eng/docs/tests -v — Passed (Ran 7 tests in 0.239s — OK)
  • python3 eng/docs/extract_arch.py — Passed (generated docs/site/assets/architecture-graph.json and docs/site/assets/architecture-data.js across 5 repos, 4 views, 21 lifecycle phase diagrams)

Pull Request Checklist

Host-Worker Contract

  • Does this PR impact the host-worker contract (e.g., gRPC messages, shared interfaces)?
    • If yes, have the changes been applied to:
      • azure_functions_worker (Python <= 3.12)
      • proxy_worker (Python 3.13-3.14)
      • r2p2 (Python >= 3.15)
    • If no, please explain why: Documentation, static architecture site (docs/site/), read-only AST/Rust/Proto extractor (eng/docs/), and GitHub Pages workflow only; no runtime or gRPC wire behavior is modified.

Worker Execution Logic

  • Does this PR affect worker execution logic (e.g., function invocation, bindings, lifecycle)?
    If yes, please answer the following:

Python Version Coverage

  • Does this change apply to all supported worker models?
  • If yes, have the changes been made to:
    • azure_functions_worker (Python <= 3.12)
    • proxy_worker (Python 3.13-3.14)
    • r2p2 (Python >= 3.15)
    • runtimes/v1 / runtimes/v2 (Python >= 3.13)
  • If no, please explain why: Documentation and GitHub Pages site only (covers <= 3.12, 3.13-3.14, and >= 3.15 R2P2 architectures in docs without modifying worker execution code).

Programming Model Compatibility (proxy_worker and r2p2)

  • Does this change apply to both:
    • V1 programming model (runtimes/v1)?
    • V2 programming model (runtimes/v2)?
  • Explanation (if limited to one model): N/A — architecture documentation covers both V1 (runtimes/v1) and V2 (runtimes/v2) across all 7 lifecycle phases.

Add a code-grounded, multi-repository interactive Architecture Atlas for the
Azure Functions Python runtime and wire up automated GitHub Pages builds:

- Extract live Python AST, Rust (R2P2), and FunctionRpc.proto symbols across
  python-worker, python-library, python-extensions, and language-worker-protobuf
  (eng/docs/extract_arch.py + eng/docs/phases_spec.py).
- Model all 3 worker generations (<= 3.12 monolith, 3.13-3.14 proxy_worker +
  runtimes split, 3.15+ R2P2 Rust/PyO3 worker) across all 7 lifecycle phases
  (Init, Specialization, Metadata, Functions Load, Invocation, Worker Status,
  and Worker Logs).
- Render algorithmic compound swimlane topologies, multi-actor sequence flows,
  cross-era phase comparisons, and deterministic Mermaid exports (docs/site/).
- Add unit tests (eng/docs/tests/test_extract_arch.py) and GitHub Pages CI/CD
  (.github/workflows/architecture-pages.yml).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Skip running the full worker build, emulator, and integration matrices when
changes only touch docs/*, eng/docs/*, or .github/workflows/architecture-pages.yml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved defects include a navigation toggle that hides main content, accumulating topology event listeners, and component controls that keyboard users cannot activate.

5 open issues
What changed in this PR

Adds an interactive Architecture Atlas for the Azure Functions Python runtime, connecting four repositories and documenting three worker generations without changing runtime behavior.

Changes:

  • Adds topology diagrams, lifecycle sequences, cross-generation comparisons, and Mermaid exports.
  • Introduces a Python/Rust/Proto extractor, lifecycle specifications, and extractor tests.
  • Adds GitHub Pages automation and excludes documentation-only changes from worker CI matrices.
File Description
eng/​docs/​tests/​test_extract_arch.py Tests graph structure, extraction, and validation.
eng/​docs/​phases_spec.py Defines seven lifecycle phases for each worker generation.
eng/​docs/​extract_arch.py Generates architecture data, source links, and Mermaid exports.
eng/​ci/​public-build.yml Excludes documentation paths from build triggers.
eng/​ci/​official-build.yml Excludes documentation paths from release-build triggers.
eng/​ci/​integration-tests.yml Skips integration triggers for documentation-only changes.
eng/​ci/​emulator-tests.yml Skips emulator triggers for documentation-only changes.
docs/​site/​README.md Documents local preview and Pages setup.
docs/​site/​index.html Loads the site and generated graph data.
docs/​site/​assets/​styles.css Styles the responsive workspace and diagrams.
docs/​site/​assets/​layoutEngine.js Computes graph layouts and edge routes.
docs/​site/​assets/​app.js Implements navigation, diagrams, inspectors, and exports.
docs/​site/​.nojekyll Disables Jekyll processing for the static site.
.gitignore Ignores generated data and sibling checkouts.
.github/​workflows/​architecture-pages.yml Validates, builds, and deploys the Pages site.

Review effort: Balanced

Comment thread docs/site/assets/app.js Resolved
Comment thread docs/site/assets/app.js Resolved
Comment thread docs/site/assets/app.js Resolved
Comment thread docs/site/assets/styles.css Resolved
Comment thread eng/docs/extract_arch.py Resolved
Varad Meru (vrdmr) and others added 2 commits November 15, 2026 19:25
…rs, layout, and AST method anchors

- Add keyboard focus, Enter/Space activation, ARIA labels, <button> boundary
  controls, and focus restoration across rerenders (app.js, styles.css).
- Unify zoom bounds (ZOOM_MIN=0.12, ZOOM_MAX=1.8) across auto-fit, +/- buttons,
  and wheel handlers (app.js).
- Dispose global window mousemove/mouseup listeners on rerender via
  AbortController (app.js).
- Fix collapsed desktop sidebar grid template to 1fr so workspace expands
  instead of collapsing into a 0px column (styles.css).
- Index class method line numbers without truncation before sym_line() lookup
  and add unit test coverage (extract_arch.py, test_extract_arch.py).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…f contract

Add Azure/azure-functions-host as the 5th linked repository across the
ecosystem topology and all 3 worker generations so the .NET Script Host
runtime (WorkerChannel / ServerDuplexChannel) is cleanly distinguished
from the shared FunctionRpc.proto wire contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/site/assets/app.js Resolved
Comment thread docs/site/assets/app.js Resolved
Comment thread eng/docs/extract_arch.py Resolved
Varad Meru (vrdmr) and others added 2 commits November 15, 2026 21:30
…and parse multiline/generic Rust fns

- Derive stepList and activeStep from seqSource so Ecosystem and All Edges
  sequence selections synchronize with Compound Topology (app.js).
- Switch topology stage panning to Pointer Events with pointer capture,
  touch-action: none, and cancel/lostpointercapture cleanup (app.js, styles.css).
- Support generic/lifetime parameters and multiline Rust function declarations
  in extract_rust_file() with regression tests for configure, typed_data_to_tuple,
  and worker_init_req_to_py (extract_arch.py, test_extract_arch.py).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restore sys.path via an autouse monkeypatch fixture so pytest-cov teardown on
Python 3.14 does not fail with ModuleNotFoundError: No module named 'http'.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants