Skip to content

Fix post-0.261.439 retest findings in native evidence and tabular planning - #1939

Merged
Paul Lizer (paullizer) merged 2 commits into
microsoft:Developmentfrom
paullizer:paullizer-consolidated-retest-plan
Nov 14, 2026
Merged

Paul Lizer (paullizer) merged 2 commits into
microsoft:Developmentfrom
paullizer:paullizer-consolidated-retest-plan

Conversation

@paullizer

Copy link
Copy Markdown
Collaborator

Summary

A ten-case retest of 0.261.439 (35 original requests, 27 saved assistants) ended with 2 passes, 2 partial results and 6 failures. Three independent read-only investigations compared the saved runs, Application Insights traces and source behavior. This PR changes only what that evidence confirmed was wrong, and adds content-free diagnostics where the evidence could not locate the failing operand, record or stage. It is not a live acceptance result: nothing was deployed or replayed.

  • Native audits carry the facts the retest found missing. A PDF's /Lang and bookmark outline and a PowerPoint's dc:language were returned by the native readers but never reached the canonical audit facts, so an answer could call them missing. They are now typed facts. The PDF bookmark state comes from the catalog's own /Root/Outlines container (absent, present_empty, present_value, or inaccessible with a reason), not from the outline reader's entry count. Audit coverage reports how many whole records were returned from the start of a chain, so a partial read is disclosed as partial. An exact title containing a vertical bar is escaped only inside Markdown table rows, and the answer decoder restores it elsewhere.
  • Calculation plans are corrected earlier and fail in a locatable way. The relational_spec schema now carries the expression grammar and column rules the runtime enforces. When a header is known exactly, a column that no step can have is returned to the planner once before approval. Planning rejections and execution failures carry stable, content-free codes and locations (check, role, category, step), and a schema failure is located in the operation a step names even when the step has several faults.
  • Follow-ups and planner behavior. Earlier source selections are compared independent of list order, so the same documents in another order are no longer reported as conflicting selections (genuinely different selections still require a choice). The planner is told that showing or comparing what a document or slide contains is a chat answer, not a new file or deck.
  • More precise logs, not more volume. New allowlisted sc_* fields name the failed check, role, step and schema branch, and action_results_missing and plan_step_budget rejections log counts. No model text, column name, source value or document ID is recorded. No setting, limit or route changes.

Linked issue

N/A (no associated GitHub issue).

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Diagnostic log fields only; no Admin Setting, governance, deployment or configuration change.

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

VERSION goes from 0.261.442 to 0.261.443. Nothing under deployers/ changed.

Testing / validation

All commands ran offline from the repository root with PYTHONPATH=application\single_app;functional_tests and Cosmos network access blocked (-p test_support.cosmos_network_guard). Each pytest group also ran on a git archive of the unmodified base commit (a47100f7, 0.261.442) with the same files and options, and the failed-test sets are identical on every step.

# Group This branch Unmodified base
1 Native evidence (4 files) 179 passed 151 passed
2 Accessibility and pipeline (4 files) 77 passed, 2 failed 77 passed, 2 failed
3 Orchestration, planner, tabular and action (18 files) 723 passed, 5 failed 648 passed, 5 failed
4 Workflow off-golden (5 files) 10 passed, 28 failed 10 passed, 28 failed
5 Docs coverage and quality (2 files) 13 passed 13 passed
6 Tabular merge (2 files) 60 passed 60 passed
7 check_cosmos_response_boundaries.py 0 review gaps (1104 owners, 430 doubles) same
8 check_cosmos_response_boundaries.py --run-contracts exit 0, 1565 passed (19m37s) not run

The extra passing tests in groups 1 and 3 (28 and 75) are the cases added here. Step 8 ran last, on the final code.

Exact commands
# 1 native evidence
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_native_audit_reliability.py functional_tests/test_native_audit_answer_delivery.py functional_tests/test_native_inspection_request_contract.py functional_tests/test_native_document_fidelity.py

# 2 accessibility and pipeline
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_document_accessibility_extraction.py functional_tests/test_document_style_extraction.py functional_tests/test_excel_document_accessibility_extraction.py functional_tests/test_native_document_fidelity_pipeline.py

# 3 orchestration, planner, tabular and action
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_orchestration_planner_failure_diagnostics.py functional_tests/test_orchestration_planner_contract_recovery.py functional_tests/test_orchestration_plan_schema.py functional_tests/test_orchestration_registry_contract.py functional_tests/test_orchestration_failure_telemetry.py functional_tests/test_orchestration_native_results.py functional_tests/test_orchestration_relational_results.py functional_tests/test_tabular_relational_calculations.py functional_tests/test_tabular_transformations_precision.py functional_tests/test_orchestration_tabular_reconciliation.py functional_tests/test_orchestration_selected_source_planning.py functional_tests/test_orchestration_document_loader_reliability.py functional_tests/test_orchestration_document_action_binding.py functional_tests/test_orchestration_deliverables.py functional_tests/test_orchestration_document_action_end_to_end.py functional_tests/test_orchestration_action_chart_delivery.py functional_tests/test_orchestration_conversation_sources.py functional_tests/test_orchestration_attachment_native_evidence.py

# 4 workflow off-golden
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_orchestration_workflow_runs_off_golden.py functional_tests/test_orchestration_workflow_results_off_golden.py functional_tests/test_orchestration_workflow_handoff_off_golden.py functional_tests/test_orchestration_workflow_run_wait_off_golden.py functional_tests/test_orchestration_workflow_setting_off_golden.py

# 5 docs
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_docs_app_surface_coverage.py functional_tests/test_docs_site_quality.py

# 6 tabular merge
python -m pytest -q -p no:cacheprovider -p no:warnings -p test_support.cosmos_network_guard --tb=line functional_tests/test_tabular_merge_engine.py functional_tests/test_tabular_merge_reconciliation.py

# 7 and 8 Cosmos response boundaries
python scripts/check_cosmos_response_boundaries.py
python scripts/check_cosmos_response_boundaries.py --run-contracts

Failures that also occur on the unmodified base, and are neither fixed nor worsened here:

  • Group 4, all 28: byte-identity goldens whose captured planner system prompt is older than the current prompt text.
  • Group 3: test_orchestration_tabular_reconciliation.py::test_composition_profile_validator_accepts_only_valid_mappings, test_orchestration_deliverables.py::test_an_invalid_deliverables_plan_gets_exactly_one_repair_call, test_orchestration_action_chart_delivery.py::test_a_gather_only_plan_gets_one_correction_round, and test_orchestration_document_action_end_to_end.py::test_native_auto_turn_reaches_a_saved_cited_answer[False-True-pdf] and [False-True-docx] (the last two pass when run alone and fail only in this run order).
  • Group 2: test_excel_document_accessibility_extraction.py::test_report_and_citations_reuse_authorized_contract and test_native_document_fidelity_pipeline.py::test_old_source_metadata_cannot_describe_a_sanitized_active_file; both raise a NameError for validate_native_report_sections at functions_search_service.py:1277.

Manual validation: bookmark states were also checked against a pypdf-written emptied outline root and, read-only through scratch copies, the two original retest PDFs (tagged: /Outlines with 4 entries is present_value; untagged: no /Outlines is absent). No source file was modified. The two review-driven corrections and the header-spelling guard below were written test-first and seen failing before their fixes (13, 5 and 1 failures).

Not run: test_tabular_relational_analysis_helpers.py cannot be collected without real Cosmos credentials. No deployment, live replay or source upload was performed; unchanged-source live acceptance needs a separate approval.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Release notes are not included yet; an entry under 0.261.443 is still to be written. Feature docs updated: DOCUMENT_ACCESSIBILITY_AND_STRUCTURE_EXTRACTION.md, MULTI_FILE_TABULAR_CALCULATIONS.md and the logging reference (docs/reference/logging-tags.md, including a corrected debug_only note). New fix doc: docs/explanation/fixes/POST_439_RETEST_REMEDIATION_FIX.md.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

No Flask route, settings payload, frontend asset or credential handling is added or changed. New log fields are allowlisted closed tokens or counts, and the added content was scanned for local paths, session identifiers and credential-like strings.

Notes for reviewers

  • Deliberately not changed, because the evidence does not locate them: the exact failed relational specs and operands; the competing source-continuation records (a genuine conflict still requires a choice); whether two actions made zero calls or lost a result (the no-usable-result refusal is unchanged, only counts are added); the first wrong stage of the supplier-identity merge; the live NASA reference handle (the strict Office package guard is unchanged, and the original package declares seven slide-master parts that are absent); and the stage that added the extra title backslash. These need the new diagnostics observed on a later, separately approved, unchanged-source run.
  • Preserved: the arithmetic engine, tool catalog and action budgets, authorization and source-version guards, no-usable-result refusals, strict package validation, XSS protection, and the optional complete-header assertion.
  • Review-driven corrections included: the PDF bookmark state is read from the catalog (an explicitly empty outline is present_empty and a malformed one is inaccessible, never a false absent), and the schema locator now prefers the alternative whose own constant matches the step's op before closeness, so correction guidance cannot tell the planner to replace a derive with filter fields.
  • One extra guard found while auditing the column preflight: the planning preview only trims header names, but the calculation reader also normalizes Unicode and collapses repeated spaces, so a preview name can differ from the name the reader uses. The preflight treats a header as exact only when both agree (new public helper display_header_name in functions_tabular_merge.py). The older datasets[].columns assertion check has the same divergence and is intentionally left unchanged.
  • Native inspection version is now 1.2, so a continuation cursor issued by an earlier version is refused as a changed source instead of silently shifting.
  • Cosmos inventory: functional_tests/cosmos_response_boundaries.json changes only the review notes and module fingerprints of five unchanged owners in functions_orchestration_context.py, whose enclosing module changed.

…nning

Carry a PDF /Lang and bookmark outline and a PowerPoint language into the canonical native audit facts, taking the bookmark state from the catalog /Outlines container rather than the outline reader entry count. Report how many native records were returned from the start of a chain, and escape a vertical bar only inside Markdown table rows. Give the relational_spec schema its expression grammar and column rules, correct a column that no step can have once before approval when the header is known exactly, and attach content-free codes and locations to planning rejections and execution failures, including a schema locator that prefers the operation a step names. Compare earlier source selections independent of list order, add content-free counts to missing action results and step-budget rejections, and tell the planner that showing a document content is a chat answer, not a new file.

Offline only: nothing was deployed or replayed, and the failed operands, competing continuation records, action call origin and title backslash stage stay evidence-gated. Bump to 0.261.443 and renew the Cosmos inventory review for the five unchanged owners in functions_orchestration_context.py. Docs: new POST_439_RETEST_REMEDIATION_FIX.md plus updates to the tabular, accessibility and logging references.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the CodeQL py/import-and-import-from note on three tests added in this PR, which imported functions_accessibility locally while the module also used a from-import. Patch the shared fitz module directly instead; it is the same object functions_accessibility uses, so the simulated reader failures and disagreements are unchanged. Test-only: no application behavior or version change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit ca1e4b0 into microsoft:Development Nov 14, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants