Skip to content

Non-UTF-8 extension catalog aborts the whole merge #5195

Closed

Description

@soarkit
  • Repository: github/spec-kit
  • Target commit: 1c799bda6c136fcf42fb3475a922ccaac57e6c71
  • File: src/specify_cli/extensions/__init__.py:4700-4725

Summary

non-UTF-8 extension catalog aborts the whole merge.

Root cause

ExtensionCatalog._fetch_single_catalog catches URLError and JSONDecodeError, but the UTF-8 decode raises UnicodeDecodeError, which escapes.

Affected code

src/specify_cli/extensions/__init__.py:

 4690  
 4691          Returns:
 4692              URL of the primary catalog
 4693  
 4694          Raises:
 4695              ValidationError: If a catalog URL is invalid
 4696          """
 4697          active = self.get_active_catalogs()
 4698          return active[0].url if active else self.DEFAULT_CATALOG_URL
 4699  
 4700      def _fetch_single_catalog(
 4701          self, entry: CatalogEntry, force_refresh: bool = False
 4702      ) -> Dict[str, Any]:
 4703          """Fetch a single catalog with per-URL caching.
 4704  
 4705          For the DEFAULT_CATALOG_URL, uses legacy cache files (self.cache_file /
 4706          self.cache_metadata_file) for backward compatibility. For all other URLs,
 4707          uses URL-hash-based cache files in self.cache_dir.
 4708  
 4709          Args:
 4710              entry: CatalogEntry describing the catalog to fetch
 4711              force_refresh: If True, bypass cache
 4712  
 4713          Returns:
 4714              Catalog data dictionary
 4715  
 4716          Raises:
 4717              ExtensionError: If catalog cannot be fetched or has invalid format
 4718          """
 4719          import urllib.error
 4720  
 4721          # Determine cache file paths (backward compat for default catalog)
 4722          if entry.url == self.DEFAULT_CATALOG_URL:
 4723              cache_file = self.cache_file
 4724              cache_meta_file = self.cache_metadata_file
 4725              is_valid = not force_refresh and self.is_cache_valid()
 4726          else:
 4727              url_hash = hashlib.sha256(entry.url.encode()).hexdigest()[:16]
 4728              cache_file = self.cache_dir / f"catalog-{url_hash}.json"
 4729              cache_meta_file = self.cache_dir / f"catalog-{url_hash}-metadata.json"
 4730              is_valid = False
 4731              if not force_refresh and cache_file.exists() and cache_meta_file.exists():
 4732                  try:
 4733                      metadata = json.loads(cache_meta_file.read_text(encoding="utf-8"))
 4734                      cached_at = datetime.fromisoformat(metadata.get("cached_at", ""))
 4735                      if cached_at.tzinfo is None:
 4736                          cached_at = cached_at.replace(tzinfo=timezone.utc)
 4737                      age = (datetime.now(timezone.utc) - cached_at).total_seconds()
 4738                      is_valid = age < self.CACHE_DURATION
 4739                  except (
 4740                      json.JSONDecodeError,
 4741                      OSError,
 4742                      UnicodeError,
 4743                      ValueError,
 4744                      KeyError,
 4745                      TypeError,
 4746                      AttributeError,
 4747                  ):
 4748                      # Cache validity is best-effort: invalid/missing metadata
 4749                      # fields, an unreadable metadata file (permissions / disk),
 4750                      # a wrongly-encoded metadata file (written by a tool using

Expected behavior

Extension catalogs warn and skip a failed source and raise only when every selected source fails.

Actual behavior

ExtensionCatalog._fetch_single_catalog parses untrusted response bytes with json.loads but catches only urllib.errors.URLError and json.JSONDecodeError; an invalid-UTF-8 response raises UnicodeDecodeError, and _get_merged_extensions catches only ExtensionError, so a non-UTF-8 source aborts the whole extension catalog operation instead of warning and continuing.

Steps to reproduce

  1. Run the real ExtensionCatalog merge with one catalog whose body is invalid UTF-8 and one healthy catalog.
  2. Observe the returned list and the exception path.
  3. Compare with an invalid-JSON catalog as a control.

Suggested fix

Translate UnicodeDecodeError into the catalog error type, or catch it alongside JSONDecodeError.

Impact

One corrupt remote catalog disables all extension operations that depend on the merged catalog, including unrelated healthy catalogs.

Activity

  1. added
    triage-must-haveVerdict: high-value, important work for Spec Kit — do first
    on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage-must-haveVerdict: high-value, important work for Spec Kit — do first

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions