Repository navigation
Non-UTF-8 extension catalog aborts the whole merge #5195
Description
- Repository:
github/spec-kit - Target commit:
1c799bda6c136fcf42fb3475a922ccaac57e6c71 - File:
src/specify_cli/extensions/__init__.py:4700-4725
Summary
non-UTF-8 extension catalog aborts the whole merge.
Root cause
ExtensionCatalog._fetch_single_catalog catches URLError and JSONDecodeError, but the UTF-8 decode raises UnicodeDecodeError, which escapes.
Affected code
src/specify_cli/extensions/__init__.py:
4690
4691 Returns:
4692 URL of the primary catalog
4693
4694 Raises:
4695 ValidationError: If a catalog URL is invalid
4696 """
4697 active = self.get_active_catalogs()
4698 return active[0].url if active else self.DEFAULT_CATALOG_URL
4699
4700 def _fetch_single_catalog(
4701 self, entry: CatalogEntry, force_refresh: bool = False
4702 ) -> Dict[str, Any]:
4703 """Fetch a single catalog with per-URL caching.
4704
4705 For the DEFAULT_CATALOG_URL, uses legacy cache files (self.cache_file /
4706 self.cache_metadata_file) for backward compatibility. For all other URLs,
4707 uses URL-hash-based cache files in self.cache_dir.
4708
4709 Args:
4710 entry: CatalogEntry describing the catalog to fetch
4711 force_refresh: If True, bypass cache
4712
4713 Returns:
4714 Catalog data dictionary
4715
4716 Raises:
4717 ExtensionError: If catalog cannot be fetched or has invalid format
4718 """
4719 import urllib.error
4720
4721 # Determine cache file paths (backward compat for default catalog)
4722 if entry.url == self.DEFAULT_CATALOG_URL:
4723 cache_file = self.cache_file
4724 cache_meta_file = self.cache_metadata_file
4725 is_valid = not force_refresh and self.is_cache_valid()
4726 else:
4727 url_hash = hashlib.sha256(entry.url.encode()).hexdigest()[:16]
4728 cache_file = self.cache_dir / f"catalog-{url_hash}.json"
4729 cache_meta_file = self.cache_dir / f"catalog-{url_hash}-metadata.json"
4730 is_valid = False
4731 if not force_refresh and cache_file.exists() and cache_meta_file.exists():
4732 try:
4733 metadata = json.loads(cache_meta_file.read_text(encoding="utf-8"))
4734 cached_at = datetime.fromisoformat(metadata.get("cached_at", ""))
4735 if cached_at.tzinfo is None:
4736 cached_at = cached_at.replace(tzinfo=timezone.utc)
4737 age = (datetime.now(timezone.utc) - cached_at).total_seconds()
4738 is_valid = age < self.CACHE_DURATION
4739 except (
4740 json.JSONDecodeError,
4741 OSError,
4742 UnicodeError,
4743 ValueError,
4744 KeyError,
4745 TypeError,
4746 AttributeError,
4747 ):
4748 # Cache validity is best-effort: invalid/missing metadata
4749 # fields, an unreadable metadata file (permissions / disk),
4750 # a wrongly-encoded metadata file (written by a tool using
Expected behavior
Extension catalogs warn and skip a failed source and raise only when every selected source fails.
Actual behavior
ExtensionCatalog._fetch_single_catalog parses untrusted response bytes with json.loads but catches only urllib.errors.URLError and json.JSONDecodeError; an invalid-UTF-8 response raises UnicodeDecodeError, and _get_merged_extensions catches only ExtensionError, so a non-UTF-8 source aborts the whole extension catalog operation instead of warning and continuing.
Steps to reproduce
- Run the real
ExtensionCatalogmerge with one catalog whose body is invalid UTF-8 and one healthy catalog. - Observe the returned list and the exception path.
- Compare with an invalid-JSON catalog as a control.
Suggested fix
Translate UnicodeDecodeError into the catalog error type, or catch it alongside JSONDecodeError.
Impact
One corrupt remote catalog disables all extension operations that depend on the merged catalog, including unrelated healthy catalogs.
Activity