Update findAndRerank sort contract by Hazel-Datastax · Pull Request #2675 · stargate/data-api

Hazel-Datastax · 2026-11-11T21:45:42Z

What this PR does:
his PR changes the findAndRerank sort contract. A sort can now be a single leg sort: $vectorize, $vector, $lexical, or field sorts such as {"age": -1}.

FindAndRerankSort is now a sealed interface with one record for each form of sort. See problem 1 in #2673 for why the old class does not work.

Out of scope:

Planning and running single leg sorts (Single-leg retrieval #2602, Filter retrieval #2603). Until then, they are rejected with UNSUPPORTED_FIND_AND_RERANK_SORT.
The rerankCandidateLimit option.

Known problems
We keep these two problems to keep this PR small. See problem 2 and the proposed design in #2673 for the details and the fix.

MCP clients cannot see the error for a single leg sort with a syntax error. They only get "Internal error". HTTP clients get the full error.
Some checks in the deserializer are a copy of the checks in CollectionSortClauseBuilder.

Which issue(s) this PR fixes:
Part of #2601

Checklist

Changes manually tested
Automated Tests added/updated
Documentation added/updated
CLA Signed: DataStax CLA

FindAndRerankSort is now a sealed interface with one record for each form of sort: HybridText and HybridObject for $hybrid, and VectorLeg, LexicalLeg and FilterLeg for the single leg sorts. SortExpression2 holds one single leg sort expression. RerankLegMode is removed. The deserializer reads $vectorize, $vector, $lexical and field sorts and checks their syntax. For field sorts and $binary vectors it calls the checks find uses, which are now public static in the find sort builders. The checks for $lexical, $vector and $vectorize are a copy of the checks in find. $hybrid is not changed. The planner still rejects every single leg sort with UNSUPPORTED_FIND_AND_RERANK_SORT, after the option checks. Behaviour changes: - A sort without $hybrid with a syntax error gets a SORT_CLAUSE_* code, not REQUEST_STRUCTURE_MISMATCH. On MCP this error is "Internal error". - A sort without $hybrid with valid syntax gets UNSUPPORTED_FIND_AND_RERANK_SORT, or an option error before it. - The message of UNSUPPORTED_FIND_AND_RERANK_SORT changed.

VectorLeg, LexicalLeg and FilterLeg no longer take the metrics features as a parameter. Each record works out its features from its type, so the features always match the sort. The $hybrid records still get their features when the sort is read. The class javadoc of FindAndRerankSort now marks the missing single leg schema as a TODO.

github-actions · 2026-11-11T21:54:15Z

📈 Unit Test Coverage Delta vs Main Branch

Metric	Value
Main Branch	56.74%
This PR	57.07%
Delta	🟢 +0.33%
✅ Coverage improved!

github-actions · 2026-11-11T21:54:17Z

Unit Test Coverage Report

Overall Project	57.07% `-0.01%`	🍏
Files changed	98.16%	🍏

File	Coverage
SortExpression2.java	100%	🍏
FindAndRerankSortClauseDeserializer.java	100%	🍏
FindAndRerankPlanner.java	98.84% `-0.78%`	🍏
FindAndRerankSort.java	96.1% `-3.9%`	🍏
SortClauseBuilder.java	83.97%	🍏
CollectionSortClauseBuilder.java	83.43%	🍏

The deserializer no longer has its own branch for an empty sort. The single leg code reads {} as the filter leg with no sorts, which is the same value as NO_ARG_SORT.

deserialize only checks for $hybrid, then calls deserializeMultiLeg or deserializeSingleLeg. The $hybrid code moved without changes. The class javadoc of the deserializer is shorter. The NOTE about the copy of the find checks keeps the three differences and the TODO.

github-actions · 2026-11-11T22:39:51Z

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Metric	Value
Main Branch	72.98%
This PR	72.75%
Delta	🔴 -0.23%
⚠️ Coverage decreased

github-actions · 2026-11-11T22:39:53Z

Integration Test Coverage Report (dse69-it)

Overall Project	72.75% `-0.45%`	🍏
Files changed	30.23%	❌

File	Coverage
FindAndRerankPlanner.java	85.47% `-4.26%`	🍏
SortClauseBuilder.java	55.73%	🍏
CollectionSortClauseBuilder.java	55.25%	🍏
FindAndRerankSortClauseDeserializer.java	38.7% `-33.27%`	❌
FindAndRerankSort.java	21.65% `-78.35%`	❌
SortExpression2.java	0%	❌

github-actions · 2026-11-11T22:41:23Z

📉 Integration Test Coverage Delta vs Main Branch (hcd-it)

Metric	Value
Main Branch	74.33%
This PR	74.13%
Delta	🔴 -0.19%
⚠️ Coverage decreased

github-actions · 2026-11-11T22:41:26Z

Integration Test Coverage Report (hcd-it)

Overall Project	74.13% `-0.4%`	🍏
Files changed	38.28%	❌

File	Coverage
FindAndRerankPlanner.java	98.06% `-0.78%`	🍏
CollectionSortClauseBuilder.java	71.27%	🍏
SortClauseBuilder.java	55.73%	🍏
FindAndRerankSortClauseDeserializer.java	45.75% `-26.58%`	❌
FindAndRerankSort.java	22.51% `-77.49%`	❌
SortExpression2.java	0%	❌

The class javadoc lists the type for each kind of sort, and the NOTE says in three points how this type will replace SortExpression.

Hazel-Datastax · 2026-11-12T00:47:53Z

+    return SortExpression.sort(path, parseAscending(path, innerValue));
+  }
+
+  public static boolean parseAscending(String path, JsonNode innerValue) {


Moved out of buildRegularSortExpression so FindAndRerankSortClauseDeserializer can reuse it.

Hazel-Datastax · 2026-11-12T00:48:04Z

  }

-  private void validateSortExpressionPath(String path) {
+  public static void validateSortExpressionPath(String path) {


Made public static so FindAndRerankSortClauseDeserializer can reuse it.

Hazel-Datastax · 2026-11-12T00:48:07Z

  protected abstract SortClause buildClauseFromDefinition(ObjectNode sortNode);

-  protected float[] tryDecodeBinaryVector(String path, JsonNode innerValue) {
+  public static float[] tryDecodeBinaryVector(String path, JsonNode innerValue) {


Made public static so FindAndRerankSortClauseDeserializer can reuse it.

Hazel-Datastax · 2026-11-12T00:49:11Z

      },
      not = ConflictingVectorSortSchema.class)
-  public static class HybridSortSchema {}
+  class HybridSortSchema {}


FindAndRerankSort is now an interface, so its member classes are already public static.

Hazel-Datastax · 2026-11-12T00:50:23Z

 *
- * <p>Note: This only checks the shape of the sort clause, the other validation is handled when
- * resolving the command to an operation.
+ * <p>NOTE1: some single leg checks are a copy of the checks in {@link CollectionSortClauseBuilder}.


I try to put the problem in the java doc, lmk if we want to remove

Hazel-Datastax · 2026-11-12T00:53:07Z

+   * Deserialize a sort without $hybrid: $lexical, $vector or $vectorize must be the only field in
+   * the sort, any other field is a field sort with the value <code>1</code> or <code>-1</code>.
+   */
+  private static FindAndRerankSort.SingleLeg deserializeSingleLeg(ObjectNode sort) {


Start from here. The $lexical, $vector and $vectorize checks below are a copy of the checks in CollectionSortClauseBuilder.

ok if we are keeping this as a direct copy of the code - rather than using the JsonFieldMatcher pattern the rest of the class uses we should call that out. Point to the class we are copying and say we are leaving the code the same so we can compare incase there are changes, and we will fix

Hazel-Datastax · 2026-11-12T00:54:18Z

- * <p>A MULTI_LEG command may still read with only one leg, e.g. <code>{"$hybrid": "cheese"}</code>
- * on a collection without a lexical index.
- */
-public enum RerankLegMode {


Removed because FindAndRerankSort is now a sealed interface: the type of the sort (MultiLeg or SingleLeg) replaces this enum.

I think this is an important "modern java" lesson to learn, a enum for a "type of thing" maybe be better expressed as a type structure

Hazel-Datastax · 2026-11-12T02:05:56Z

+      FindAndRerankCommand command,
+      FindAndRerankSort.MultiLeg multiLeg,
+      CollectionSchemaObject schemaObject) {


command already contains this sort, so it is passed twice. Options:

Keep as is: plan() already matched the MultiLeg, so this method does not need to cast it. The cost is one more parameter.

Pass only command, as on main: case MultiLeg ignored -> multiLegs(command, schemaObject), one less parameter, but this method only gets a FindAndRerankSort and needs a cast to use it as a MultiLeg.

Pass hybridLimits instead of command: this method only uses the sort and hybridLimits.

If we are passing things around more than necessary it may be that this should not be static, and we should create a Planner instance. Consider for the next work on this class.

Hazel-Datastax · 2026-11-12T18:12:28Z

+      sorts.add(
+          new FieldSortExpression(
+              field.getKey(),
+              CollectionSortClauseBuilder.parseAscending(field.getKey(), field.getValue())));


Flag for a decision: field sort values follow find and only accept 1 and -1. Any other value gets SORT_CLAUSE_VALUE_INVALID, so the design doc example {"age": 0} fails.

The design doc says "If 1 the sort is ascending, any other value is descending."

"If 1 the sort is ascending, any other value is descending." <- this sounds like something I would say/write as a design spec. And I am guessing the implementation better follows what Mongo did. OK to leave for now.

amorton

+1 this looks really really good, the new type structures makes the code so much cleaner. When we are finished this is going to look great.

amorton · 2026-11-13T20:37:18Z

    // Start by checking "special" sort expressions, like lexical and vector sorts;
    // they must be the only expression in the sort clause.

+    // NOTE: FindAndRerankSortClauseDeserializer has a copy of the checks below for $lexical,


amorton · 2026-11-13T20:41:59Z

+   * $vector.
+   */
+  record HybridObject(
+      @JsonIgnore String vectorize,


nit - may be nice to put a comment at the top about why all the properties of the sub interfaces are marked as JsonIgnore and if that should be done for any future change

amorton · 2026-11-13T20:43:22Z

+      return (obj instanceof HybridObject other)
+          && Objects.equals(vectorize, other.vectorize)
+          && Objects.equals(lexical, other.lexical)
+          && Arrays.equals(vector, other.vector)


nit: are we overriding equals() on the record because otherwise it does reference equality on the array ? if so a small comment if not why are we ?

amorton · 2026-11-13T20:46:21Z

+    @Override
+    public CommandFeatures commandFeatures() {
+      return switch (sort) {
+        case VectorizeSortExpression ignored -> CommandFeatures.of(CommandFeature.VECTORIZE);


Java 22 introduces the "_" to do just this

amorton · 2026-11-13T20:47:51Z

-        Arrays.hashCode(vectorSort),
-        commandFeatures);
-  }
+  class ConflictingVectorSortSchema {}


+1 for comments

amorton · 2026-11-13T20:55:12Z

+   * Deserialize a sort without $hybrid: $lexical, $vector or $vectorize must be the only field in
+   * the sort, any other field is a field sort with the value <code>1</code> or <code>-1</code>.
+   */
+  private static FindAndRerankSort.SingleLeg deserializeSingleLeg(ObjectNode sort) {


ok if we are keeping this as a direct copy of the code - rather than using the JsonFieldMatcher pattern the rest of the class uses we should call that out. Point to the class we are copying and say we are leaving the code the same so we can compare incase there are changes, and we will fix

amorton · 2026-11-13T20:56:49Z

+      sorts.add(
+          new FieldSortExpression(
+              field.getKey(),
+              CollectionSortClauseBuilder.parseAscending(field.getKey(), field.getValue())));


"If 1 the sort is ascending, any other value is descending." <- this sounds like something I would say/write as a design spec. And I am guessing the implementation better follows what Mongo did. OK to leave for now.

amorton · 2026-11-13T20:59:37Z

- * <p>A MULTI_LEG command may still read with only one leg, e.g. <code>{"$hybrid": "cheese"}</code>
- * on a collection without a lexical index.
- */
-public enum RerankLegMode {


I think this is an important "modern java" lesson to learn, a enum for a "type of thing" maybe be better expressed as a type structure

amorton · 2026-11-13T21:06:58Z

+      FindAndRerankCommand command,
+      FindAndRerankSort.MultiLeg multiLeg,
+      CollectionSchemaObject schemaObject) {


If we are passing things around more than necessary it may be that this should not be static, and we should create a Planner instance. Consider for the next work on this class.

amorton · 2026-11-13T21:09:44Z

  /** The text to vectorize: the $hybrid text, or the $vectorize value, or <code>null</code>. */
  private static String vectorizeText(FindAndRerankSort sort) {
-    return sort.hybridSort() != null ? sort.hybridSort() : sort.vectorizeSort();
+    return switch (sort) {


for the next round - these may be good candidates to be 'default' methods on FindAndRerankSort interface.

Hazel-Datastax added 3 commits November 11, 2026 12:47

update java doc

4878e0f

Hazel-Datastax added 3 commits November 11, 2026 14:00

read an empty findAndRerank sort as a single leg sort

c749997

The deserializer no longer has its own branch for an empty sort. The single leg code reads {} as the filter leg with no sorts, which is the same value as NO_ARG_SORT.

update java doc

bb00c05

Hazel-Datastax added 3 commits November 11, 2026 15:21

update java doc

7d592ce

shorten the SortExpression2 javadoc

8b3af76

The class javadoc lists the type for each kind of sort, and the NOTE says in three points how this type will replace SortExpression.

update name and java doc

daf539e

Hazel-Datastax commented Nov 12, 2026

View reviewed changes

Hazel-Datastax marked this pull request as ready for review November 12, 2026 02:06

Hazel-Datastax requested a review from a team as a code owner November 12, 2026 02:06

Hazel-Datastax commented Nov 12, 2026

View reviewed changes

Hazel-Datastax added this pull request to stack #2677 November 13, 2026 00:17

amorton approved these changes Nov 13, 2026

View reviewed changes

Hazel-Datastax added 3 commits November 13, 2026 15:53

add comments based on review

3b79e58

add comments based on review

b7dc723

add comments based on review

283af17

Hazel-Datastax merged commit c16c285 into main Nov 14, 2026
3 checks passed

Hazel-Datastax deleted the hazel/farr-sort-typed-legs branch November 14, 2026 00:34

Conversation

Hazel-Datastax commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 11, 2026

📈 Unit Test Coverage Delta vs Main Branch

Uh oh!

github-actions Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Coverage Report

Uh oh!

github-actions Bot commented Nov 11, 2026

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Uh oh!

github-actions Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration Test Coverage Report (dse69-it)

Uh oh!

github-actions Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📉 Integration Test Coverage Delta vs Main Branch (hcd-it)

Uh oh!

github-actions Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration Test Coverage Report (hcd-it)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amorton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Hazel-Datastax commented Nov 11, 2026 •

edited

Loading

github-actions Bot commented Nov 11, 2026 •

edited

Loading

github-actions Bot commented Nov 11, 2026 •

edited

Loading

github-actions Bot commented Nov 11, 2026 •

edited

Loading

github-actions Bot commented Nov 11, 2026 •

edited

Loading