[mbien]

changes default from full index to two years to maintain somewhat constant index sizes.

topic came up on ASF slack with @neilcsmith-net

[matthiasblaesing]

I think I raised this question before: What happens to projects not releasing new code (yes there are projects that are in their final state)? Will they fall out of the index?

[mbien]

yes they would be dropped, its purely date based. But since local and remote index are merged it is usually not a problem.

[matthiasblaesing]

I don't think this is great, but I have no intention to block this.

[mbien]

I didn't feel this is a big change since this is only changing the default value. I have also no issues with using the dev list, but given that the maven 3.10 thread doesn't even have a single reply I haven't even considered posting this on the list tbh.

[neilcsmith-net]

@matthiasblaesing to answer the bit of your comment edited away, then large architectural changes spanning multiple PRs should absolutely be discussed on dev@, as should major things discussed on Slack, or any other Slack or PR issue anyone wants moved there. Or the old "If it didn't happen on the mailing list, it didn't happen".

This specific thing came up in the context of discussing the indexing while testing #9642 and Maven 3.10. Which has a thread already as @mbien says.

Testing that PR as requested (on Slack), indexing worked, although it felt somewhat slower to me. Testing central indexing is also becoming harder due to being blocked with HTTP 429 very easily.

Personally, I think we should reconsider the default here, but I wouldn't land it this week before branching. If Maven 3.10 goes in and we get more evidence of indexing issues, then we should consider this minor change for delivery.

[mbien]

I remember even when I implemented the filter I thought that a sliding window through time would be the only realistic mid term solution as default. The only long term solution would be a time based segmentation on maven central's side or giving up on the concept of getting the remote index entirely at some point. (both has the factor of time in it)

change was merged 2023 (#5922), look at the stats: https://mvnrepository.com/repos/central
it contains a million ~11.8 million packages more by now probably due to robots taking over releases.

Btw I use the full setting locally, but early on we also got often feedback by users regarding index size. I think there is a thread somewhere claiming that NB is downloading maven central or something like that.

impl note: the filter is only really doing something during extraction, so if you have it at 2 years and wait a year your index contains 3 years. So its not an actual sliding window, just the initial cutoff.

[ebarboni]

no more for NB32 I suppose

[neilcsmith-net]

We do need a discussion about defaults given that the index has basically doubled in size since the filters were first introduced. But let's not do that this week!

[mbien]

i set the alarm to one day before next freeze for changing defaults

[mbien]

did some digging with luke the index walker and my index has currently 115.009.414 docs and is 7.4 GB large in full mode.

This looks big, but its almost exactly 64 bytes per document. Its actually surprisingly compact all things considered.

Having less rows has still some benefits: upkeep is easier since merge and group list computing becomes faster, fewer items are returned for "get latest" queries. While looking through this I remembered having to bump limits a few times otherwise it wouldn't compute "latest" correctly anymore (#8198).