Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
585e56a
docs(store): document the auto-sized RocksDB memory budget
byteayan Oct 3, 2026
a516891
Merge remote-tracking branch 'apache-doc/master' into doc/store-rocks…
byteayan Oct 16, 2026
120c614
docs(store): note the 1 MiB floor on the derived write_buffer_size
byteayan Oct 16, 2026
29bfda5
docs(store): say where the RocksDB budget comes from without a limit
byteayan Oct 16, 2026
1585f96
docs(store): count unset direct memory as the full heap
byteayan Oct 17, 2026
b9e2b34
docs(store): describe when the launcher adds the direct-memory cap
byteayan Oct 17, 2026
f299f36
Merge branch 'origin/master' into doc/store-rocksdb-memory-budget
imbajin Oct 17, 2026
0bb8278
docs(store): include -XX:MaxHeapSize when launcher skips direct-memor…
imbajin Oct 17, 2026
d11a103
docs(store): add a direct-memory cap to the production launcher example
byteayan Oct 17, 2026
849be33
Merge apache/master into doc/store-rocksdb-memory-budget
byteayan Oct 17, 2026
9e5b7ba
Merge remote-tracking branch 'origin/master' into doc/store-rocksdb-m…
imbajin Oct 18, 2026
a451d1d
docs(store): note the unknown-memory fallback and size the JVM for a …
byteayan Oct 19, 2026
1da9f42
Merge remote-tracking branch 'origin/master' into doc/store-rocksdb-m…
imbajin Oct 19, 2026
781384b
docs(store): polish memory budget phrasing and wrap long lines
imbajin Oct 19, 2026
638a816
docs(store): clarify the memory budget's preconditions and limits
byteayan Oct 20, 2026
9ad959d
Merge remote-tracking branch 'apache-doc/master' into doc/store-rocks…
byteayan Oct 20, 2026
4b6ceae
docs(store): mark the write_buffer_size code default as derived
byteayan Oct 20, 2026
29ae923
docs(store): cover the application-pd.yml copy that PD keeps
byteayan Oct 21, 2026
df36c54
docs(store): note that Raft log memtables are charged to the log cache
byteayan Oct 22, 2026
e964a68
docs(store): say how a PD-held budget actually gets applied
byteayan Oct 23, 2026
64448e8
ci: re-run the site build
byteayan Oct 23, 2026
1373759
docs(store): document auto for rocksdb.total_memory_size
byteayan Oct 26, 2026
221df8e
docs(store): keep S:FS readable by earlier Stores and note the restar…
byteayan Oct 27, 2026
eab9804
docs(store): say a Docker Store stops on S:Timestamp instead of resta…
byteayan Oct 27, 2026
5c27615
docs(store): say a restarted Docker Store keeps JAVA_OPTS and a plain…
byteayan Oct 28, 2026
0cb4850
docs(store): note that a JAVA_TOOL_OPTIONS or JDK_JAVA_OPTIONS cap is…
byteayan Oct 28, 2026
a70b481
Merge master into doc/store-rocksdb-memory-budget
byteayan Oct 28, 2026
b5d55e1
docs(store): check the memory rows against the commit that merged hug…
byteayan Oct 29, 2026
3e5589d
Merge branch 'master' into doc/store-rocksdb-memory-budget
imbajin Oct 29, 2026
2d4de64
docs: streamline rocksdb memory budget and align cn/en docs
imbajin Oct 29, 2026
06e75d3
docs(store): give the auto-sizing ceiling as 32 GB and say what allow…
byteayan Oct 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions content/cn/docs/guides/hugegraph-docker-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,10 +196,15 @@ curl -o /dev/null -w '%{http_code}\n' -u "admin:${HUGEGRAPH_ADMIN_PASSWORD}" \
| `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` |
| `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` |
| `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` |
| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size`,`auto` 或字节数 |
{.td-table--wide}

> **已弃用的别名**:`PD_ADDRESS` → `HG_STORE_PD_ADDRESS`、`GRPC_HOST` → `HG_STORE_GRPC_HOST`、`RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS`

RocksDB 缓存使用堆外内存。未设置 `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` 时,系统将优先根据容器内存限制(cgroup)自动计算;未配置容器限制时则按宿主机内存推导。
由于默认 Docker Compose 未限制容器内存,各 Store 均会以宿主机整机容量估算;若单机部署多个 Store,建议显式配置容器内存限制,或直接为该变量指定字节数。
此外,Store 启动脚本默认会根据系统空闲内存分配 JVM 堆与直接内存(上限各 2 GiB)。建议结合容器规格,通过 `JAVA_OPTS` 显式约束 JVM 内存(例如容器为 4 GiB 时配置 `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"`)。详细推导公式见 [HStore 配置参考](/cn/docs/quickstart/hugegraph/hugegraph-hstore/#rocksdb-memory-budget)。

### Server 变量

与 PD、Store 不同,Server 入口脚本没有必填变量:只有实际设置了的变量才会被写入配置文件。但分布式部署至少需要 `HG_SERVER_BACKEND` 和 `HG_SERVER_PD_PEERS`。
Expand Down
22 changes: 17 additions & 5 deletions content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ docker run -d \
| `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` | gRPC 服务端口。 |
| `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` | REST API 端口。 |
| `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` | 数据存储路径。 |
| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size` | RocksDB 内存预算:`auto` 或字节数。空值视为未设置;其他值会使入口脚本以退出码 2 退出。未设置时使用 `application-pd.yml` 中的值或自动计算。 |
{.td-table--wide}

当前主线 entrypoint 将这些变量写入 `SPRING_APPLICATION_JSON`,覆盖镜像内 `conf/application.yml`;其余配置仍从该文件和 `application-pd.yml` 读取,然后以前台方式启动 Java。旧变量 `PD_ADDRESS`、`GRPC_HOST`、`RAFT_ADDRESS` 仍可用,但会输出弃用警告。
Expand All @@ -151,7 +152,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [

#### 4.1 application.yml

以下配置参考表核对的是 server 主线提交 `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4` 的发行目录值和 Java 默认注入值;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。
以下配置参考表核对的是 Server 主线提交的发行目录值和 Java 默认注入值;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。

#### 4.2 application-pd.yml

Expand Down Expand Up @@ -183,7 +184,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [
| 配置项 | 当前主线目录值 | 代码默认值 | 说明 |
|--------|--------|------------|------|
| `raft.address` | `127.0.0.1:8510` | 必填 | 本节点的 Raft 服务地址,格式为 `host:port`,必须能被其他 Store 节点访问。这里不需要配置 peer 列表:分区 Raft 组的成员由 PD 下发。 |
| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数取最接近的 2 的幂,再乘以 32。 |
| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数(至少为 1)取最接近的 2 的幂,再乘以 32。 |
| `raft.max-log-file-size` | `600000000000` | `50000000000` | Raft 日志的最大字节数。 |
| `raft.snapshotInterval` | `1800` | `300` | 生成 Raft 快照的时间间隔,单位秒。 |
| `raft.snapshotLogIndexMargin` | 未设置 | `0` | 距上次快照的最小 applied index 差值,达到后才真正生成快照。设为 `0` 关闭该判断。 |
Expand Down Expand Up @@ -212,12 +213,20 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [

| 配置项 | 当前主线目录值 | 代码默认值 | 说明 |
|--------|--------|------------|------|
| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | 本节点所有 RocksDB 实例共享的内存预算。缺失或为 `0` 时使用 JVM 最大堆内存。 |
| `rocksdb.write_buffer_size` | `32000000` | `33554432` | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`。 |
| `rocksdb.total_memory_size` | `auto` | 自动计算,见说明 | 本节点所有 RocksDB 实例共享的堆外内存预算(取值为 `auto` 或具体字节数)。详细自动推导规则见下方说明。 |
| `rocksdb.write_buffer_size` | `32000000` | 自动推导,见说明 | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`,最小为 `1048576`(1 MiB)。 |
| `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | 落盘前合并的 memtable 数量。 |
| `rocksdb.write_buffer_ratio` | 未设置 | `0.66` | `total_memory_size` 中分配给写缓存的比例,其余作为 block cache。 |
{.td-table--wide}

### RocksDB 内存预算自动计算规则 {#rocksdb-memory-budget}

> [!NOTE]
> 当 `rocksdb.total_memory_size` 设置为 `auto`(或未设置、为 `0`)时,系统自动计算堆外 RocksDB 缓存预算:
> - **推导公式**:以物理内存(容器环境下优先取可见的 cgroup 限制)减去 JVM 最大堆(`-Xmx`)、直接内存(`MaxDirectMemorySize`)和 1.5 GiB 系统预留后,取剩余量的一半,范围钳制在 256 MiB 至 32 GB(`32000000000` 字节)之间。
> - **写缓存与 Block Cache**:根据 `rocksdb.write_buffer_ratio`(默认 0.66)切分。该预算为软目标容量;大量并发写入时可将 `rocksdb.write_buffer_allow_stall` 设为 `true`,以增加写入延迟为代价把写缓存限制在其份额内。Block cache 并非严格上限,本地内存仍可能超出预算,容器仍可能被 OOM 终止。
> - **升级与覆盖**:历史配置若保留了旧固定值 `32000000000`,可通过设置环境变量 `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` 或启动参数 `-Drocksdb.total_memory_size=auto` 启用自动推导。

`org/apache/hugegraph/rocksdb/access/RocksDBOptions.java` 中定义的其他选项都可以加在同一个 `rocksdb:` 块下,例如 `rocksdb.max_background_jobs`、`rocksdb.level0_file_num_compaction_trigger` 或 `rocksdb.bloom_filter_bits_per_key`。

**线程池**
Expand Down Expand Up @@ -330,7 +339,10 @@ master 构建接受 `g1`、`G1`、`zgc` 和 `ZGC`。
- `-d true`(默认):以后台守护进程方式运行,脚本立即返回,并把 Java 进程号写入 `bin/pid`。
- `-d false`:以前台模式运行,脚本通过 `exec` 替换为 Java 进程,容器或进程管理器的进程即为 Java 本身。在 Docker 或进程管理器(systemd、supervisord)下运行时请使用此参数,以便在崩溃时自动检测并重启服务。

JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。脚本还会加上 `-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。生产节点通常需要大得多的堆,请显式指定,例如 `-j "-Xmx32g -Xms32g"`。
JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。
脚本还会加上与该堆大小相同的 `-XX:MaxDirectMemorySize`、`-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。
生产节点通常需要大得多的堆,请显式指定,并同时设置直接内存上限,例如 `-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`。
不设上限时,JVM 允许的直接内存与 32 GB 的堆相同,RocksDB 预算会把它计算在内,在内存不足约 66 GB 的主机上会降到 256 MiB 的下限。

当 `ulimit -n` 或 `ulimit -u` 低于 1024 时脚本会拒绝启动;在 x86_64 和 arm64 上,如果能成功下载并校验对应的动态库,脚本会预加载 jemalloc。

Expand Down
6 changes: 6 additions & 0 deletions content/en/docs/guides/hugegraph-docker-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,10 +196,16 @@ The following tables describe current master entrypoints and Compose files, not
| `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` |
| `HG_STORE_REST_PORT` | No | `8520` | `server.port` |
| `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` |
| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size`, `auto` or bytes |
{.td-table--wide}

> **Deprecated aliases**: `PD_ADDRESS` → `HG_STORE_PD_ADDRESS`, `GRPC_HOST` → `HG_STORE_GRPC_HOST`, `RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS`.

RocksDB caches use off-heap memory. When `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` is unset, the system sizes the cache automatically from the container memory limit (cgroup), or falls back to the host memory if no container limit is configured.
Because the default Docker Compose file does not set a memory limit, each Store sizes its budget using the host's full capacity. When deploying multiple Stores on a single host, specify a container memory limit or set this variable directly to an explicit byte value.
Additionally, the Store startup script sizes the JVM heap and direct memory based on host free memory by default (up to 2 GiB each). Constrain JVM memory explicitly via `JAVA_OPTS` to match your container budget (for example, `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"` for a 4 GiB container). See the [HStore configuration reference](/docs/quickstart/hugegraph/hugegraph-hstore/#rocksdb-memory-budget) for the detailed calculation rules.


### Server Variables

Unlike PD and Store, Server has no mandatory entrypoint variables: only set values are written to configuration. Distributed deployment nevertheless requires at least `HG_SERVER_BACKEND` and `HG_SERVER_PD_PEERS`.
Expand Down
23 changes: 18 additions & 5 deletions content/en/docs/quickstart/hugegraph/hugegraph-hstore.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ docker run -d \
| `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` | gRPC service port. |
| `HG_STORE_REST_PORT` | No | `8520` | `server.port` | REST API port. |
| `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` | Data path. |
| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size` | RocksDB memory budget: `auto` or a number of bytes. An empty value counts as unset; any other value makes the entrypoint exit with code 2. When unset, the value in `application-pd.yml` or auto-sizing applies. |
{.td-table--wide}

The master entrypoint writes these values into `SPRING_APPLICATION_JSON`, overriding image `conf/application.yml`. Other options still come from that file and `application-pd.yml`; Java runs in the foreground. Deprecated `PD_ADDRESS`, `GRPC_HOST`, and `RAFT_ADDRESS` remain supported with warnings.
Expand All @@ -152,7 +153,7 @@ Store reads two files under `conf/`. Release 1.7.0 sources: [application.yml](ht

#### 4.1 application.yml

The following tables were checked against distribution values and Java defaults at Server master commit `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4`. Use tagged configuration for release 1.7.0 without replacing it with another version's files.
The following tables were checked against distribution values and Java defaults at Server master. Use tagged configuration for release 1.7.0 without replacing it with another version's files.

#### 4.2 application-pd.yml

Expand Down Expand Up @@ -185,7 +186,7 @@ This file contains only `rocksdb` and Actuator exposure settings.
| Key | Master distribution value | Code default | Meaning |
|-----|---------|--------------|---------|
| `raft.address` | `127.0.0.1:8510` | required | Raft service address of this node, `host:port`. Must be reachable from every other Store node. There is no peer list to configure: PD tells each node which peers belong to a partition's Raft group. |
| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB to the nearest power of two and multiplying by 32. |
| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB (at least 1) to the nearest power of two and multiplying by 32. |
| `raft.max-log-file-size` | `600000000000` | `50000000000` | Maximum byte size of Raft logs. |
| `raft.snapshotInterval` | `1800` | `300` | Seconds between Raft snapshots. |
| `raft.snapshotLogIndexMargin` | not set | `0` | Minimum applied-index distance since the last snapshot before a snapshot is actually written. `0` disables the distance check. |
Expand Down Expand Up @@ -214,12 +215,20 @@ This file contains only `rocksdb` and Actuator exposure settings.

| Key | Master distribution value | Code default | Meaning |
|-----|---------|--------------|---------|
| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | Memory budget shared by all RocksDB instances on this node. When absent or `0`, the node uses the JVM max heap instead. |
| `rocksdb.write_buffer_size` | `32000000` | `33554432` | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`. |
| `rocksdb.total_memory_size` | `auto` | Auto-sized, see note | Off-heap memory budget shared across all RocksDB instances on this node (`auto` or byte count). See below for derivation rules. |
| `rocksdb.write_buffer_size` | `32000000` | Derived, see Meaning | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`, at least `1048576` (1 MiB). |
| `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | Number of memtables merged together before a flush. |
| `rocksdb.write_buffer_ratio` | not set | `0.66` | Share of `total_memory_size` given to the write cache. The rest becomes the block cache. |
{.td-table--wide}

### RocksDB memory budget sizing rules {#rocksdb-memory-budget}

> [!NOTE]
> When `rocksdb.total_memory_size` is set to `auto` (or absent / `0`), the node sizes the off-heap RocksDB budget automatically:
> - **Calculation Formula**: Takes physical memory (using the visible cgroup limit in containers) minus JVM maximum heap (`-Xmx`), direct memory (`MaxDirectMemorySize`), and a 1.5 GiB system reserve, then allocates half the remainder, clamped between 256 MiB and 32 GB (`32000000000` bytes).
> - **Write Cache & Block Cache**: Budget is divided according to `rocksdb.write_buffer_ratio` (default 0.66). This is a soft capacity target; under heavy concurrent write loads, set `rocksdb.write_buffer_allow_stall` to `true` to hold the write cache to its share at the cost of write latency. The block cache is not a strict limit, so native memory can still exceed the budget and a container can still be OOM-killed.
> - **Upgrading Existing Deployments**: Deployments carrying legacy `32000000000` can enable auto-sizing by setting environment variable `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` or JVM property `-Drocksdb.total_memory_size=auto`.

Any other option defined in `org/apache/hugegraph/rocksdb/access/RocksDBOptions.java` can be added under the same `rocksdb:` block, for example `rocksdb.max_background_jobs`, `rocksdb.level0_file_num_compaction_trigger` or `rocksdb.bloom_filter_bits_per_key`.

**Thread pools**
Expand Down Expand Up @@ -335,7 +344,11 @@ Daemon mode:
- `-d true` (default): run as a background daemon. The script returns immediately and writes the Java pid to `bin/pid`.
- `-d false`: run in the foreground. The script `exec`s Java, so the container or supervisor process is Java itself. Use this under Docker or a process supervisor (systemd, supervisord) so crashes are detected and the service is restarted automatically.

JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range. The script also adds `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at `logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly, for example `-j "-Xmx32g -Xms32g"`.
JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range.
The script also adds `-XX:MaxDirectMemorySize` equal to that heap, `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at
`logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly together with a direct-memory cap, for example
`-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`. Without the cap the JVM allows as much direct memory as the 32 GB heap, the RocksDB budget counts it,
and on a host with less than about 66 GB it falls to the 256 MiB floor.

The script refuses to start if `ulimit -n` or `ulimit -u` is below 1024, and it preloads jemalloc on x86_64 and arm64 when the shared object can be downloaded and verified.

Expand Down
Loading