diff --git a/content/cn/docs/guides/hugegraph-docker-cluster.md b/content/cn/docs/guides/hugegraph-docker-cluster.md index 55338f250e..cb986bb7f2 100644 --- a/content/cn/docs/guides/hugegraph-docker-cluster.md +++ b/content/cn/docs/guides/hugegraph-docker-cluster.md @@ -196,10 +196,15 @@ curl -o /dev/null -w '%{http_code}\n' -u "admin:${HUGEGRAPH_ADMIN_PASSWORD}" \ | `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` | | `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` | | `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size`,`auto` 或字节数 | {.td-table--wide} > **已弃用的别名**:`PD_ADDRESS` → `HG_STORE_PD_ADDRESS`、`GRPC_HOST` → `HG_STORE_GRPC_HOST`、`RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS` +RocksDB 缓存使用堆外内存。未设置 `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` 时,系统将优先根据容器内存限制(cgroup)自动计算;未配置容器限制时则按宿主机内存推导。 +由于默认 Docker Compose 未限制容器内存,各 Store 均会以宿主机整机容量估算;若单机部署多个 Store,建议显式配置容器内存限制,或直接为该变量指定字节数。 +此外,Store 启动脚本默认会根据系统空闲内存分配 JVM 堆与直接内存(上限各 2 GiB)。建议结合容器规格,通过 `JAVA_OPTS` 显式约束 JVM 内存(例如容器为 4 GiB 时配置 `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"`)。详细推导公式见 [HStore 配置参考](/cn/docs/quickstart/hugegraph/hugegraph-hstore/#rocksdb-memory-budget)。 + ### Server 变量 与 PD、Store 不同,Server 入口脚本没有必填变量:只有实际设置了的变量才会被写入配置文件。但分布式部署至少需要 `HG_SERVER_BACKEND` 和 `HG_SERVER_PD_PEERS`。 diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md b/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md index 919a3b1af9..bb0c9e9496 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-hstore.md @@ -136,6 +136,7 @@ docker run -d \ | `HG_STORE_GRPC_PORT` | 否 | `8500` | `grpc.port` | gRPC 服务端口。 | | `HG_STORE_REST_PORT` | 否 | `8520` | `server.port` | REST API 端口。 | | `HG_STORE_DATA_PATH` | 否 | `/hugegraph-store/storage` | `app.data-path` | 数据存储路径。 | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | 否 | 无 | `rocksdb.total_memory_size` | RocksDB 内存预算:`auto` 或字节数。空值视为未设置;其他值会使入口脚本以退出码 2 退出。未设置时使用 `application-pd.yml` 中的值或自动计算。 | {.td-table--wide} 当前主线 entrypoint 将这些变量写入 `SPRING_APPLICATION_JSON`,覆盖镜像内 `conf/application.yml`;其余配置仍从该文件和 `application-pd.yml` 读取,然后以前台方式启动 Java。旧变量 `PD_ADDRESS`、`GRPC_HOST`、`RAFT_ADDRESS` 仍可用,但会输出弃用警告。 @@ -151,7 +152,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ #### 4.1 application.yml -以下配置参考表核对的是 server 主线提交 `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4` 的发行目录值和 Java 默认注入值;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。 +以下配置参考表核对的是 Server 主线提交的发行目录值和 Java 默认注入值;1.7.0 发布包请使用其版本标签对应的配置文件,不要跨版本覆盖。 #### 4.2 application-pd.yml @@ -183,7 +184,7 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ | 配置项 | 当前主线目录值 | 代码默认值 | 说明 | |--------|--------|------------|------| | `raft.address` | `127.0.0.1:8510` | 必填 | 本节点的 Raft 服务地址,格式为 `host:port`,必须能被其他 Store 节点访问。这里不需要配置 peer 列表:分区 Raft 组的成员由 PD 下发。 | -| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数取最接近的 2 的幂,再乘以 32。 | +| `raft.disruptorBufferSize` | `1024` | `0` | Raft 任务队列大小。设为 `0` 时按 `rocksdb.total_memory_size` 推导:将该内存量的 GB 数(至少为 1)取最接近的 2 的幂,再乘以 32。 | | `raft.max-log-file-size` | `600000000000` | `50000000000` | Raft 日志的最大字节数。 | | `raft.snapshotInterval` | `1800` | `300` | 生成 Raft 快照的时间间隔,单位秒。 | | `raft.snapshotLogIndexMargin` | 未设置 | `0` | 距上次快照的最小 applied index 差值,达到后才真正生成快照。设为 `0` 关闭该判断。 | @@ -212,12 +213,20 @@ Store 从 `conf/` 读取两个配置文件。1.7.0 发布标签的源文件见 [ | 配置项 | 当前主线目录值 | 代码默认值 | 说明 | |--------|--------|------------|------| -| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | 本节点所有 RocksDB 实例共享的内存预算。缺失或为 `0` 时使用 JVM 最大堆内存。 | -| `rocksdb.write_buffer_size` | `32000000` | `33554432` | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`。 | +| `rocksdb.total_memory_size` | `auto` | 自动计算,见说明 | 本节点所有 RocksDB 实例共享的堆外内存预算(取值为 `auto` 或具体字节数)。详细自动推导规则见下方说明。 | +| `rocksdb.write_buffer_size` | `32000000` | 自动推导,见说明 | memtable 大小,单位字节。缺失或为 `0` 时取 `total_memory_size / 1000`,最小为 `1048576`(1 MiB)。 | | `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | 落盘前合并的 memtable 数量。 | | `rocksdb.write_buffer_ratio` | 未设置 | `0.66` | `total_memory_size` 中分配给写缓存的比例,其余作为 block cache。 | {.td-table--wide} +### RocksDB 内存预算自动计算规则 {#rocksdb-memory-budget} + +> [!NOTE] +> 当 `rocksdb.total_memory_size` 设置为 `auto`(或未设置、为 `0`)时,系统自动计算堆外 RocksDB 缓存预算: +> - **推导公式**:以物理内存(容器环境下优先取可见的 cgroup 限制)减去 JVM 最大堆(`-Xmx`)、直接内存(`MaxDirectMemorySize`)和 1.5 GiB 系统预留后,取剩余量的一半,范围钳制在 256 MiB 至 32 GB(`32000000000` 字节)之间。 +> - **写缓存与 Block Cache**:根据 `rocksdb.write_buffer_ratio`(默认 0.66)切分。该预算为软目标容量;大量并发写入时可将 `rocksdb.write_buffer_allow_stall` 设为 `true`,以增加写入延迟为代价把写缓存限制在其份额内。Block cache 并非严格上限,本地内存仍可能超出预算,容器仍可能被 OOM 终止。 +> - **升级与覆盖**:历史配置若保留了旧固定值 `32000000000`,可通过设置环境变量 `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` 或启动参数 `-Drocksdb.total_memory_size=auto` 启用自动推导。 + `org/apache/hugegraph/rocksdb/access/RocksDBOptions.java` 中定义的其他选项都可以加在同一个 `rocksdb:` 块下,例如 `rocksdb.max_background_jobs`、`rocksdb.level0_file_num_compaction_trigger` 或 `rocksdb.bloom_filter_bits_per_key`。 **线程池** @@ -330,7 +339,10 @@ master 构建接受 `g1`、`G1`、`zgc` 和 `ZGC`。 - `-d true`(默认):以后台守护进程方式运行,脚本立即返回,并把 Java 进程号写入 `bin/pid`。 - `-d false`:以前台模式运行,脚本通过 `exec` 替换为 Java 进程,容器或进程管理器的进程即为 Java 本身。在 Docker 或进程管理器(systemd、supervisord)下运行时请使用此参数,以便在崩溃时自动检测并重启服务。 -JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。脚本还会加上 `-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。生产节点通常需要大得多的堆,请显式指定,例如 `-j "-Xmx32g -Xms32g"`。 +JVM 内存方面,如果没有自行设置 `JAVA_OPTIONS`:`-Xms512m`,`-Xmx` 取空闲内存的一半并限制在 512 MB 到 2048 MB 之间。 +脚本还会加上与该堆大小相同的 `-XX:MaxDirectMemorySize`、`-XX:MetaspaceSize=256M`、OOM 时把堆转储写入 `logs/`,以及滚动的 GC 日志 `logs/gc.log`。 +生产节点通常需要大得多的堆,请显式指定,并同时设置直接内存上限,例如 `-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`。 +不设上限时,JVM 允许的直接内存与 32 GB 的堆相同,RocksDB 预算会把它计算在内,在内存不足约 66 GB 的主机上会降到 256 MiB 的下限。 当 `ulimit -n` 或 `ulimit -u` 低于 1024 时脚本会拒绝启动;在 x86_64 和 arm64 上,如果能成功下载并校验对应的动态库,脚本会预加载 jemalloc。 diff --git a/content/en/docs/guides/hugegraph-docker-cluster.md b/content/en/docs/guides/hugegraph-docker-cluster.md index 590b57c119..b17f7e7236 100644 --- a/content/en/docs/guides/hugegraph-docker-cluster.md +++ b/content/en/docs/guides/hugegraph-docker-cluster.md @@ -196,10 +196,16 @@ The following tables describe current master entrypoints and Compose files, not | `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` | | `HG_STORE_REST_PORT` | No | `8520` | `server.port` | | `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size`, `auto` or bytes | {.td-table--wide} > **Deprecated aliases**: `PD_ADDRESS` → `HG_STORE_PD_ADDRESS`, `GRPC_HOST` → `HG_STORE_GRPC_HOST`, `RAFT_ADDRESS` → `HG_STORE_RAFT_ADDRESS`. +RocksDB caches use off-heap memory. When `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` is unset, the system sizes the cache automatically from the container memory limit (cgroup), or falls back to the host memory if no container limit is configured. +Because the default Docker Compose file does not set a memory limit, each Store sizes its budget using the host's full capacity. When deploying multiple Stores on a single host, specify a container memory limit or set this variable directly to an explicit byte value. +Additionally, the Store startup script sizes the JVM heap and direct memory based on host free memory by default (up to 2 GiB each). Constrain JVM memory explicitly via `JAVA_OPTS` to match your container budget (for example, `JAVA_OPTS="-Xmx1g -XX:MaxDirectMemorySize=512m"` for a 4 GiB container). See the [HStore configuration reference](/docs/quickstart/hugegraph/hugegraph-hstore/#rocksdb-memory-budget) for the detailed calculation rules. + + ### Server Variables Unlike PD and Store, Server has no mandatory entrypoint variables: only set values are written to configuration. Distributed deployment nevertheless requires at least `HG_SERVER_BACKEND` and `HG_SERVER_PD_PEERS`. diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md b/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md index 5d39d077ee..14a3bab6f7 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-hstore.md @@ -137,6 +137,7 @@ docker run -d \ | `HG_STORE_GRPC_PORT` | No | `8500` | `grpc.port` | gRPC service port. | | `HG_STORE_REST_PORT` | No | `8520` | `server.port` | REST API port. | | `HG_STORE_DATA_PATH` | No | `/hugegraph-store/storage` | `app.data-path` | Data path. | +| `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE` | No | None | `rocksdb.total_memory_size` | RocksDB memory budget: `auto` or a number of bytes. An empty value counts as unset; any other value makes the entrypoint exit with code 2. When unset, the value in `application-pd.yml` or auto-sizing applies. | {.td-table--wide} The master entrypoint writes these values into `SPRING_APPLICATION_JSON`, overriding image `conf/application.yml`. Other options still come from that file and `application-pd.yml`; Java runs in the foreground. Deprecated `PD_ADDRESS`, `GRPC_HOST`, and `RAFT_ADDRESS` remain supported with warnings. @@ -152,7 +153,7 @@ Store reads two files under `conf/`. Release 1.7.0 sources: [application.yml](ht #### 4.1 application.yml -The following tables were checked against distribution values and Java defaults at Server master commit `2f827d6e8c9c62ae858f2fc122b3a192d015e2f4`. Use tagged configuration for release 1.7.0 without replacing it with another version's files. +The following tables were checked against distribution values and Java defaults at Server master. Use tagged configuration for release 1.7.0 without replacing it with another version's files. #### 4.2 application-pd.yml @@ -185,7 +186,7 @@ This file contains only `rocksdb` and Actuator exposure settings. | Key | Master distribution value | Code default | Meaning | |-----|---------|--------------|---------| | `raft.address` | `127.0.0.1:8510` | required | Raft service address of this node, `host:port`. Must be reachable from every other Store node. There is no peer list to configure: PD tells each node which peers belong to a partition's Raft group. | -| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB to the nearest power of two and multiplying by 32. | +| `raft.disruptorBufferSize` | `1024` | `0` | Raft task queue size. `0` derives it from `rocksdb.total_memory_size`, by rounding that size in GB (at least 1) to the nearest power of two and multiplying by 32. | | `raft.max-log-file-size` | `600000000000` | `50000000000` | Maximum byte size of Raft logs. | | `raft.snapshotInterval` | `1800` | `300` | Seconds between Raft snapshots. | | `raft.snapshotLogIndexMargin` | not set | `0` | Minimum applied-index distance since the last snapshot before a snapshot is actually written. `0` disables the distance check. | @@ -214,12 +215,20 @@ This file contains only `rocksdb` and Actuator exposure settings. | Key | Master distribution value | Code default | Meaning | |-----|---------|--------------|---------| -| `rocksdb.total_memory_size` | `32000000000` | `51539607552` | Memory budget shared by all RocksDB instances on this node. When absent or `0`, the node uses the JVM max heap instead. | -| `rocksdb.write_buffer_size` | `32000000` | `33554432` | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`. | +| `rocksdb.total_memory_size` | `auto` | Auto-sized, see note | Off-heap memory budget shared across all RocksDB instances on this node (`auto` or byte count). See below for derivation rules. | +| `rocksdb.write_buffer_size` | `32000000` | Derived, see Meaning | Memtable size in bytes. When absent or `0`, the node uses `total_memory_size / 1000`, at least `1048576` (1 MiB). | | `rocksdb.min_write_buffer_number_to_merge` | `16` | `16` | Number of memtables merged together before a flush. | | `rocksdb.write_buffer_ratio` | not set | `0.66` | Share of `total_memory_size` given to the write cache. The rest becomes the block cache. | {.td-table--wide} +### RocksDB memory budget sizing rules {#rocksdb-memory-budget} + +> [!NOTE] +> When `rocksdb.total_memory_size` is set to `auto` (or absent / `0`), the node sizes the off-heap RocksDB budget automatically: +> - **Calculation Formula**: Takes physical memory (using the visible cgroup limit in containers) minus JVM maximum heap (`-Xmx`), direct memory (`MaxDirectMemorySize`), and a 1.5 GiB system reserve, then allocates half the remainder, clamped between 256 MiB and 32 GB (`32000000000` bytes). +> - **Write Cache & Block Cache**: Budget is divided according to `rocksdb.write_buffer_ratio` (default 0.66). This is a soft capacity target; under heavy concurrent write loads, set `rocksdb.write_buffer_allow_stall` to `true` to hold the write cache to its share at the cost of write latency. The block cache is not a strict limit, so native memory can still exceed the budget and a container can still be OOM-killed. +> - **Upgrading Existing Deployments**: Deployments carrying legacy `32000000000` can enable auto-sizing by setting environment variable `HG_STORE_ROCKSDB_TOTAL_MEMORY_SIZE=auto` or JVM property `-Drocksdb.total_memory_size=auto`. + Any other option defined in `org/apache/hugegraph/rocksdb/access/RocksDBOptions.java` can be added under the same `rocksdb:` block, for example `rocksdb.max_background_jobs`, `rocksdb.level0_file_num_compaction_trigger` or `rocksdb.bloom_filter_bits_per_key`. **Thread pools** @@ -335,7 +344,11 @@ Daemon mode: - `-d true` (default): run as a background daemon. The script returns immediately and writes the Java pid to `bin/pid`. - `-d false`: run in the foreground. The script `exec`s Java, so the container or supervisor process is Java itself. Use this under Docker or a process supervisor (systemd, supervisord) so crashes are detected and the service is restarted automatically. -JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range. The script also adds `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at `logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly, for example `-j "-Xmx32g -Xms32g"`. +JVM memory, unless you set `JAVA_OPTIONS` yourself: `-Xms512m`, and `-Xmx` set to half the free memory, clamped to the 512 MB to 2048 MB range. +The script also adds `-XX:MaxDirectMemorySize` equal to that heap, `-XX:MetaspaceSize=256M`, a heap dump on out-of-memory into `logs/`, and a rolling GC log at +`logs/gc.log`. Production nodes normally need a much larger heap, so pass one explicitly together with a direct-memory cap, for example +`-j "-Xmx32g -Xms32g -XX:MaxDirectMemorySize=2g"`. Without the cap the JVM allows as much direct memory as the 32 GB heap, the RocksDB budget counts it, +and on a host with less than about 66 GB it falls to the 256 MiB floor. The script refuses to start if `ulimit -n` or `ulimit -u` is below 1024, and it preloads jemalloc on x86_64 and arm64 when the shared object can be downloaded and verified.