diff --git a/deploy/README.md b/deploy/README.md new file mode 100644 index 0000000..7b95baa --- /dev/null +++ b/deploy/README.md @@ -0,0 +1,245 @@ +# HealthyCode Deployment documentation + +This is the installation documentation for HealthyCode. + +It explains how to deploy HealthyCode four components: +- PurlDB frontend +- ScanCode.io worker with its health pipeline and HealthyCode +- GrimoireLab backend + +The sequence is to install GrimoireLab, then ScanCode.io and then PurlDB. +The installation assume that we have a Linux installation (Ubuntu or Debian) +with these packages: + - git + - docker and docker compose + - build essentials + +The machine should have at least 32GB RAM (more is better), and about 500GB +for the base installation. For on-going production, more disk space is required +in order to keep git clones of all the tracked open source projects. + + +## GrimoireLab installation + +This is deployed using the docker compose at: +docker-compose/docker-compose-grimoirelab2.yml + + +### Set the max_map_count on the server to support OpenSearch needs + +GrimoireLab runs a combo of MariaDB, OpenSearch and Valkey(Redis). OpenSearch +needs some specific memory config on the server that runs these containers. + +Hence, before we start the OpenSearch container, there is a one time operation +to make sure the Linux server host setting `vm.max_map_count` is set to at least +262144 (instead of the default 65530). Set this value on the host machine as +explained below: + +- Check the current value, with this command: + +``` +cat /proc/sys/vm/max_map_count +``` + +- To increase the value, add (or edit) the following line to /etc/sysctl.conf +(using sudo): + +``` +vm.max_map_count=262144 +``` + +Then run `sudo sysctl -p` to reload the kernel runtime parameters configuration. + +### Ports and networks + +The main GrimoireLab services needed from outside its docker compose network +are `grimoirelab_core`, served by `nginx` on port 8000` by default and `opensearch` +served on port `9200` by default. Both of the two services are included in a +docker network named `grimoirelab_shared`, and they will be reachable using +these endpoints from inside the docker network: + + - Web server for the GrimoireLab API: http://grimoirelab-nginx:8000/ + - OpenSearch endpoint: https://grimoirelab-opensearch:9200/ + +### GrimoireLab credentials and authentication + +By default healthycode installation does not expose the bulk of GrimoireLab +services outside of the docker network, and therefore we need limited password +on each instance including OpenSearch. ScanCode.io access these through the +private docker network. + +There are provisions to set passwords on the various services if needed but +are minimally documented here. + +This various values for credentials are configured using environment variables +and defaults in docker compose file at: +docker-compose/docker-compose-grimoirelab2.yml + +You can update that file as needed. + +#### OpenSearch (aka. archivist) + +The GrimoireLab events are stored in OpenSearch cluster. + +The credentials for the `admin` user are set with the password variable +`OPENSEARCH_INITIAL_ADMIN_PASSWORD`. + +Note that OpenSearch demands a string passwords and will not start up unless +the password is strong. These OpenSearch credentials are the ones needed by the +`grimoirelab_core_archivists` component. The user is `admin` and the password +is the same you set for OpenSearch: + + - `GRIMOIRELAB_ARCHIVIST_STORAGE_USERNAME` + - `GRIMOIRELAB_ARCHIVIST_STORAGE_PASSWORD` + +#### Scheduler + +Set the credentials for the GrimoireLab scheduler and its API. This component is +used by healthycode to schedule the data collection. Set the following + variables for the scheduler: + + - `GRIMOIRELAB_SUPERUSER_USERNAME` + - `GRIMOIRELAB_SUPERUSER_PASSWORD` + +This uses two containers (`valkey` and `maria_db`) that are accessed with +empty credentials through the docker network only. + +NOTE that this is not a good practice for production services, yet only +containers inside this compose project can reach them. + +The good practice is to set passwords using the following variables for a +production enviroment: + + - For mariadb, set enviroment variables ending in: + - `*_DB_USER` + - `*_DB_PASSWORD` + + - For Valkey, set enviroment variables ending in: + - `*_REDIS_PASSWORD` + +## GrimoireLab Startup + +Once the initial configutaion above is completed, the next step if launch the +containers with these commands, from a clone of this repository: + +``` +cd docker-compose/ +docker compose -f docker-compose-grimoirelab2.yml up -d +``` + +## Installation of with PurlDB and ScanCode.io + +### Base installation + +The first step is to install PurlDB and ScanCode.io using their standard +documentation. + +- Setup PurlDB using code and documentation at https://github.com/aboutcode-org/purldb + +Once completed, make sure that PurlDB is up and running. + + +- Setup ScanCode.io as a PurlDB scan worker using code and documentation at +https://github.com/aboutcode-org/scancode.io + +### Add shared GrimoireLab docker network to ScanCode.io + +There is a temaplte for this docker network configuration in: +etc/scancodeio/network.yml. We copy that to the host: + +``` +cd healthycode-deploy/ +sudo cp ./etc/scancodeio/network.yml /etc/scancodeio/network.yml +``` + +### Edit ScanCode.io systemctl config to run with healthycode profile + +``` +sudo vi /etc/systemd/system/scancodeio.service +``` + +Update ExecStart as below: +``` +ExecStart=/usr/bin/docker compose \ + -f /opt/scancodeio/docker-compose.purldb-scan-worker.yml \ + -f /etc/scancodeio/networks.yml \ + --profile healthycode \ + up --scale worker=${NUM_WORKERS} +``` + +``` +sudo systemctl daemon-reload +``` + +### Add PurlDB base URL and API key to ScanCode.io + +Get or create a PurlDB API key for the scanworker: +``` +# on purldb +docker compose -f /opt/purldb/docker-compose.production.yml exec web python \ + manage_purldb.py create-scan-queue-worker-user +``` + +``` +sudo vi /etc/scancodeio/.env +``` + +Populate with this config, using localhost assuming eveything is co-located on +one server: +``` +PURLDB_URL= +PURLDB_API_KEY="scanworker-api-key" +``` + +### Configure GrimoireLab environment in ScanCode.io + +``` +sudo vi /etc/scancodeio/.env +``` + +Populate with this config: +``` +GRIMOIRELAB_URL="http://grimoirelab-nginx:8000/" +GRIMOIRELAB_OPENSEARCH_URL="https://grimoirelab-opensearch:9200/" +GRIMOIRELAB_ECOSYSTEM="npmlargerecosystem" +GRIMOIRELAB_PROJECT="NPMMostUsed" +``` + +Populate these fields based on the same secret that may have been set in +`./docker-compose/docker-compose-grimoirelab2.yml` + +``` +GRIMOIRELAB_USERNAME="" +GRIMOIRELAB_PASSWORD="" +GRIMOIRELAB_OPENSEARCH_INDEX="" +GRIMOIRELAB_OPENSEARCH_USERNAME="" +GRIMOIRELAB_OPENSEARCH_PASSWORD="" +``` + +### Expose ScanCode.io on port 8443 + +This is optional, and enables access to the ScanCode.io scan worker. +If using this, make sure you configure authentication in ScanCode.io first. + +``` +sudo vi /opt/scancodeio/.env +``` + +Add the below env variables: +``` +NGINX_PUBLISHED_HTTP_PORT=8081 +NGINX_PUBLISHED_HTTPS_PORT=8443 +``` + +### Restart ScanCode.io +``` +sudo systemctl restart scancodeio +``` + +## Verify everything is working end to end + + +Visit the PurlDB health API at: https:///api/health/ + +Make a test request with https:///api/health/?purl=pkg:npm/lodash&format=json + diff --git a/deploy/docker-compose/default-grimoirelab-settings/nginx.conf.template b/deploy/docker-compose/default-grimoirelab-settings/nginx.conf.template new file mode 100644 index 0000000..d020e7a --- /dev/null +++ b/deploy/docker-compose/default-grimoirelab-settings/nginx.conf.template @@ -0,0 +1,23 @@ +upstream grimoirelab_core { + server grimoirelab_core:9314; +} +server { + include mime.types; + sendfile on; + listen 8000; + + server_name localhost nginx; + + location / { + include /etc/nginx/uwsgi_params; + uwsgi_pass grimoirelab_core; + uwsgi_param Host $host; + uwsgi_param X-Real-IP $remote_addr; + uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for; + uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto; + } + + location /static/ { + alias /grimoirelab_core/; + } +} diff --git a/deploy/docker-compose/default-grimoirelab-settings/uwsgi_params b/deploy/docker-compose/default-grimoirelab-settings/uwsgi_params new file mode 100644 index 0000000..1eab45f --- /dev/null +++ b/deploy/docker-compose/default-grimoirelab-settings/uwsgi_params @@ -0,0 +1,13 @@ +uwsgi_param QUERY_STRING $query_string; +uwsgi_param REQUEST_METHOD $request_method; +uwsgi_param CONTENT_TYPE $content_type; +uwsgi_param CONTENT_LENGTH $content_length; +uwsgi_param REQUEST_URI $request_uri; +uwsgi_param PATH_INFO $document_uri; +uwsgi_param DOCUMENT_ROOT $document_root; +uwsgi_param SERVER_PROTOCOL $server_protocol; +uwsgi_param HTTPS $https if_not_empty; +uwsgi_param REMOTE_ADDR $remote_addr; +uwsgi_param REMOTE_PORT $remote_port; +uwsgi_param SERVER_PORT $server_port; +uwsgi_param SERVER_NAME $server_name; diff --git a/deploy/docker-compose/docker-compose-grimoirelab2.yml b/deploy/docker-compose/docker-compose-grimoirelab2.yml new file mode 100644 index 0000000..6e63143 --- /dev/null +++ b/deploy/docker-compose/docker-compose-grimoirelab2.yml @@ -0,0 +1,191 @@ +name: compose + +services: + mariadb: + image: mariadb:11.4 + expose: + - "3306" + environment: + MYSQL_ROOT_PASSWORD: "" + MYSQL_ALLOW_EMPTY_PASSWORD: yes + healthcheck: + test: [ "CMD", "/usr/local/bin/healthcheck.sh", "--su=root", "--connect", "--innodb_initialized" ] + retries: 5 + volumes: + - mariadb_data:/var/lib/mysql + + valkey: + image: valkey/valkey:8 + expose: + - "6379" + healthcheck: + test: [ "CMD", "valkey-cli", "--raw", "incr", "ping" ] + retries: 5 + volumes: + - valkey_data:/data + + grimoirelab_core: + restart: on-failure:3 + image: grimoirelab/grimoirelab:2.x + environment: + GRIMOIRELAB_DB_HOST: mariadb + GRIMOIRELAB_DB_PORT: 3306 + GRIMOIRELAB_DB_DATABASE: core_db + GRIMOIRELAB_DB_USER: root + GRIMOIRELAB_DB_PASSWORD: "" + GRIMOIRELAB_REDIS_HOST: valkey + GRIMOIRELAB_REDIS_PORT: 6379 + GRIMOIRELAB_REDIS_PASSWORD: "" + GRIMOIRELAB_SUPERUSER_USERNAME: admin + GRIMOIRELAB_SUPERUSER_PASSWORD: admin + GRIMOIRELAB_ALLOWED_HOST: "grimoirelab-nginx" + expose: + - "9314" + volumes: + - core-static:/core_static/ + depends_on: + mariadb: + condition: service_healthy + valkey: + condition: service_healthy + command: grimoirelab run server + + grimoirelab_core_eventizers: + restart: on-failure:3 + image: grimoirelab/grimoirelab:2.x + environment: + GRIMOIRELAB_DB_HOST: mariadb + GRIMOIRELAB_DB_PORT: 3306 + GRIMOIRELAB_DB_DATABASE: core_db + GRIMOIRELAB_DB_USER: root + GRIMOIRELAB_DB_PASSWORD: "" + GRIMOIRELAB_REDIS_HOST: valkey + GRIMOIRELAB_REDIS_PORT: 6379 + GRIMOIRELAB_REDIS_PASSWORD: "" + depends_on: + mariadb: + condition: service_healthy + valkey: + condition: service_healthy + command: grimoirelab run eventizers + + grimoirelab_core_archivists: + restart: on-failure:3 + image: grimoirelab/grimoirelab:2.x + environment: + GRIMOIRELAB_DB_HOST: mariadb + GRIMOIRELAB_DB_PORT: 3306 + GRIMOIRELAB_DB_DATABASE: core_db + GRIMOIRELAB_DB_USER: root + GRIMOIRELAB_DB_PASSWORD: "" + GRIMOIRELAB_REDIS_HOST: valkey + GRIMOIRELAB_REDIS_PORT: 6379 + GRIMOIRELAB_REDIS_PASSWORD: "" + GRIMOIRELAB_ARCHIVIST_STORAGE_URL: "https://opensearch-node1:9200" + GRIMOIRELAB_ARCHIVIST_STORAGE_USERNAME: "admin" + GRIMOIRELAB_ARCHIVIST_STORAGE_PASSWORD: "GrimoireLab.1" + depends_on: + mariadb: + condition: service_healthy + valkey: + condition: service_healthy + opensearch-node1: + condition: service_started + command: grimoirelab run archivists + + grimoirelab_core_ushers: + restart: on-failure:3 + image: grimoirelab/grimoirelab:2.x + environment: + GRIMOIRELAB_DB_HOST: mariadb + GRIMOIRELAB_DB_PORT: 3306 + GRIMOIRELAB_DB_DATABASE: core_db + GRIMOIRELAB_DB_USER: root + GRIMOIRELAB_DB_PASSWORD: "" + GRIMOIRELAB_REDIS_HOST: valkey + GRIMOIRELAB_REDIS_PORT: 6379 + GRIMOIRELAB_REDIS_PASSWORD: "" + depends_on: + mariadb: + condition: service_healthy + valkey: + condition: service_healthy + command: grimoirelab run ushers + + sortinghat_worker: + image: grimoirelab/sortinghat-worker + environment: + SORTINGHAT_SECRET_KEY: secret + SORTINGHAT_DB_HOST: mariadb + SORTINGHAT_DB_PORT: 3306 + SORTINGHAT_DB_DATABASE: core_db + SORTINGHAT_DB_USER: root + SORTINGHAT_DB_PASSWORD: "" + SORTINGHAT_REDIS_HOST: valkey + SORTINGHAT_REDIS_PORT: 6379 + SORTINGHAT_REDIS_PASSWORD: "" + depends_on: + mariadb: + condition: service_healthy + valkey: + condition: service_healthy + + opensearch-node1: + image: opensearchproject/opensearch:3 + environment: + cluster.name: opensearch-cluster + node.name: opensearch-node1 + discovery.type: single-node + bootstrap.memory_lock: true + OPENSEARCH_JAVA_OPTS: "-Xms2g -Xmx2g" + OPENSEARCH_INITIAL_ADMIN_PASSWORD: "GrimoireLab.1" + ulimits: + memlock: + soft: -1 + hard: -1 + nofile: + soft: 65536 + hard: 65536 + expose: + - "9200" + - "9600" + volumes: + - opensearch_data:/usr/share/opensearch/data + networks: + default: + shared: + aliases: + - grimoirelab-opensearch + + nginx: + restart: on-failure:3 + image: nginx:latest + volumes: + - ./default-grimoirelab-settings/nginx.conf.template:/etc/nginx/templates/default.conf.template + - ./default-grimoirelab-settings/uwsgi_params:/etc/nginx/uwsgi_params + - core-static:/grimoirelab_core:ro + expose: + - "8000" + networks: + default: + shared: + aliases: + - grimoirelab-nginx + depends_on: + - grimoirelab_core + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/"] + interval: 30s + timeout: 10s + retries: 5 + +volumes: + mariadb_data: + opensearch_data: + valkey_data: + core-static: + +networks: + default: + shared: + name: grimoirelab_shared diff --git a/deploy/etc/scancodeio/network.yml b/deploy/etc/scancodeio/network.yml new file mode 100644 index 0000000..dd6c815 --- /dev/null +++ b/deploy/etc/scancodeio/network.yml @@ -0,0 +1,4 @@ +networks: + default: + name: grimoirelab_shared + external: true \ No newline at end of file