diff --git a/README.md b/README.md index ccd31d1..1f722d5 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,8 @@ In this tutorial we will cover: - Programming models in Python for GPU applications: array expressions, fusion, library iterators, kernel authoring, and interoperability with CUDA C++ kernels. - Reading an application's profile with Nsight Systems and Nsight Compute. - Case studies of horizontal Python and C++ integration in domain sciences such as cancer research and high energy physics. -- cuPhoton and DragonHPC (placeholder). +- DragonHPC fundamentals enabling `multiprocessing` to scale across HPC multi-node resources. +- cuPhoton astronomical image processing workflows orchestrated using DragonHPC. Attendees are expected to have general knowledge of HPC and Python programming. Some familiarity with NumPy and C/C++ is helpful but not required. The outline and the notebook list are in diff --git a/brev/docker-compose.yml b/brev/docker-compose.yml index 589ffe9..b7e8c34 100644 --- a/brev/docker-compose.yml +++ b/brev/docker-compose.yml @@ -9,7 +9,7 @@ x-gpu: &gpu ulimits: memlock: -1 stack: 67108864 - shm_size: 1g + shm_size: 6g # Dragon 0.15 creates its memory pool in shared memory and asks for 6 GB. deploy: resources: reservations: diff --git a/brev/requirements.txt b/brev/requirements.txt index 8b4fea6..cd9571d 100644 --- a/brev/requirements.txt +++ b/brev/requirements.txt @@ -39,7 +39,7 @@ nsightful[notebook] @ git+https://github.com/brycelelbach/nsightful.git@a4198940 # Distributed computing. The mpi4py wheel loads the system OpenMPI. mpi4py == 4.1.2 -dragonhpc == 0.14.2 +dragonhpc == 0.15.0 # Runtime and notebook execution utilities psutil == 7.2.2 diff --git a/notebooks/distributed/12__dragonhpc_cuphoton__intro.ipynb b/notebooks/distributed/12__dragonhpc_cuphoton__intro.ipynb new file mode 100644 index 0000000..a789aa3 --- /dev/null +++ b/notebooks/distributed/12__dragonhpc_cuphoton__intro.ipynb @@ -0,0 +1,724 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0122416a", + "metadata": {}, + "source": [ + "# SC26 DragonHPC Multiprocessing, Native, and DDict Tutorial\n", + "\n", + "**Estimated time:** ~30-45 minutes \n", + "**Format:** 6 short exercises - each has a problem and a hidden answer you can reveal.\n", + "\n", + "---\n", + "\n", + "## What you will learn\n", + "\n", + "This notebook introduces Python multiprocessing with DragonHPC for single-node\n", + "and multi-node runs. You will practice how to:\n", + "\n", + "- launch processes\n", + "- synchronize worker execution\n", + "- communicate data between processes\n", + "- observe process placement across nodes\n", + "\n", + "The exercises are beginner friendly, while also connecting to concepts that\n", + "intermediate and advanced users already know from Python multiprocessing.\n", + "\n", + "---\n", + "\n", + "## Setup - run this first!\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fc68fa9d-cd2d-4056-94df-040ecd062dbe", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "import os\n", + "import socket\n", + "import time\n", + "\n", + "import dragon\n", + "import multiprocessing as mp\n", + "\n", + "# Shim for this tutorial environment; eliminated in future Dragon release\n", + "import dragon.workflows.runtime as runtime\n", + "dragon._patch_multiprocessing()\n", + "dragon_rt = runtime.start_local_runtime(timeout=5)\n", + "runtime.attach(runtime=dragon_rt, auth_token=dragon_rt.auth_token)\n", + "dragon.native.pool.current_process = mp.current_process\n", + "\n", + "mp.set_start_method(\"dragon\")\n", + "\n", + "print(\"Dragon multiprocessing ready.\")" + ] + }, + { + "cell_type": "markdown", + "id": "bfb17bfe", + "metadata": {}, + "source": [ + "`multiprocessing` synchronization and communication objects (for example\n", + "`Queue`, `Event`, `Lock`, `Value`) happily work with `\"dragon\"` set as\n", + "the start method in `multiprocessing` just as they would with any of the\n", + "other start methods in `multiprocessing` (i.e. `\"spawn\"`, `\"fork\"`, or\n", + "`\"forkserver\"`).\n", + "\n", + "When the Dragon runtime is available, a number of opportunities for\n", + "optimization arise which is why there are additional tools available\n", + "in the `dragon` module. After exercising the use of Dragon through\n", + "standard `multiprocessing`, we will begin by introducing the Dragon\n", + "Native `Process` object which exposes further control over the creation\n", + "and placement of processes, noting how it intentionally mirrors the\n", + "familiar multiprocessing API patterns." + ] + }, + { + "cell_type": "markdown", + "id": "03fdc111", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 1 - Parallel map with multiprocessing Pool\n", + "\n", + "**Background:**\n", + "\n", + "`multiprocessing.Pool` is useful when the same function should run over many\n", + "inputs. With Dragon configured as the multiprocessing backend, you can use\n", + "`Pool.map` for data-parallel workloads executing across many nodes and all\n", + "communication will travel over Dragon's HSTA (High Speed Transfer Agent).\n", + "\n", + "Ref: https://docs.python.org/3/library/multiprocessing.html#introduction\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "def double(x):\n", + " return 2 * x\n", + "\n", + "with mp.Pool(processes=4) as pool:\n", + " out = pool.map(double, [1, 2, 3, 4])\n", + "\n", + "print(out) # [2, 4, 6, 8]\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Write a function `transform(x)` that returns `x * x + 1`\n", + "2. Use `mp.Pool(processes=4)`\n", + "3. Call `pool.map(transform, data)` where `data = list(range(12))`\n", + "4. Print the mapped results\n", + "5. Print the sum of the mapped results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ef13d43c", + "metadata": {}, + "outputs": [], + "source": [ + "# -- Exercise 1 -- your code here -----------------------------------------------\n" + ] + }, + { + "cell_type": "markdown", + "id": "b7fcc5f5", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def transform(x):\n", + " return x * x + 1\n", + "\n", + "data = list(range(12))\n", + "\n", + "with mp.Pool(processes=4) as pool:\n", + " mapped = pool.map(transform, data)\n", + "\n", + "print(mapped)\n", + "print(\"sum:\", sum(mapped))\n", + "```\n", + "\n", + "Expected mapped output:\n", + "```\n", + "[1, 2, 5, 10, 17, 26, 37, 50, 65, 82, 101, 122]\n", + "sum: 518\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "d8e54e8b", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 2 - Launch processes and join, using queues to communicate\n", + "\n", + "**Background:**\n", + "\n", + "A `mp.Process` is started with `start()` and waited on with `join()`. This is the\n", + "first basic building block for parallel programs though in contrast to the\n", + "prior example employing `mp.Pool`, the return values from target\n", + "functions are not collected and returned to the caller. This motivates the\n", + "use of `mp.Queue` for process-safe message passing which, among many other uses,\n", + "feels like an especially natural fit for producer-consumer code patterns.\n", + "\n", + "By setting `multiprocessing`'s start method to `\"dragon\"`, all of these tools\n", + "from the `multiprocessing` module will leverage the Dragon runtime. But Dragon\n", + "also offers tools to unlock opportunities for optimization, tuning, and\n", + "control that `multiprocessing` does not. The first basic example of this is\n", + "the Dragon Native process object, `dragon.native.Process`, which supports the\n", + "standard `mp.Process` API but expands on it as we will see in a later exercise.\n", + "\n", + "Here is the core Dragon Native process pattern:\n", + "\n", + "```python\n", + "def hello_worker(name):\n", + " print(f\"hello from {name}, pid={os.getpid()}\") # stdout is not generally a best practice for collection\n", + "\n", + "p = dragon.native.Process(target=hello_worker, args=(\"dragon\",))\n", + "p.start()\n", + "p.join()\n", + "```\n", + "\n", + "Here is the core pattern for `Queue`:\n", + "\n", + "```python\n", + "q = mp.Queue()\n", + "\n", + "def say_hello(q):\n", + " q.put((\"msg\", \"hello from worker\"))\n", + "\n", + "p = dragon.native.Process(target=say_hello, args=(q,)) # mp.Process works here too\n", + "p.start()\n", + "message = q.get()\n", + "p.join()\n", + "print(message)\n", + "```\n", + "\n", + "\n", + "**Your task:**\n", + "\n", + "1. Write a worker function `square_worker(i, q)` that accepts an `int` i and `Queue` q, constructs a nicely formmated `str` containing the worker id `i`, PID, and `i*i`, and places that string into the queue.\n", + "2. Launch 4 processes using `dragon.native.Process` (though `mp.Process` would also work; try both?)\n", + "3. Start all processes, read and print the 4 messages placed into the queue by the workers, then join all processes\n", + "4. Print `All workers finished` at the end" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "29b212cc", + "metadata": {}, + "outputs": [], + "source": [ + "# -- Exercise 2 -- your code here -----------------------------------------------\n" + ] + }, + { + "cell_type": "markdown", + "id": "f35a5475", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def square_worker(i, q):\n", + " q.put(f\"worker={i} pid={os.getpid()} square={i*i}\")\n", + "\n", + "q = mp.Queue()\n", + "procs = [dragon.native.Process(target=square_worker, args=(i, q)) for i in range(4)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "print(\"All workers finished\")\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "1975b9da", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 3 - Phase synchronization with Barrier\n", + "\n", + "**Background:**\n", + "\n", + "A `Barrier` blocks each worker until a fixed number of participants arrive.\n", + "This is useful for staged workflows where all workers must reach the same\n", + "checkpoint before continuing together.\n", + "\n", + "With Dragon, `Barrier` objects work in single-node or multi-node\n", + "execution environments.\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "barrier = mp.Barrier(2)\n", + "\n", + "def waiter(barrier):\n", + " print(\"before barrier\")\n", + " barrier.wait()\n", + " print(\"after barrier\")\n", + "\n", + "p = dragon.native.Process(target=waiter, args=(barrier,))\n", + "p.start()\n", + "time.sleep(0.5)\n", + "barrier.wait()\n", + "p.join()\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Create a `Barrier` named `phase_barrier` with 6 participants (5 workers + parent)\n", + "2. Write worker code that prints `ready`, waits on the barrier, then prints `running`\n", + "3. Launch 5 workers\n", + "4. Sleep for 2 seconds in the parent and then call `phase_barrier.wait()`\n", + "5. Join workers" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dbf19fcd", + "metadata": {}, + "outputs": [], + "source": [ + "# -- Exercise 3 -- your code here -----------------------------------------------\n" + ] + }, + { + "cell_type": "markdown", + "id": "f8f3674b", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def staged_worker(worker_id, phase_barrier):\n", + " print(f\"worker={worker_id} ready\")\n", + " phase_barrier.wait()\n", + " print(f\"worker={worker_id} running\")\n", + "\n", + "phase_barrier = mp.Barrier(6)\n", + "procs = [dragon.native.Process(target=staged_worker, args=(i, phase_barrier)) for i in range(5)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "time.sleep(2.0)\n", + "print(\"Parent arriving at barrier\")\n", + "phase_barrier.wait()\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "90e25403", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 4 - Control CPU affinity with Dragon Policy\n", + "\n", + "**Background:**\n", + "\n", + "Dragon `Policy` lets you control affinity. For CPUs, use `cpu_affinity`.\n", + "For example, `Policy(cpu_affinity=[1])` requests placement targeting CPU 1.\n", + "A `Policy` can be used with `dragon.native.Process` but not with `mp.Process`.\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "def report_affinity(out_q):\n", + " out_q.put((os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n", + "\n", + "q = mp.Queue()\n", + "pol = Policy(cpu_affinity=[0])\n", + "p = dragon.native.Process(target=report_affinity, args=(q,), policy=pol)\n", + "p.start()\n", + "print(q.get())\n", + "p.join()\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Create two policies: one for CPU 0 and one for CPU 1\n", + "2. Launch two workers with those explicit policies\n", + "3. Each worker should report `(rank, pid, CUDA_VISIBLE_DEVICES)`\n", + "4. Print both reports\n", + "\n", + "If your system has fewer than 2 CPUs, use CPU fallback and still run both\n", + "workers (the environment report may be the same)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "93054056", + "metadata": {}, + "outputs": [], + "source": [ + "# -- Exercise 4 -- your code here -----------------------------------------------\n", + "def affinity_worker(rank, out_q):\n", + " out_q.put((rank, os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n" + ] + }, + { + "cell_type": "markdown", + "id": "aa55840e", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def affinity_worker(rank, out_q):\n", + " out_q.put((rank, os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n", + "\n", + "q = mp.Queue()\n", + "num_cpus = mp.cpu_count()\n", + "cpu0 = 0\n", + "cpu1 = 1 if num_cpus > 1 else 0\n", + "\n", + "policies = [dragon.Policy(cpu_affinity=[cpu0]), dragon.Policy(cpu_affinity=[cpu1])]\n", + "procs = [\n", + " dragon.native.Process(target=affinity_worker, args=(0, q), policy=policies[0]),\n", + " dragon.native.Process(target=affinity_worker, args=(1, q), policy=policies[1]),\n", + "]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for _p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "146c2f10-2ec6-4f63-95b9-a75f7447fd0f", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 5 — Create, write, read, and destroy a `DDict`\n", + "\n", + "**Background:**\n", + "\n", + "A `DDict` is Dragon's **distributed in-memory key-value store**. It looks and\n", + "feels like a Python `dict`, but the data lives across **one or more manager\n", + "processes** that can span multiple nodes. The best part is that the organization\n", + "of the data is transparent to the user. When user code interacts with it, the\n", + "code does not need to know the location of any key-value pair stored in it.\n", + "\n", + "\n", + "\n", + "Key characteristics:\n", + "- Standard `dict`-like interface (`d[key] = val`, `d[key]`, `del d[key]`, `len(d)`, ...)\n", + "- Values are transparently serialized with `cloudpickle` or a pickler of your choice\n", + "- Data is shared safely across **multiple processes** without locks\n", + "- Supports **checkpointing**, **persistent keys**, and **filtering** \n", + "- Other functionality not covered here like **broadcast puts** and **batch puts**\n", + "- Data is automatically balanced across all the managers\n", + "\n", + "Instantiating and using an empty `DDict` object is as easy as:\n", + "\n", + "```python\n", + "from dragon import DDict\n", + "\n", + "d = DDict()\n", + "\n", + "d[\"greeting\"] = \"howdy\"\n", + "d[\"meaning\"] = 42\n", + "print(len(d)) # 2\n", + "```\n", + "\n", + "**Your Task:**\n", + "\n", + "1. Create a `DDict`\n", + "2. Store `{\"cat\": 1, \"dog\": 2, \"bird\": 3}`\n", + "3. Print all keys by iterating over `d.keys()`\n", + "4. Check whether `\"dog\"` and `\"fish\"` are in the dictionary\n", + "5. Delete `\"cat\"` and confirm the length dropped to 2\n", + "6. Use `d.get(\"fish\", \"not found\")` and print the result\n", + "7. Destroy the dictionary (to release its allocated memory)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "435a7f82-d8e9-4ba7-8d04-5c72ebcd9bca", + "metadata": {}, + "outputs": [], + "source": [ + "# ── Exercise 5 ── your code here ─────────────────────────────────────────────\n" + ] + }, + { + "cell_type": "markdown", + "id": "fe0ec91f-7c22-4898-88fd-d3a2176903fa", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "d = dragon.DDict()\n", + "\n", + "d[\"cat\"] = 1\n", + "d[\"dog\"] = 2\n", + "d[\"bird\"] = 3\n", + "\n", + "print(\"keys:\", list(d.keys())) # ['cat', 'dog', 'bird'] (order may vary)\n", + "print(\"dog in d:\", \"dog\" in d) # True\n", + "print(\"fish in d:\", \"fish\" in d) # False\n", + "\n", + "del d[\"cat\"]\n", + "print(\"len after delete:\", len(d)) # 2\n", + "\n", + "print(d.get(\"fish\", \"not found\")) # not found\n", + "\n", + "d.destroy()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "2de80c25-b9bd-4643-b24d-19a3530488fa", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 6 — Sharing a `DDict` across processes\n", + "\n", + "**Background:**\n", + "\n", + "A `DDict` is serializable (`pickle`-able). Passing a `DDict` to a distinct process does not cause an expensive transfer or serialization of any its contents (the keys or values). Instead, that process will share the **same** `DDict`, capable of being accessed or mutated by both child and parent.\n", + "\n", + "When instantiating a `DDict` object, you may specify the number of managers and nodes\n", + "on which the `DDict` will hold its distributed data. A `DDict` automatically starts\n", + "up a specified number of managers and an orchestrator (not shown in the picture) and\n", + "user client code interacts with these processes to transparently provide you with a\n", + "distributed key-value store that can scale to any size and automatically distributes\n", + "your data evenly across all the mangers.\n", + "\n", + "\n", + "```python\n", + "import dragon.native\n", + "\n", + "def worker(d, key, value):\n", + " d[key] = value # writes into the shared DDict\n", + " d[\"hello\"] = \"world\" # keys and values don't have to be passed to child proceses\n", + "\n", + "d = DDict(1, 1, 32 * 1024 * 1024) # DDict(managers_per_node, n_nodes, total_mem_bytes)\n", + "proc = dragon.native.Process(target=worker, args=(d, \"x\", 99)) # or use mp.Process\n", + "proc.start() # starts the process\n", + "proc.join() # waits for the process to exit\n", + "print(d[\"hello\"]) # and then prints world\n", + "d.destroy()\n", + "```\n", + "\n", + "This tells Dragon to instantiate and bring up a `DDict` with `managers_per_node * n_nodes` managers. The `total_mem_bytes` is spread across all the managers. For more information on creating `DDict` objects [see the DDict API](https://dragonhpc.github.io/dragon/doc/_build/html/ref/data/dragon.data.DDict.html#dragon.data.DDict). \n", + "\n", + "- `managers_per_node=1`, `n_nodes=1` → one manager, single node\n", + "- `total_mem_bytes=64*1024*1024` → 64 MB pool\n", + "- Remember to call `d.destroy()` to release resources when finished\n", + "\n", + "\n", + "**Your Task:**\n", + "\n", + "Write a function `count_words(label, text, d)` that splits `text` on spaces and stores\n", + "`d[label] = count` as a count of the words in the supplied text.\n", + "\n", + "Hint: Calling `s.split()` on a `str` object `s` in Python will return a list of all the space \n", + "separated *words* of the string.\n", + "\n", + "Then:\n", + "1. Create a `DDict` (2 managers, 1 node, 64 MB)\n", + "2. Launch **3 processes**, each calling `count_words` with a different song lyric\n", + "3. After all processes finish, print every key-value pair in the dictionary\n", + "4. Destroy the dictionary\n", + "\n", + "Song lyrics and associated labels to use:\n", + "```\n", + "(\"never gonna give you up never gonna let you down\", \"rick\")\n", + "(\"hello is it me you're looking for\", \"lionel\")\n", + "(\"cause I'll be rocking this party eight days a week\", \"beastie\")\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "098b4e60-63d1-4626-89f1-97ca635b9f13", + "metadata": {}, + "outputs": [], + "source": [ + "# ── Exercise 6 ── your code here ─────────────────────────────────────────────\n", + "lyrics_and_labels = [\n", + " (\"never gonna give you up never gonna let you down\", \"rick\"),\n", + " (\"hello is it me you're looking for\", \"lionel\"),\n", + " (\"cause I'll be rocking this party eight days a week\", \"beastie\"),\n", + "]\n" + ] + }, + { + "cell_type": "markdown", + "id": "31d72121-7b06-4a8e-a7a4-dd1c8a6b0b6a", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "lyrics_and_labels = [\n", + " (\"never gonna give you up never gonna let you down\", \"rick\"),\n", + " (\"hello is it me you're looking for\", \"lionel\"),\n", + " (\"cause I'll be rocking this party eight days a week\", \"beastie\"),\n", + "]\n", + "\n", + "def count_words(label, text, d):\n", + " d[label] = len(text.split())\n", + "\n", + "d = dragon.DDict(2, 1, 64 * 1024 * 1024)\n", + "\n", + "procs = [dragon.native.Process(target=count_words, args=(l, t, d)) for t, l in lyrics_and_labels]\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "for k, v in d.items():\n", + " print(f\" {k}: {v}\")\n", + "\n", + "d.destroy()\n", + "```\n", + "\n", + "
\n", + "\n", + "**Synchronizing Updates to the Same Key in Parallel** \n", + "\n", + "Because multiple processes may attempt to store an updated value for a key which depends upon the\n", + "existing associated value, care must be taken because whichever process writes last wins. Depending\n", + "upon your code's needs, look at some of the functionality available on the `DDict` that has no\n", + "equivalent on the standard `dict` such as the atomic update `fetch_add` method to add values to\n", + "the existing stored value.\n", + "\n", + "For example, the `fetch_add` will operate atomically to add an integer value to a specified key/value pair. \n", + "\n", + "```python\n", + "def count_words(d, text):\n", + " for word, count in Counter(text.split()).items():\n", + " d.fetch_add(word, count)\n", + "```\n", + "\n", + "Using `fetch_add` to retrieve fetch/added values works in both Python and C++ Dragon APIs.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "28472c80", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Summary\n", + "\n", + "You used DragonHPC with familiar multiprocessing concepts to launch workers,\n", + "synchronize phases, communicate data, and run data-parallel map operations.\n", + "The same API patterns apply whether you run on one node or many.\n", + "\n", + "| Concept | API |\n", + "|---|---|\n", + "| Start Dragon backend | `mp.set_start_method(\"dragon\")` |\n", + "| Launch process | `mp.Process` and/or `dragon.native.Process(target=..., args=...)` |\n", + "| Wait for completion | `p.join()` |\n", + "| Mutual exclusion | `mp.Lock()` |\n", + "| Shared scalar | `mp.Value(...)` |\n", + "| Message passing | `mp.Queue()` |\n", + "| Phase synchronization | `mp.Barrier(n)` |\n", + "| Pool map | `mp.Pool(...).map(func, data)` |\n", + "| Multi-node placement check | `socket.gethostname()` |\n", + "\n", + "### What next?\n", + "\n", + "Try adapting these exercises to your own application by replacing toy data with\n", + "real workloads and scaling the worker count to your full Dragon allocation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "168332f4", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/distributed/12__dragonhpc_cuphoton__workflow.ipynb b/notebooks/distributed/13__dragonhpc_cuphoton__workflow.ipynb similarity index 99% rename from notebooks/distributed/12__dragonhpc_cuphoton__workflow.ipynb rename to notebooks/distributed/13__dragonhpc_cuphoton__workflow.ipynb index 1ade714..d32d9a1 100644 --- a/notebooks/distributed/12__dragonhpc_cuphoton__workflow.ipynb +++ b/notebooks/distributed/13__dragonhpc_cuphoton__workflow.ipynb @@ -5,7 +5,10 @@ "id": "sc26-00", "metadata": {}, "source": [ - "# From real sky images to GPU workers: cuPhoton with Dragon\n", + "# SC26 cuPhoton + DragonHPC Tutorial: From real sky images to GPU workers\n", + "\n", + "**Estimated time:** ~35 minutes \n", + "**Format:** Runnable demos as walkthru of major cuPhoton features and Dragon integration.\n", "\n", "A 35-minute walkthrough for the SC26 cuPhoton and Dragon session. We begin with\n", "observational FITS pixels, add controlled changes, and follow the same data through\n", @@ -26,7 +29,7 @@ "\n", "The optional final section adds 5–7 minutes of automatic detection and uncertainty\n", "analysis. Preparation, downloads, native builds, and training happen **before**\n", - "the session; follow [README.md](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/examples/dragon-imaging-pipeline/README.md). Run cells in order. All computation\n", + "the session and are not described here. Run cells in order. All computation\n", "requires Linux, CUDA 13, and the documented Python 3.12 environment.\n" ] }, diff --git a/notebooks/distributed/images/ddict_overview.png b/notebooks/distributed/images/ddict_overview.png new file mode 100644 index 0000000..945f46a --- /dev/null +++ b/notebooks/distributed/images/ddict_overview.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e3ce575f89e8302cfe239937da51c579a24f0a93aa688528dad83d818279577 +size 65569 diff --git a/notebooks/distributed/solutions/12__dragonhpc_cuphoton__intro__SOLUTION.ipynb b/notebooks/distributed/solutions/12__dragonhpc_cuphoton__intro__SOLUTION.ipynb new file mode 100644 index 0000000..70a2499 --- /dev/null +++ b/notebooks/distributed/solutions/12__dragonhpc_cuphoton__intro__SOLUTION.ipynb @@ -0,0 +1,888 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0122416a", + "metadata": {}, + "source": [ + "# SC26 DragonHPC Multiprocessing, Native, and DDict Tutorial\n", + "\n", + "**Estimated time:** ~30-45 minutes \n", + "**Format:** 6 short exercises - each has a problem and a hidden answer you can reveal.\n", + "\n", + "---\n", + "\n", + "## What you will learn\n", + "\n", + "This notebook introduces Python multiprocessing with DragonHPC for single-node\n", + "and multi-node runs. You will practice how to:\n", + "\n", + "- launch processes\n", + "- synchronize worker execution\n", + "- communicate data between processes\n", + "- observe process placement across nodes\n", + "\n", + "The exercises are beginner friendly, while also connecting to concepts that\n", + "intermediate and advanced users already know from Python multiprocessing.\n", + "\n", + "---\n", + "\n", + "## Setup - run this first!\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "fc68fa9d-cd2d-4056-94df-040ecd062dbe", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "info={'node_index': 0, 'gs_qd': 'BV8AAAAAAAAAAgAAAAAAAAA3AAAAAAAAAAAAAAAAAAAAcfNluE+0TwcFLtd6GM2leAAAAAAAAAAA4NjABAAAAAAvREQwMDAxMzVpbmYwTQADAIAAAAAAAAAAAAAAAAAACOABAAAAAAA=', 'ls_qd': 'BV8AAAAAAAAAAAAAAAAAAEA3AAAAAAAAAAAAAAAAAAAAcfNluE+0TwcFLtd6GM2leAAAAAAAAAAA4NjABAAAAAAvREQwMDAxMzVpbmYwTQAAACAAAAAAAAAAAAAAAAAACOABAAAAAAA=', 'default_pd': 'AAAAAAAAAEBx82W4T7RPBwUu13oYzaV4AAAAAAAAAACA6QAHAAAAAC9ERDAwMDEzNWRlZjBNAA==', 'inf_pd': 'AAAAAAAAAABx82W4T7RPBwUu13oYzaV4AAAAAAAAAADg2MAEAAAAAC9ERDAwMDEzNWluZjBNAA==', 'rt_uid': '8693580160732442117', 'auth_token': 'c047cb39c9de6919a17499113a4cfcace0aead0f9c1bd8036411a6a1d95be510'}\n", + "Dragon multiprocessing ready.\n" + ] + } + ], + "source": [ + "import os\n", + "import socket\n", + "import time\n", + "\n", + "import dragon\n", + "import multiprocessing as mp\n", + "\n", + "# Shim for this tutorial environment; eliminated in future Dragon release\n", + "import dragon.workflows.runtime as runtime\n", + "dragon._patch_multiprocessing()\n", + "dragon_rt = runtime.start_local_runtime(timeout=5)\n", + "runtime.attach(runtime=dragon_rt, auth_token=dragon_rt.auth_token)\n", + "dragon.native.pool.current_process = mp.current_process\n", + "\n", + "mp.set_start_method(\"dragon\")\n", + "\n", + "print(\"Dragon multiprocessing ready.\")" + ] + }, + { + "cell_type": "markdown", + "id": "bfb17bfe", + "metadata": {}, + "source": [ + "`multiprocessing` synchronization and communication objects (for example\n", + "`Queue`, `Event`, `Lock`, `Value`) happily work with `\"dragon\"` set as\n", + "the start method in `multiprocessing` just as they would with any of the\n", + "other start methods in `multiprocessing` (i.e. `\"spawn\"`, `\"fork\"`, or\n", + "`\"forkserver\"`).\n", + "\n", + "When the Dragon runtime is available, a number of opportunities for\n", + "optimization arise which is why there are additional tools available\n", + "in the `dragon` module. After exercising the use of Dragon through\n", + "standard `multiprocessing`, we will begin by introducing the Dragon\n", + "Native `Process` object which exposes further control over the creation\n", + "and placement of processes, noting how it intentionally mirrors the\n", + "familiar multiprocessing API patterns." + ] + }, + { + "cell_type": "markdown", + "id": "03fdc111", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 1 - Parallel map with multiprocessing Pool\n", + "\n", + "**Background:**\n", + "\n", + "`multiprocessing.Pool` is useful when the same function should run over many\n", + "inputs. With Dragon configured as the multiprocessing backend, you can use\n", + "`Pool.map` for data-parallel workloads executing across many nodes and all\n", + "communication will travel over Dragon's HSTA (High Speed Transfer Agent).\n", + "\n", + "Ref: https://docs.python.org/3/library/multiprocessing.html#introduction\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "def double(x):\n", + " return 2 * x\n", + "\n", + "with mp.Pool(processes=4) as pool:\n", + " out = pool.map(double, [1, 2, 3, 4])\n", + "\n", + "print(out) # [2, 4, 6, 8]\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Write a function `transform(x)` that returns `x * x + 1`\n", + "2. Use `mp.Pool(processes=4)`\n", + "3. Call `pool.map(transform, data)` where `data = list(range(12))`\n", + "4. Print the mapped results\n", + "5. Print the sum of the mapped results" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "ef13d43c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[1, 2, 5, 10, 17, 26, 37, 50, 65, 82, 101, 122]\n", + "sum: 518\n" + ] + } + ], + "source": [ + "# -- Exercise 1 -- your code here -----------------------------------------------\n", + "def transform(x):\n", + " return x * x + 1\n", + "\n", + "data = list(range(12))\n", + "\n", + "with mp.Pool(processes=4) as pool:\n", + " mapped = pool.map(transform, data)\n", + "\n", + "print(mapped)\n", + "print(\"sum:\", sum(mapped))" + ] + }, + { + "cell_type": "markdown", + "id": "b7fcc5f5", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def transform(x):\n", + " return x * x + 1\n", + "\n", + "data = list(range(12))\n", + "\n", + "with mp.Pool(processes=4) as pool:\n", + " mapped = pool.map(transform, data)\n", + "\n", + "print(mapped)\n", + "print(\"sum:\", sum(mapped))\n", + "```\n", + "\n", + "Expected mapped output:\n", + "```\n", + "[1, 2, 5, 10, 17, 26, 37, 50, 65, 82, 101, 122]\n", + "sum: 518\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "d8e54e8b", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 2 - Launch processes and join, using queues to communicate\n", + "\n", + "**Background:**\n", + "\n", + "A `mp.Process` is started with `start()` and waited on with `join()`. This is the\n", + "first basic building block for parallel programs though in contrast to the\n", + "prior example employing `mp.Pool`, the return values from target\n", + "functions are not collected and returned to the caller. This motivates the\n", + "use of `mp.Queue` for process-safe message passing which, among many other uses,\n", + "feels like an especially natural fit for producer-consumer code patterns.\n", + "\n", + "By setting `multiprocessing`'s start method to `\"dragon\"`, all of these tools\n", + "from the `multiprocessing` module will leverage the Dragon runtime. But Dragon\n", + "also offers tools to unlock opportunities for optimization, tuning, and\n", + "control that `multiprocessing` does not. The first basic example of this is\n", + "the Dragon Native process object, `dragon.native.Process`, which supports the\n", + "standard `mp.Process` API but expands on it as we will see in a later exercise.\n", + "\n", + "Here is the core Dragon Native process pattern:\n", + "\n", + "```python\n", + "def hello_worker(name):\n", + " print(f\"hello from {name}, pid={os.getpid()}\") # stdout is not generally a best practice for collection\n", + "\n", + "p = dragon.native.Process(target=hello_worker, args=(\"dragon\",))\n", + "p.start()\n", + "p.join()\n", + "```\n", + "\n", + "Here is the core pattern for `Queue`:\n", + "\n", + "```python\n", + "q = mp.Queue()\n", + "\n", + "def say_hello(q):\n", + " q.put((\"msg\", \"hello from worker\"))\n", + "\n", + "p = dragon.native.Process(target=say_hello, args=(q,)) # mp.Process works here too\n", + "p.start()\n", + "message = q.get()\n", + "p.join()\n", + "print(message)\n", + "```\n", + "\n", + "\n", + "**Your task:**\n", + "\n", + "1. Write a worker function `square_worker(i, q)` that accepts an `int` i and `Queue` q, constructs a nicely formmated `str` containing the worker id `i`, PID, and `i*i`, and places that string into the queue.\n", + "2. Launch 4 processes using `dragon.native.Process` (though `mp.Process` would also work; try both?)\n", + "3. Start all processes, read and print the 4 messages placed into the queue by the workers, then join all processes\n", + "4. Print `All workers finished` at the end" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "29b212cc", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "worker=0 pid=52321 square=0\n", + "worker=2 pid=52323 square=4\n", + "worker=1 pid=52322 square=1\n", + "worker=3 pid=52324 square=9\n", + "All workers finished\n" + ] + } + ], + "source": [ + "# -- Exercise 2 -- your code here -----------------------------------------------\n", + "def square_worker(i, q):\n", + " q.put(f\"worker={i} pid={os.getpid()} square={i*i}\")\n", + "\n", + "q = mp.Queue()\n", + "procs = [dragon.native.Process(target=square_worker, args=(i, q)) for i in range(4)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "print(\"All workers finished\")" + ] + }, + { + "cell_type": "markdown", + "id": "f35a5475", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def square_worker(i, q):\n", + " q.put(f\"worker={i} pid={os.getpid()} square={i*i}\")\n", + "\n", + "q = mp.Queue()\n", + "procs = [dragon.native.Process(target=square_worker, args=(i, q)) for i in range(4)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "print(\"All workers finished\")\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "1975b9da", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 3 - Phase synchronization with Barrier\n", + "\n", + "**Background:**\n", + "\n", + "A `Barrier` blocks each worker until a fixed number of participants arrive.\n", + "This is useful for staged workflows where all workers must reach the same\n", + "checkpoint before continuing together.\n", + "\n", + "With Dragon, `Barrier` objects work in single-node or multi-node\n", + "execution environments.\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "barrier = mp.Barrier(2)\n", + "\n", + "def waiter(barrier):\n", + " print(\"before barrier\")\n", + " barrier.wait()\n", + " print(\"after barrier\")\n", + "\n", + "p = dragon.native.Process(target=waiter, args=(barrier,))\n", + "p.start()\n", + "time.sleep(0.5)\n", + "barrier.wait()\n", + "p.join()\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Create a `Barrier` named `phase_barrier` with 6 participants (5 workers + parent)\n", + "2. Write worker code that prints `ready`, waits on the barrier, then prints `running`\n", + "3. Launch 5 workers\n", + "4. Sleep for 2 seconds in the parent and then call `phase_barrier.wait()`\n", + "5. Join workers" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "dbf19fcd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Parent arriving at barrier\n" + ] + } + ], + "source": [ + "# -- Exercise 3 -- your code here -----------------------------------------------\n", + "def staged_worker(worker_id, phase_barrier):\n", + " print(f\"worker={worker_id} ready\")\n", + " phase_barrier.wait()\n", + " print(f\"worker={worker_id} running\")\n", + "\n", + "phase_barrier = mp.Barrier(6)\n", + "procs = [dragon.native.Process(target=staged_worker, args=(i, phase_barrier)) for i in range(5)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "time.sleep(2.0)\n", + "print(\"Parent arriving at barrier\")\n", + "phase_barrier.wait()\n", + "\n", + "for p in procs:\n", + " p.join()" + ] + }, + { + "cell_type": "markdown", + "id": "f8f3674b", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def staged_worker(worker_id, phase_barrier):\n", + " print(f\"worker={worker_id} ready\")\n", + " phase_barrier.wait()\n", + " print(f\"worker={worker_id} running\")\n", + "\n", + "phase_barrier = mp.Barrier(6)\n", + "procs = [dragon.native.Process(target=staged_worker, args=(i, phase_barrier)) for i in range(5)]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "time.sleep(2.0)\n", + "print(\"Parent arriving at barrier\")\n", + "phase_barrier.wait()\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "90e25403", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 4 - Control CPU affinity with Dragon Policy\n", + "\n", + "**Background:**\n", + "\n", + "Dragon `Policy` lets you control affinity. For CPUs, use `cpu_affinity`.\n", + "For example, `Policy(cpu_affinity=[1])` requests placement targeting CPU 1.\n", + "A `Policy` can be used with `dragon.native.Process` but not with `mp.Process`.\n", + "\n", + "Demo pattern:\n", + "\n", + "```python\n", + "def report_affinity(out_q):\n", + " out_q.put((os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n", + "\n", + "q = mp.Queue()\n", + "pol = Policy(cpu_affinity=[0])\n", + "p = dragon.native.Process(target=report_affinity, args=(q,), policy=pol)\n", + "p.start()\n", + "print(q.get())\n", + "p.join()\n", + "```\n", + "\n", + "**Your task:**\n", + "\n", + "1. Create two policies: one for CPU 0 and one for CPU 1\n", + "2. Launch two workers with those explicit policies\n", + "3. Each worker should report `(rank, pid, CUDA_VISIBLE_DEVICES)`\n", + "4. Print both reports\n", + "\n", + "If your system has fewer than 2 CPUs, use CPU fallback and still run both\n", + "workers (the environment report may be the same)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "93054056", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(1, 52538, 'unset')\n", + "(0, 52537, 'unset')\n" + ] + } + ], + "source": [ + "# -- Exercise 4 -- your code here -----------------------------------------------\n", + "def affinity_worker(rank, out_q):\n", + " out_q.put((rank, os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n", + "\n", + "q = mp.Queue()\n", + "num_cpus = mp.cpu_count()\n", + "cpu0 = 0\n", + "cpu1 = 1 if num_cpus > 1 else 0\n", + "\n", + "policies = [dragon.Policy(cpu_affinity=[cpu0]), dragon.Policy(cpu_affinity=[cpu1])]\n", + "procs = [\n", + " dragon.native.Process(target=affinity_worker, args=(0, q), policy=policies[0]),\n", + " dragon.native.Process(target=affinity_worker, args=(1, q), policy=policies[1]),\n", + "]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for _p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()" + ] + }, + { + "cell_type": "markdown", + "id": "aa55840e", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "def affinity_worker(rank, out_q):\n", + " out_q.put((rank, os.getpid(), os.environ.get(\"CUDA_VISIBLE_DEVICES\", \"unset\")))\n", + "\n", + "q = mp.Queue()\n", + "num_cpus = mp.cpu_count()\n", + "cpu0 = 0\n", + "cpu1 = 1 if num_cpus > 1 else 0\n", + "\n", + "policies = [dragon.Policy(cpu_affinity=[cpu0]), dragon.Policy(cpu_affinity=[cpu1])]\n", + "procs = [\n", + " dragon.native.Process(target=affinity_worker, args=(0, q), policy=policies[0]),\n", + " dragon.native.Process(target=affinity_worker, args=(1, q), policy=policies[1]),\n", + "]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "\n", + "for _p in procs:\n", + " print(q.get())\n", + "\n", + "for p in procs:\n", + " p.join()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "146c2f10-2ec6-4f63-95b9-a75f7447fd0f", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 5 — Create, write, read, and destroy a `DDict`\n", + "\n", + "**Background:**\n", + "\n", + "A `DDict` is Dragon's **distributed in-memory key-value store**. It looks and\n", + "feels like a Python `dict`, but the data lives across **one or more manager\n", + "processes** that can span multiple nodes. The best part is that the organization\n", + "of the data is transparent to the user. When user code interacts with it, the\n", + "code does not need to know the location of any key-value pair stored in it.\n", + "\n", + "\n", + "\n", + "Key characteristics:\n", + "- Standard `dict`-like interface (`d[key] = val`, `d[key]`, `del d[key]`, `len(d)`, ...)\n", + "- Values are transparently serialized with `cloudpickle` or a pickler of your choice\n", + "- Data is shared safely across **multiple processes** without locks\n", + "- Supports **checkpointing**, **persistent keys**, and **filtering** \n", + "- Other functionality not covered here like **broadcast puts** and **batch puts**\n", + "- Data is automatically balanced across all the managers\n", + "\n", + "Instantiating and using an empty `DDict` object is as easy as:\n", + "\n", + "```python\n", + "from dragon import DDict\n", + "\n", + "d = DDict()\n", + "\n", + "d[\"greeting\"] = \"howdy\"\n", + "d[\"meaning\"] = 42\n", + "print(len(d)) # 2\n", + "```\n", + "\n", + "**Your Task:**\n", + "\n", + "1. Create a `DDict`\n", + "2. Store `{\"cat\": 1, \"dog\": 2, \"bird\": 3}`\n", + "3. Print all keys by iterating over `d.keys()`\n", + "4. Check whether `\"dog\"` and `\"fish\"` are in the dictionary\n", + "5. Delete `\"cat\"` and confirm the length dropped to 2\n", + "6. Use `d.get(\"fish\", \"not found\")` and print the result\n", + "7. Destroy the dictionary (to release its allocated memory)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "435a7f82-d8e9-4ba7-8d04-5c72ebcd9bca", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "keys: ['bird', 'dog', 'cat']\n", + "dog in d: True\n", + "fish in d: False\n", + "len after delete: 2\n", + "not found\n" + ] + } + ], + "source": [ + "# ── Exercise 5 ── your code here ─────────────────────────────────────────────\n", + "d = dragon.DDict()\n", + "\n", + "d[\"cat\"] = 1\n", + "d[\"dog\"] = 2\n", + "d[\"bird\"] = 3\n", + "\n", + "print(\"keys:\", list(d.keys())) # ['cat', 'dog', 'bird'] (order may vary)\n", + "print(\"dog in d:\", \"dog\" in d) # True\n", + "print(\"fish in d:\", \"fish\" in d) # False\n", + "\n", + "del d[\"cat\"]\n", + "print(\"len after delete:\", len(d)) # 2\n", + "\n", + "print(d.get(\"fish\", \"not found\")) # not found\n", + "\n", + "d.destroy()" + ] + }, + { + "cell_type": "markdown", + "id": "fe0ec91f-7c22-4898-88fd-d3a2176903fa", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "d = dragon.DDict()\n", + "\n", + "d[\"cat\"] = 1\n", + "d[\"dog\"] = 2\n", + "d[\"bird\"] = 3\n", + "\n", + "print(\"keys:\", list(d.keys())) # ['cat', 'dog', 'bird'] (order may vary)\n", + "print(\"dog in d:\", \"dog\" in d) # True\n", + "print(\"fish in d:\", \"fish\" in d) # False\n", + "\n", + "del d[\"cat\"]\n", + "print(\"len after delete:\", len(d)) # 2\n", + "\n", + "print(d.get(\"fish\", \"not found\")) # not found\n", + "\n", + "d.destroy()\n", + "```\n", + "\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "2de80c25-b9bd-4643-b24d-19a3530488fa", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Exercise 6 — Sharing a `DDict` across processes\n", + "\n", + "**Background:**\n", + "\n", + "A `DDict` is serializable (`pickle`-able). Passing a `DDict` to a distinct process does not cause an expensive transfer or serialization of any its contents (the keys or values). Instead, that process will share the **same** `DDict`, capable of being accessed or mutated by both child and parent.\n", + "\n", + "When instantiating a `DDict` object, you may specify the number of managers and nodes\n", + "on which the `DDict` will hold its distributed data. A `DDict` automatically starts\n", + "up a specified number of managers and an orchestrator (not shown in the picture) and\n", + "user client code interacts with these processes to transparently provide you with a\n", + "distributed key-value store that can scale to any size and automatically distributes\n", + "your data evenly across all the mangers.\n", + "\n", + "\n", + "```python\n", + "import dragon.native\n", + "\n", + "def worker(d, key, value):\n", + " d[key] = value # writes into the shared DDict\n", + " d[\"hello\"] = \"world\" # keys and values don't have to be passed to child proceses\n", + "\n", + "d = DDict(1, 1, 32 * 1024 * 1024) # DDict(managers_per_node, n_nodes, total_mem_bytes)\n", + "proc = dragon.native.Process(target=worker, args=(d, \"x\", 99)) # or use mp.Process\n", + "proc.start() # starts the process\n", + "proc.join() # waits for the process to exit\n", + "print(d[\"hello\"]) # and then prints world\n", + "d.destroy()\n", + "```\n", + "\n", + "This tells Dragon to instantiate and bring up a `DDict` with `managers_per_node * n_nodes` managers. The `total_mem_bytes` is spread across all the managers. For more information on creating `DDict` objects [see the DDict API](https://dragonhpc.github.io/dragon/doc/_build/html/ref/data/dragon.data.DDict.html#dragon.data.DDict). \n", + "\n", + "- `managers_per_node=1`, `n_nodes=1` → one manager, single node\n", + "- `total_mem_bytes=64*1024*1024` → 64 MB pool\n", + "- Remember to call `d.destroy()` to release resources when finished\n", + "\n", + "\n", + "**Your Task:**\n", + "\n", + "Write a function `count_words(label, text, d)` that splits `text` on spaces and stores\n", + "`d[label] = count` as a count of the words in the supplied text.\n", + "\n", + "Hint: Calling `s.split()` on a `str` object `s` in Python will return a list of all the space \n", + "separated *words* of the string.\n", + "\n", + "Then:\n", + "1. Create a `DDict` (2 managers, 1 node, 64 MB)\n", + "2. Launch **3 processes**, each calling `count_words` with a different song lyric\n", + "3. After all processes finish, print every key-value pair in the dictionary\n", + "4. Destroy the dictionary\n", + "\n", + "Song lyrics and associated labels to use:\n", + "```\n", + "(\"never gonna give you up never gonna let you down\", \"rick\")\n", + "(\"hello is it me you're looking for\", \"lionel\")\n", + "(\"cause I'll be rocking this party eight days a week\", \"beastie\")\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "098b4e60-63d1-4626-89f1-97ca635b9f13", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " rick: 10\n", + " beastie: 10\n", + " lionel: 7\n" + ] + } + ], + "source": [ + "# ── Exercise 6 ── your code here ─────────────────────────────────────────────\n", + "lyrics_and_labels = [\n", + " (\"never gonna give you up never gonna let you down\", \"rick\"),\n", + " (\"hello is it me you're looking for\", \"lionel\"),\n", + " (\"cause I'll be rocking this party eight days a week\", \"beastie\"),\n", + "]\n", + "\n", + "def count_words(label, text, d):\n", + " d[label] = len(text.split())\n", + "\n", + "d = dragon.DDict(2, 1, 64 * 1024 * 1024)\n", + "\n", + "procs = [dragon.native.Process(target=count_words, args=(l, t, d)) for t, l in lyrics_and_labels]\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "for k, v in d.items():\n", + " print(f\" {k}: {v}\")\n", + "\n", + "d.destroy()" + ] + }, + { + "cell_type": "markdown", + "id": "31d72121-7b06-4a8e-a7a4-dd1c8a6b0b6a", + "metadata": {}, + "source": [ + "
\n", + "▶ Show Answer\n", + "\n", + "```python\n", + "lyrics_and_labels = [\n", + " (\"never gonna give you up never gonna let you down\", \"rick\"),\n", + " (\"hello is it me you're looking for\", \"lionel\"),\n", + " (\"cause I'll be rocking this party eight days a week\", \"beastie\"),\n", + "]\n", + "\n", + "def count_words(label, text, d):\n", + " d[label] = len(text.split())\n", + "\n", + "d = dragon.DDict(2, 1, 64 * 1024 * 1024)\n", + "\n", + "procs = [dragon.native.Process(target=count_words, args=(l, t, d)) for t, l in lyrics_and_labels]\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "for k, v in d.items():\n", + " print(f\" {k}: {v}\")\n", + "\n", + "d.destroy()\n", + "```\n", + "\n", + "
\n", + "\n", + "**Synchronizing Updates to the Same Key in Parallel** \n", + "\n", + "Because multiple processes may attempt to store an updated value for a key which depends upon the\n", + "existing associated value, care must be taken because whichever process writes last wins. Depending\n", + "upon your code's needs, look at some of the functionality available on the `DDict` that has no\n", + "equivalent on the standard `dict` such as the atomic update `fetch_add` method to add values to\n", + "the existing stored value.\n", + "\n", + "For example, the `fetch_add` will operate atomically to add an integer value to a specified key/value pair. \n", + "\n", + "```python\n", + "def count_words(d, text):\n", + " for word, count in Counter(text.split()).items():\n", + " d.fetch_add(word, count)\n", + "```\n", + "\n", + "Using `fetch_add` to retrieve fetch/added values works in both Python and C++ Dragon APIs.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "28472c80", + "metadata": {}, + "source": [ + "---\n", + "\n", + "## Summary\n", + "\n", + "You used DragonHPC with familiar multiprocessing concepts to launch workers,\n", + "synchronize phases, communicate data, and run data-parallel map operations.\n", + "The same API patterns apply whether you run on one node or many.\n", + "\n", + "| Concept | API |\n", + "|---|---|\n", + "| Start Dragon backend | `mp.set_start_method(\"dragon\")` |\n", + "| Launch process | `mp.Process` and/or `dragon.native.Process(target=..., args=...)` |\n", + "| Wait for completion | `p.join()` |\n", + "| Mutual exclusion | `mp.Lock()` |\n", + "| Shared scalar | `mp.Value(...)` |\n", + "| Message passing | `mp.Queue()` |\n", + "| Phase synchronization | `mp.Barrier(n)` |\n", + "| Pool map | `mp.Pool(...).map(func, data)` |\n", + "| Multi-node placement check | `socket.gethostname()` |\n", + "\n", + "### What next?\n", + "\n", + "Try adapting these exercises to your own application by replacing toy data with\n", + "real workloads and scaling the worker count to your full Dragon allocation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "168332f4", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/distributed/solutions/13__dragonhpc_cuphoton__workflow__SOLUTION.ipynb b/notebooks/distributed/solutions/13__dragonhpc_cuphoton__workflow__SOLUTION.ipynb new file mode 100644 index 0000000..d32d9a1 --- /dev/null +++ b/notebooks/distributed/solutions/13__dragonhpc_cuphoton__workflow__SOLUTION.ipynb @@ -0,0 +1,1624 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "sc26-00", + "metadata": {}, + "source": [ + "# SC26 cuPhoton + DragonHPC Tutorial: From real sky images to GPU workers\n", + "\n", + "**Estimated time:** ~35 minutes \n", + "**Format:** Runnable demos as walkthru of major cuPhoton features and Dragon integration.\n", + "\n", + "A 35-minute walkthrough for the SC26 cuPhoton and Dragon session. We begin with\n", + "observational FITS pixels, add controlled changes, and follow the same data through\n", + "loading, registration, subtraction, candidate fitting, and classification.\n", + "At each distributed stage, compare an ordinary Python calculation with a visible\n", + "`cuphoton` command launched by Dragon. Finish by retaining the complete pipeline\n", + "inside each GPU worker and selecting the newer native CUDA C++ implementations.\n", + "\n", + "| Minutes | Question |\n", + "|---|---|\n", + "| 0–3 | What are the observations, injected signals, and independent work items? |\n", + "| 3–8 | How do FITS loading and WCS registration prepare a pair? |\n", + "| 8–13 | Can several workers subtract the same batch correctly? |\n", + "| 13–19 | What does a candidate fit tell us, and where does it fail? |\n", + "| 19–23 | How do we apply a small trained classifier across GPUs? |\n", + "| 23–31 | What changes when the stages share one persistent GPU worker? |\n", + "| 31–35 | Where do native implementations remove runtime kernel compilation? |\n", + "\n", + "The optional final section adds 5–7 minutes of automatic detection and uncertainty\n", + "analysis. Preparation, downloads, native builds, and training happen **before**\n", + "the session and are not described here. Run cells in order. All computation\n", + "requires Linux, CUDA 13, and the documented Python 3.12 environment.\n" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-01", + "metadata": {}, + "source": [ + "## 1. Choose a prepared workload and launch mode\n", + "\n", + "The background pixels come from real observations. The moving sources, sharp\n", + "artifacts, and labels are controlled injections; this is not a discovery of\n", + "astrophysical transients. The prepared model is trained on those injections and\n", + "is useful for demonstrating the execution path, not survey-ready classification.\n", + "Its held-out results and exact training provenance remain beside the checkpoint.\n", + "\n", + "Set `CUPHOTON_DEMO_ROOT` to the prepared directory and\n", + "`CUPHOTON_DEMO_WORKERS` for your allocation. The executable `%%bash` cells\n", + "use one host (`dragon -s`) by default. Each distributed stage also has complete\n", + "Slurm and SSH commands: replace the single-host command when using a cluster.\n", + "Multinode runs require identical absolute paths for the environment, inputs,\n", + "checkpoint, and outputs on every node.\n", + "\n", + "Python setup exports paths to the shell. The command cells contain the actual\n", + "CLI arguments, followed by Python cells that inspect their results. Each command\n", + "has a 10-minute timeout, a regular-file log, and a GNU `time` wall-clock record.\n", + "The timeout bounds the launcher; after a failure, inspect its owned workers\n", + "before rerunning. The cell shows the command and the last 40 log lines; the complete log remains\n", + "under `RUNS`. Linux Bash, GNU `time` (`/usr/bin/time`), and `timeout` are required." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-02", + "metadata": {}, + "outputs": [], + "source": [ + "# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\n", + "#\n", + "# SPDX-License-Identifier: Apache-2.0\n", + "\n", + "import gc\n", + "import json\n", + "import os\n", + "from pathlib import Path\n", + "import sys\n", + "import time\n", + "import uuid\n", + "\n", + "import numpy as np\n", + "import pandas as pd\n", + "import yaml\n", + "from IPython.display import display\n", + "\n", + "# A served output copy may live elsewhere; point it at the source checkout.\n", + "repo_override = os.environ.get(\"CUPHOTON_DEMO_REPO\")\n", + "search_roots = ([Path(repo_override).expanduser().resolve()] if repo_override\n", + " else [Path.cwd(), *Path.cwd().parents])\n", + "EXAMPLE = next((\n", + " candidate\n", + " for parent in search_roots\n", + " for candidate in (parent, parent / \"examples\" / \"dragon-imaging-pipeline\")\n", + " if (candidate / \"demo.py\").is_file()\n", + "), None)\n", + "if EXAMPLE is None:\n", + " raise RuntimeError(\n", + " \"Cannot find examples/dragon-imaging-pipeline/demo.py. Start Jupyter from \"\n", + " \"the cuPhoton checkout, or set CUPHOTON_DEMO_REPO to its absolute path.\"\n", + " )\n", + "REPO = EXAMPLE.parent.parent\n", + "sys.path.insert(0, str(EXAMPLE))\n", + "from demo import compare_arrays, image_grid, load_json, summarize_run\n", + "\n", + "DEMO = Path(os.environ.get(\n", + " \"CUPHOTON_DEMO_ROOT\", REPO / \"scratch\" / \"sc26-dragon\"\n", + ")).expanduser().resolve()\n", + "if not (DEMO / \"prepared.json\").is_file():\n", + " raise FileNotFoundError(\n", + " f\"No prepared.json in {DEMO}. Set CUPHOTON_DEMO_ROOT to the completed output \"\n", + " \"of prepare.py; see README.md.\"\n", + " )\n", + "prepared = load_json(DEMO / \"prepared.json\")\n", + "WORKERS = int(os.environ.get(\"CUPHOTON_DEMO_WORKERS\", \"1\"))\n", + "NODES = int(os.environ.get(\"CUPHOTON_DEMO_NODES\", \"2\"))\n", + "HOSTFILE = str(Path(os.environ.get(\n", + " \"CUPHOTON_DEMO_HOSTFILE\", DEMO / \"dragon-hosts.txt\"\n", + ")).expanduser().resolve())\n", + "assert WORKERS > 0 and NODES > 0\n", + "SESSION = f\"notebook-{time.strftime('%Y%m%d-%H%M%S')}-{uuid.uuid4().hex[:6]}\"\n", + "RUNS = DEMO / \"runs\" / SESSION\n", + "RUNS.mkdir(parents=True, exist_ok=False)\n", + "# %%bash inherits these environment variables; PATH selects this kernel's tools.\n", + "os.environ[\"PATH\"] = str(Path(sys.executable).parent) + os.pathsep + os.environ[\"PATH\"]\n", + "os.environ.update({\n", + " \"RUNS\": str(RUNS), \"WORKERS\": str(WORKERS), \"NODES\": str(NODES), \"HOSTFILE\": HOSTFILE,\n", + " \"PAIRS_MANIFEST\": prepared[\"pairs_manifest\"],\n", + " \"PIPELINE_MANIFEST\": prepared[\"pipeline_manifest\"],\n", + " \"NATIVE_PIPELINE_MANIFEST\": prepared[\"native_pipeline_manifest\"],\n", + " \"MODEL_DIR\": prepared[\"model_dir\"], \"DEMO_DATASET\": prepared[\"dataset_dir\"],\n", + " \"XFIT_INPUT\": prepared[\"xfit_input\"], \"XFIT_FEATURES\": prepared[\"xfit_feature_dir\"],\n", + "})\n", + "print({\"data\": str(DEMO), \"runs\": str(RUNS), \"workers\": WORKERS, \"python\": sys.executable})" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-03", + "metadata": {}, + "source": [ + "### Workload manifests and Dragon placement are separate inputs\n", + "\n", + "The YAML below lists **whole aligned image pairs** for xPois. It does not select\n", + "hosts. Dragon selects hosts through `-s`, a Slurm allocation, or a host file;\n", + "cuPhoton binds workers to the available GPUs. xFit and xScan divide their inputs\n", + "into candidate or minibatch chunks. The resident pipeline uses a JSON manifest\n", + "containing a checkpoint, numerical settings, and pairs with candidate positions.\n", + "\n", + "Dragon queues carry control messages and completion metadata. Workers read image\n", + "pixels from shared files. They do not send image arrays through those queues.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-04", + "metadata": {}, + "outputs": [], + "source": [ + "pairs_path = Path(prepared[\"pairs_manifest\"])\n", + "pairs_payload = yaml.safe_load(pairs_path.read_text())\n", + "pairs = pairs_payload[\"pairs\"]\n", + "print(yaml.safe_dump({\"schema\": pairs_payload[\"schema\"], \"pairs\": pairs[:2]},\n", + " sort_keys=False))\n", + "print(f\"{len(pairs)} independent pairs; at most {WORKERS} GPU workers\")\n", + "pipeline_path = Path(prepared[\"pipeline_manifest\"])\n", + "pipeline_payload = load_json(pipeline_path)\n", + "science_settings = pipeline_payload[\"configuration\"]\n", + "xpois_settings = science_settings[\"xpois\"]\n", + "print(\"Pipeline stages:\", \"xPois → xFit → xScan\")\n", + "print(\"Model:\", prepared[\"model_dir\"])\n", + "print(\"Numerical xFit policy:\", science_settings[\"xfit\"].get(\"solver_version\", \"v0.2\"))\n", + "\n", + "# The literal Bash examples below use these published demo settings.\n", + "assert xpois_settings[\"kernel_shape\"] == [15, 15]\n", + "assert xpois_settings[\"basis_sigmas\"] == [1.5] and xpois_settings[\"basis_degrees\"] == [0]\n", + "assert xpois_settings[\"background_degree\"] == 0 and not xpois_settings[\"flux_conserve\"]\n", + "assert science_settings[\"xfit\"][\"max_evaluations\"] == 400\n", + "assert prepared[\"config\"][\"batch_size\"] == 32\n", + "assert prepared[\"config\"][\"inference_split\"] == \"test\"" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-05", + "metadata": {}, + "source": [ + "### Check the numerical environment\n", + "\n", + "Torch is imported before CuPy initializes CUDA. The notebook owns one local GPU\n", + "for the Python comparisons; each Dragon launch starts a fresh coordinator and\n", + "binds its workers before their numerical libraries initialize. Explicit backends\n", + "make a missing GPU or native extension an error instead of silently changing the\n", + "calculation to a CPU run.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-06", + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "import cupy as cp\n", + "import cuphoton\n", + "from astropy.io import fits\n", + "from astropy.wcs import WCS\n", + "\n", + "assert torch.cuda.is_available()\n", + "torch.cuda.set_device(0)\n", + "cp.cuda.Device(0).use()\n", + "print({\"cuPhoton\": cuphoton.__version__, \"Torch\": torch.__version__,\n", + " \"CuPy\": cp.__version__, \"GPU\": torch.cuda.get_device_name(0),\n", + " \"CUDA runtime\": cp.cuda.runtime.runtimeGetVersion()})\n", + "first = pairs[0]\n", + "REFERENCE = Path(first[\"reference\"])\n", + "TARGET = Path(first[\"target\"])\n", + "HDU = int(first.get(\"reference_hdu\", 1))\n", + "data_manifest = load_json(prepared[\"data_manifest\"])\n", + "source_info = data_manifest[\"source\"]\n", + "display(pd.Series({key: source_info[key] for key in (\"title\", \"units\", \"credit\", \"dataset_doi\")}))\n", + "_ = image_grid([fits.getdata(data_manifest[\"crop\"][\"path\"], 1)],\n", + " [\"M51: original archival crop, 3.6 microns\"], columns=1)\n", + "\n", + "assert HDU == int(first.get(\"target_hdu\", 1)) == int(first.get(\"variance_hdu\", 1)) == 1\n", + "os.environ.update({\n", + " \"REFERENCE\": str(REFERENCE), \"TARGET\": str(TARGET),\n", + " \"VARIANCE\": first[\"variance\"], \"FIT_MASK\": first[\"fit_mask\"],\n", + "})" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-07", + "metadata": {}, + "source": [ + "## 2. Read and inspect real FITS pixels · 3–8 minutes\n", + "\n", + "xDR returns device arrays. The receipt below identifies the selected reader and\n", + "HDU; it is not proof of GPUDirect Storage or a hardware decompression engine.\n", + "Display downloads are intentional here. The later resident pipeline keeps its\n", + "large intermediate arrays on the worker GPU.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-08", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.core.fits_io import inspect_fits_images, read_fits_images\n", + "\n", + "planes = inspect_fits_images(REFERENCE)\n", + "display(pd.DataFrame([{\"hdu\": p.hdu, \"shape\": p.shape,\n", + " \"dtype\": str(p.dtype), \"compression\": p.compression}\n", + " for p in planes]))\n", + "reference_read = read_fits_images(REFERENCE, [HDU], reader=\"xdr\", device=True)\n", + "target_read = read_fits_images(TARGET, [HDU], reader=\"xdr\", device=True)\n", + "reference_device = reference_read.arrays[0]\n", + "target_device = target_read.arrays[0]\n", + "reference = cp.asnumpy(reference_device)\n", + "target = cp.asnumpy(target_device)\n", + "print(reference_read.metadata())\n", + "_ = image_grid([reference, target], [\"Observed background + reference injections\",\n", + " \"Controlled later scene\"],\n", + " columns=2, shared_scale=True)\n" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-09", + "metadata": {}, + "source": [ + "### Register onto an explicit reference WCS\n", + "\n", + "The preparation step also saves a shifted copy of the first target with a\n", + "corresponding WCS. xRep uses the WCS to return it to the reference pixel grid.\n", + "The aligned pair used below for subtraction was prepared in advance so every\n", + "worker reads the same immutable bytes.\n", + "\n", + "FITS loading and xRep are local operations in this walkthrough. xRep has a normal\n", + "CLI, but no `--executor dragon`; we do not manufacture a distributed equivalent.\n", + "Once registration is complete, Dragon distributes complete independent pairs.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-10", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xrep import BBox, Grid, build_stack_spec_from_fits, reproject_stack\n", + "\n", + "reprojection = data_manifest[\"reprojection\"]\n", + "RAW_TARGET = Path(reprojection[\"raw_target\"])\n", + "target_wcs = WCS(fits.getheader(REFERENCE, HDU), naxis=2)\n", + "height, width = reference.shape\n", + "sources, stack_spec = build_stack_spec_from_fits(\n", + " [REFERENCE, RAW_TARGET], hdu=HDU,\n", + " grid=Grid.from_wcs(target_wcs), output_bbox=BBox(0, 0, width, height),\n", + " interpolation=\"lanczos3\", mapping_grid_step=1,\n", + " fits_reader=\"xdr\", device=True,\n", + ")\n", + "aligned = reproject_stack(sources, stack_spec, backend=\"cupy\")\n", + "raw_target = fits.getdata(RAW_TARGET, HDU)\n", + "_ = image_grid([raw_target, aligned.images[1], target],\n", + " [\"Shifted detector pixels\", \"xRep: reference grid\", \"Prepared aligned target\"],\n", + " shared_scale=True)\n", + "\n", + "os.environ[\"RAW_TARGET\"] = str(RAW_TARGET)\n", + "# A local view makes the known three-pixel shift visible at notebook scale.\n", + "anchor = data_manifest[\"pairs\"][0][\"candidates\"][0]\n", + "x, y = int(anchor[\"x\"]), int(anchor[\"y\"])\n", + "view = np.s_[y - 25:y + 26, x - 25:x + 26]\n", + "_ = image_grid([raw_target[view], aligned.images[1][view],\n", + " (raw_target - aligned.images[1])[view]],\n", + " [\"Before WCS registration: 51 px\", \"After WCS registration: 51 px\",\n", + " \"Before minus after: own scale\"], diverging=[False, False, True])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-11", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xrep-cupy.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xrep-cupy.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " cuphoton xrep reproject-stack \\\n", + " --inputs \"$REFERENCE,$RAW_TARGET\" --hdu 1 \\\n", + " --target-wcs \"$REFERENCE\" --target-hdu 1 \\\n", + " --backend cupy --fits-reader xdr --interpolation lanczos3 \\\n", + " --mapping-grid-step 1 --write-fits \\\n", + " --output-dir \"$RUNS\" --name xrep-cupy >\"$RUNS/xrep-cupy.log\" 2>&1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-12", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xrep-cupy: launcher-to-exit seconds:\", float((RUNS / \"xrep-cupy.elapsed\").read_text()))\n", + "cli_aligned = np.load(RUNS / \"xrep-cupy\" / \"artifacts\" / \"reprojected_stack.npy\")\n", + "print(compare_arrays(aligned.images, cli_aligned, label=\"xRep Python / CLI\"))" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-13", + "metadata": {}, + "source": [ + "## 3. Match the PSF and subtract · 8–13 minutes\n", + "\n", + "Fit a kernel and differential background so the reference resembles the target,\n", + "then form **target − matched reference**. The injected changing sources are\n", + "excluded by the prepared fit mask. The basis widths are inputs, not parameters\n", + "optimized by this solve. Both routes use the same basis, mask and variance.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-14", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xpois import GaussianBasisComponent, solve_constant_kernel_device\n", + "\n", + "components = [GaussianBasisComponent(sigma=s, degree=d)\n", + " for s, d in zip(xpois_settings[\"basis_sigmas\"],\n", + " xpois_settings[\"basis_degrees\"], strict=True)]\n", + "variance = fits.getdata(first[\"variance\"], int(first.get(\"variance_hdu\", 1)))\n", + "fit_mask_path = Path(first[\"fit_mask\"])\n", + "fit_mask = (np.load(fit_mask_path, allow_pickle=False) if fit_mask_path.suffix == \".npy\"\n", + " else fits.getdata(fit_mask_path, 1)).astype(bool)\n", + "solve_options = dict(\n", + " kernel_shape=tuple(xpois_settings[\"kernel_shape\"]),\n", + " variance=cp.asarray(variance), fit_mask=cp.asarray(fit_mask),\n", + " background_degree=xpois_settings[\"background_degree\"],\n", + " flux_conserve=xpois_settings[\"flux_conserve\"],\n", + ")\n", + "subtraction = solve_constant_kernel_device(\n", + " reference_device, target_device, components, backend=\"cupy\", **solve_options\n", + ")\n", + "difference = cp.asnumpy(subtraction.residual)\n", + "print({\"backend\": subtraction.backend, \"fit_pixels\": int(subtraction.fit_pixel_count),\n", + " \"chi2\": float(subtraction.chi2), \"kernel_sum\": float(subtraction.kernel.sum())})\n", + "_ = image_grid([reference, target, cp.asnumpy(subtraction.matched)],\n", + " [\"Reference\", \"Target\", \"Matched reference\"], shared_scale=True)\n", + "_ = image_grid([cp.asnumpy(subtraction.kernel), difference, fit_mask],\n", + " [\"Fitted kernel\", \"Difference: target − matched\", \"Kernel-fit pixels\"],\n", + " diverging=[False, True, False])\n" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-15", + "metadata": {}, + "source": [ + "### Run the same subtraction over the image-pair batch with Dragon\n", + "\n", + "One pair is one item; one GPU worker can process several items. Increasing the\n", + "number of candidates within a single pair does not create more xPois workers.\n", + "The single-host cell and the cluster alternatives use the same numerical arguments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-16", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xpois-dragon.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xpois-dragon.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xpois fit-batch \\\n", + " --executor dragon --backend cupy --manifest \"$PAIRS_MANIFEST\" --fits-reader xdr \\\n", + " --kernel-height 15 --kernel-width 15 --basis-sigmas 1.5 --basis-degrees 0 \\\n", + " --background-degree 0 --output-dir \"$RUNS\" --name xpois-dragon \\\n", + " --max-workers \"$WORKERS\" >\"$RUNS/xpois-dragon.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-17", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xpois fit-batch \\\n", + " --executor dragon --backend cupy --manifest \"$PAIRS_MANIFEST\" --fits-reader xdr \\\n", + " --kernel-height 15 --kernel-width 15 --basis-sigmas 1.5 --basis-degrees 0 \\\n", + " --background-degree 0 --output-dir \"$RUNS\" --name xpois-dragon \\\n", + " --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xpois fit-batch \\\n", + " --executor dragon --backend cupy --manifest \"$PAIRS_MANIFEST\" --fits-reader xdr \\\n", + " --kernel-height 15 --kernel-width 15 --basis-sigmas 1.5 --basis-degrees 0 \\\n", + " --background-degree 0 --output-dir \"$RUNS\" --name xpois-dragon \\\n", + " --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-18", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xpois-dragon: launcher-to-exit seconds:\", float((RUNS / \"xpois-dragon.elapsed\").read_text()))\n", + "_ = summarize_run(RUNS / \"xpois-dragon\")\n", + "cli_difference = np.load(RUNS / \"xpois-dragon\" / \"items\" / first[\"id\"] /\n", + " \"artifacts\" / \"residual.npy\")\n", + "print(compare_arrays(difference, cli_difference, label=\"xPois Python / Dragon\"))\n", + "_ = image_grid([difference, cli_difference, difference - cli_difference],\n", + " [\"Python difference\", \"Dragon difference\", \"Numerical difference\"],\n", + " diverging=True)" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-19", + "metadata": {}, + "source": [ + "## 4. Fit candidate stamps · 13–19 minutes\n", + "\n", + "The prepared candidates contain known injections and controls; this section is\n", + "not a source detector. Each input stamp and candidate ID is frozen in an NPZ\n", + "file derived from the same subtraction settings. Both routes derive the same\n", + "initial guesses from those stamps. A Gaussian dipole models\n", + "a positive and a negative lobe. A sharp artifact, blank stamp, or single brightening\n", + "source can violate that model even when the optimizer returns parameters.\n", + "\n", + "Keep **solver policy v0.2**, float64, and candidate chunk size identical in the\n", + "Python and Dragon runs. Inspect convergence and uncertainty validity separately.\n", + "The final optional chain adds detection, quality flags, and single-source refits.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-20", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xfit import GaussianDipoleModel, LMConfig, fit_dipoles\n", + "from cuphoton.xfit.io import load_xfit_dataset\n", + "\n", + "xfit_data = load_xfit_dataset(prepared[\"xfit_input\"], model=\"gaussian\", mode=\"difference\")\n", + "CHUNK = int(prepared.get(\"config\", {}).get(\"chunk_size\", 32))\n", + "MAX_EVALUATIONS = int(science_settings[\"xfit\"][\"max_evaluations\"])\n", + "assert CHUNK == 32 and MAX_EVALUATIONS == 400\n", + "fit_config = LMConfig(solver_version=\"v0.2\", max_evaluations=MAX_EVALUATIONS)\n", + "python_fits = [\n", + " fit_dipoles(\n", + " xfit_data.images[start:start + CHUNK], model=\"gaussian\", mode=\"difference\",\n", + " initial=None if xfit_data.initial is None else xfit_data.initial[start:start + CHUNK],\n", + " backend=\"cupy\", config=fit_config,\n", + " )\n", + " for start in range(0, xfit_data.batch_size, CHUNK)\n", + "]\n", + "python_parameters = np.concatenate([result.parameters for result in python_fits])\n", + "python_converged = np.concatenate([result.converged for result in python_fits])\n", + "python_valid = np.concatenate([result.uncertainty_valid for result in python_fits])\n", + "print({\"candidates\": xfit_data.batch_size, \"converged\": int(python_converged.sum()),\n", + " \"valid_covariance\": int(python_valid.sum()), \"solver_policy\": \"v0.2\"})\n", + "model = GaussianDipoleModel(xfit_data.images.shape[-2:], dtype=np.float64)\n", + "fit_model = np.asarray(model.evaluate(python_fits[0].parameters, mode=\"difference\"))\n", + "_ = image_grid([xfit_data.images[0], fit_model[0]],\n", + " [\"Difference stamp\", \"Gaussian dipole model\"],\n", + " columns=2, diverging=True, shared_scale=True)\n", + "fit_residual = -python_fits[0].residuals[0] # xFit returns model minus data.\n", + "stamp_peak = float(np.max(np.abs(xfit_data.images[0])))\n", + "residual_peak = float(np.max(np.abs(fit_residual)))\n", + "print({\"residual_peak_image_units\": residual_peak,\n", + " \"residual_peak_over_stamp_peak\": residual_peak / max(stamp_peak, 1e-30)})\n", + "_ = image_grid([fit_residual],\n", + " [f\"Data − model; own scale; peak/stamp = {residual_peak / max(stamp_peak, 1e-30):.2%}\"],\n", + " columns=1, diverging=True)" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-21", + "metadata": {}, + "source": [ + "### Recover the known motion, then compare executors\n", + "\n", + "The stamp and model share a color range. The residual has its own labeled range\n", + "so a good fit's remaining structure stays visible; saturated residual colors do\n", + "not mean errors as large as the source.\n", + "\n", + "The injected old and new centers supply an independent recovery check. xFit\n", + "positions are relative to the center of each odd-sized stamp. Add the candidate's\n", + "full-image `(x, y)` before comparing the positive lobe to the new position and the\n", + "negative lobe to the old position. These errors are in **pixels**. We report every\n", + "injected mover, including failed fits; backend agreement alone would not show\n", + "whether either calculation recovered the signal. Circular lobes can leave\n", + "orientation poorly constrained, so inspect center error separately from the\n", + "full covariance's validity flag.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-22", + "metadata": {}, + "outputs": [], + "source": [ + "truth_by_id = {candidate[\"candidate_id\"]: candidate\n", + " for item in data_manifest[\"pairs\"] for candidate in item[\"candidates\"]}\n", + "recovery_rows = []\n", + "for index, candidate_id in enumerate(xfit_data.candidate_id):\n", + " candidate = truth_by_id[str(candidate_id)]\n", + " if candidate[\"kind\"] != \"moving_source\":\n", + " continue\n", + " origin = np.array([candidate[\"x\"], candidate[\"y\"]], dtype=float)\n", + " fitted_new = python_parameters[index, 4:6] + origin\n", + " fitted_old = python_parameters[index, 6:8] + origin\n", + " true_new = np.array(candidate[\"truth\"][\"target_xy\"])\n", + " true_old = np.array(candidate[\"truth\"][\"reference_xy\"])\n", + " recovery_rows.append({\n", + " \"candidate_id\": str(candidate_id),\n", + " \"positive_center_error_px\": float(np.linalg.norm(fitted_new - true_new)),\n", + " \"negative_center_error_px\": float(np.linalg.norm(fitted_old - true_old)),\n", + " \"motion_vector_error_px\": float(np.linalg.norm((fitted_new - fitted_old) - (true_new - true_old))),\n", + " \"converged\": bool(python_converged[index]),\n", + " \"uncertainty_valid\": bool(python_valid[index]),\n", + " })\n", + "recovery = pd.DataFrame(recovery_rows)\n", + "display(recovery.head(12))\n", + "error_columns = [\"positive_center_error_px\", \"negative_center_error_px\", \"motion_vector_error_px\"]\n", + "display(recovery[error_columns].describe(percentiles=[0.5, 0.9]))\n", + "# A small histogram table includes failed fits rather than filtering them away.\n", + "bins = [0, 0.1, 0.25, 0.5, 1, 2, np.inf]\n", + "histogram = {\"error_px\": [\"0–0.1\", \"0.1–0.25\", \"0.25–0.5\", \"0.5–1\", \"1–2\", \"2+\"]}\n", + "for column in error_columns:\n", + " histogram[column] = np.histogram(recovery[column], bins=bins)[0]\n", + "display(pd.DataFrame(histogram))\n", + "print({\"moving_sources\": len(recovery), \"converged\": int(recovery.converged.sum()),\n", + " \"valid_covariance\": int(recovery.uncertainty_valid.sum()),\n", + " \"nonfinite_error_rows\": int((~np.isfinite(recovery[error_columns])).any(axis=1).sum())})\n", + "recovery.to_csv(RUNS / \"injection-recovery.csv\", index=False)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-23", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xfit-dragon.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xfit-dragon.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend cupy --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-dragon\" --max-workers \"$WORKERS\" >\"$RUNS/xfit-dragon.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-24", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend cupy --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-dragon\" --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend cupy --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-dragon\" --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-25", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xfit-dragon: launcher-to-exit seconds:\", float((RUNS / \"xfit-dragon.elapsed\").read_text()))\n", + "fit_table = pd.read_parquet(RUNS / \"xfit-dragon\" / \"scientific\" / \"fits.parquet\")\n", + "assert fit_table[\"candidate_id\"].tolist() == xfit_data.candidate_id.tolist()\n", + "print(compare_arrays(python_parameters,\n", + " fit_table[list(python_fits[0].parameter_names)].to_numpy(),\n", + " label=\"xFit Python / Dragon parameters\"))\n", + "np.testing.assert_array_equal(python_converged, fit_table[\"converged\"].to_numpy())\n", + "display(fit_table[[\"candidate_id\", \"converged\", \"uncertainty_valid\"]].head(8))\n", + "_ = summarize_run(RUNS / \"xfit-dragon\")" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-26", + "metadata": {}, + "source": [ + "## 5. Score the candidate batch · 19–23 minutes\n", + "\n", + "The small triplet-plus-xFit model was trained in preparation, with disjoint\n", + "training, validation and held-out inputs. The table below reports that particular\n", + "injection experiment. Training runs 20 GPU epochs and retains the checkpoint\n", + "with the best validation ROC AUC; the qualified run retained **epoch 1**. The\n", + "history below shows the later overfitting. Scores are uncalibrated and narrowly\n", + "clustered near 0.5; inspect their ranking and logits as well as the validation\n", + "threshold. They are not probabilities that an arbitrary sky source is real.\n", + "\n", + "The Python and CLI treatments load the same checkpoint and feature schema, and\n", + "preserve the same inference minibatches. The stock dataset inference API is used\n", + "here so both treatments use the same host sigmoid and sample ordering." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-27", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xscan.dataset import StampDataset\n", + "from cuphoton.xscan.training import load_model_from_checkpoint, predict_dataset\n", + "\n", + "training_summary = load_json(Path(prepared[\"model_dir\"]) / \"summary.json\")\n", + "training_history = pd.DataFrame(load_json(Path(prepared[\"model_dir\"]) / \"history.json\"))\n", + "best_epoch = int(training_summary[\"early_stopping\"][\"best_epoch\"])\n", + "print({\"epochs_completed\": training_summary[\"epochs_completed\"],\n", + " \"retained_best_epoch\": best_epoch, \"selection_metric\": training_summary[\"best_metric\"]})\n", + "# The beginning, best epoch, and end show the training trend without a long table.\n", + "history_rows = training_history[\"epoch\"].isin([1, 2, 3, best_epoch, 10, 20])\n", + "display(training_history.loc[history_rows, [\"epoch\", \"train_loss\", \"val_loss\", \"val_roc_auc\"]])\n", + "\n", + "metrics = prepared[\"metrics\"]\n", + "if isinstance(metrics, str):\n", + " metrics = load_json(metrics)\n", + "metric_columns = [\"sample_count\", \"balanced_accuracy\", \"roc_auc\", \"precision\", \"recall\", \"fp\", \"fn\"]\n", + "display(pd.DataFrame(metrics[\"splits\"]).T.loc[[\"val\", \"test\"], metric_columns])\n", + "print(\"Threshold selected on validation:\", metrics[\"decision_threshold\"])\n", + "BATCH = int(prepared.get(\"config\", {}).get(\"batch_size\", 32))\n", + "SPLIT = prepared.get(\"config\", {}).get(\"inference_split\", \"all\")\n", + "assert BATCH == 32 and SPLIT == \"test\"\n", + "classifier, checkpoint, performance = load_model_from_checkpoint(\n", + " Path(prepared[\"model_dir\"]), device=torch.device(\"cuda:0\")\n", + ")\n", + "dataset = StampDataset(\n", + " Path(prepared[\"dataset_dir\"]), input_mode=\"triplet\", split=SPLIT,\n", + " xfit_feature_dir=Path(prepared[\"xfit_feature_dir\"]),\n", + " xfit_feature_names=tuple(classifier.xfit_feature_names),\n", + ")\n", + "python_predictions = predict_dataset(\n", + " model=classifier, dataset=dataset, batch_size=BATCH,\n", + " device=torch.device(\"cuda:0\"), performance=performance,\n", + ")\n", + "# Fixed first two examples of each injected class, independent of model score.\n", + "prediction_rows = python_predictions[\"metadata_rows\"]\n", + "contact_positions = [\n", + " position\n", + " for kind in (\"moving_source\", \"detector_artifact\", \"unchanged_control\")\n", + " for position in [i for i, row in enumerate(prediction_rows) if row[\"kind\"] == kind][:2]\n", + "]\n", + "contact_samples = [int(dataset.indices[position]) for position in contact_positions]\n", + "_ = image_grid([dataset.difference[sample] for sample in contact_samples],\n", + " [prediction_rows[position][\"kind\"] + chr(10) + prediction_rows[position][\"candidate_id\"]\n", + " for position in contact_positions], diverging=True)\n", + "# Rank 1 is the highest score across the entire demonstration candidate batch.\n", + "score_ranks = pd.Series(python_predictions[\"probabilities\"]).rank(method=\"min\", ascending=False)\n", + "print({\"uncalibrated_score_min\": float(np.min(python_predictions[\"probabilities\"])),\n", + " \"uncalibrated_score_max\": float(np.max(python_predictions[\"probabilities\"]))})\n", + "# Truth is known from injection; scores, ranks and decisions come from the model.\n", + "threshold = float(metrics[\"decision_threshold\"])\n", + "display(pd.DataFrame([\n", + " {\"candidate_id\": prediction_rows[position][\"candidate_id\"],\n", + " \"injected_kind\": prediction_rows[position][\"kind\"],\n", + " \"injected_label\": int(python_predictions[\"labels\"][position]),\n", + " \"model_logit\": float(python_predictions[\"logits\"][position]),\n", + " \"score_rank\": int(score_ranks.iloc[position]),\n", + " \"uncalibrated_score\": float(python_predictions[\"probabilities\"][position]),\n", + " \"model_decision\": bool(python_predictions[\"probabilities\"][position] >= threshold)}\n", + " for position in contact_positions\n", + "]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-28", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xscan-dragon.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xscan-dragon.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xscan infer-real-bogus \\\n", + " --executor dragon --run-dir \"$MODEL_DIR\" --dataset-dir \"$DEMO_DATASET\" \\\n", + " --split test --batch-size 32 --num-workers 0 --task-batches 1 \\\n", + " --use-xfit-features --xfit-feature-dir \"$XFIT_FEATURES\" \\\n", + " --output-dir \"$RUNS/xscan-dragon\" --max-workers \"$WORKERS\" >\"$RUNS/xscan-dragon.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-29", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xscan infer-real-bogus \\\n", + " --executor dragon --run-dir \"$MODEL_DIR\" --dataset-dir \"$DEMO_DATASET\" \\\n", + " --split test --batch-size 32 --num-workers 0 --task-batches 1 \\\n", + " --use-xfit-features --xfit-feature-dir \"$XFIT_FEATURES\" \\\n", + " --output-dir \"$RUNS/xscan-dragon\" --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xscan infer-real-bogus \\\n", + " --executor dragon --run-dir \"$MODEL_DIR\" --dataset-dir \"$DEMO_DATASET\" \\\n", + " --split test --batch-size 32 --num-workers 0 --task-batches 1 \\\n", + " --use-xfit-features --xfit-feature-dir \"$XFIT_FEATURES\" \\\n", + " --output-dir \"$RUNS/xscan-dragon\" --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-30", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xscan-dragon: launcher-to-exit seconds:\", float((RUNS / \"xscan-dragon.elapsed\").read_text()))\n", + "scientific = RUNS / \"xscan-dragon\" / \"scientific\"\n", + "np.testing.assert_array_equal(dataset.indices, np.load(scientific / \"sample_index.npy\"))\n", + "print(compare_arrays(python_predictions[\"logits\"], np.load(scientific / \"logits.npy\"),\n", + " label=\"xScan Python / Dragon logits\"))\n", + "print(compare_arrays(python_predictions[\"probabilities\"],\n", + " np.load(scientific / \"probabilities.npy\"),\n", + " label=\"xScan Python / Dragon probabilities\"))\n", + "_ = summarize_run(RUNS / \"xscan-dragon\")" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-31", + "metadata": {}, + "source": [ + "## 6. Keep the whole pipeline in one worker · 23–31 minutes\n", + "\n", + "The separate commands above make stage boundaries inspectable. The resident\n", + "pipeline processes an entire pair through xPois → xFit → feature conversion →\n", + "xScan in one worker, reusing the model and CUDA context. CuPy and Torch share\n", + "compatible device buffers via DLPack. It writes compact results instead of full\n", + "intermediate images.\n", + "\n", + "This pipeline takes already aligned pairs and supplied candidate coordinates.\n", + "It does not perform xRep or detection, and its xFit feature contract is unmasked,\n", + "unweighted Gaussian fitting. Its xPois variance and mask do not become xFit\n", + "weights. The optional survey chain below has a different, richer contract.\n", + "\n", + "Preparation fits each full corpus in one local xFit call. The live xFit comparison\n", + "uses 32-candidate chunks, and the resident pipeline refits each 32-candidate pair.\n", + "Poorly constrained artifact fits can vary with batch size. Standalone inference\n", + "uses the saved feature bundle, so its scores are a separate comparison from\n", + "resident feature recomputation.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-32", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xscan.device_pipeline import (\n", + " DeviceWorkerContext, decode_device_pipeline_evidence, run_device_pipeline_item,\n", + ")\n", + "from cuphoton.xscan.pipeline_executor import load_pipeline_manifest\n", + "\n", + "pipeline_config, pipeline_items = load_pipeline_manifest(pipeline_path)\n", + "context = DeviceWorkerContext.initialize(pipeline_config)\n", + "try:\n", + " # Readable API example: one pair. Full-corpus timings use the CLI below.\n", + " started = time.perf_counter()\n", + " api_result = run_device_pipeline_item(pipeline_items[0], context)\n", + " api_item_seconds = time.perf_counter() - started\n", + "finally:\n", + " context.producer_stream.synchronize()\n", + " context.consumer_stream.synchronize()\n", + "print({\"context_setup_sec\": context.load_seconds, \"one_item_sec\": api_item_seconds})\n", + "display(pd.json_normalize(api_result.transfers.to_payload()))\n", + "# Release the local model before launching fresh workers on the same GPU.\n", + "del context, classifier\n", + "cp.get_default_memory_pool().free_all_blocks()\n", + "torch.cuda.empty_cache()\n", + "gc.collect()\n" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-33", + "metadata": {}, + "source": [ + "### Run the full corpus locally, then with Dragon\n", + "\n", + "The local command is `cuphoton xscan run-pipeline --executor processes`. It uses\n", + "one persistent process on one explicitly visible GPU. Both CLI treatments run\n", + "all prepared pairs with one warmup and three measured rounds; the earlier Python\n", + "cell runs one representative pair solely to expose the API and data flow.\n", + "\n", + "The local child uses the first GPU already visible to the notebook, preserving\n", + "an inherited scheduler mask. Dragon can use the configured worker count across\n", + "its selected hosts. This compares execution arrangements for the same corpus;\n", + "keep worker count and device differences visible alongside timing results.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-34", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "# Use the notebook's first visible GPU, preserving an inherited scheduler mask.\n", + "export CUDA_VISIBLE_DEVICES=\"${CUDA_VISIBLE_DEVICES:-0}\"\n", + "export CUDA_VISIBLE_DEVICES=\"${CUDA_VISIBLE_DEVICES%%,*}\"\n", + "trap 'tail -n 40 \"$RUNS/pipeline-local.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/pipeline-local.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " cuphoton xscan run-pipeline \\\n", + " --executor processes --workers-per-gpu 1 --manifest \"$PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-local --warmup-rounds 1 --measure-rounds 3 >\"$RUNS/pipeline-local.log\" 2>&1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-35", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"pipeline-local: launcher-to-exit seconds:\", float((RUNS / \"pipeline-local.elapsed\").read_text()))\n", + "_ = summarize_run(RUNS / \"pipeline-local\")\n", + "local_rounds = sorted((RUNS / \"pipeline-local\" / \"rounds\").glob(\"measure-*\"))\n", + "assert len(local_rounds) == 3\n", + "local_by_id = {\n", + " item.item_id: load_json(local_rounds[-1] / \"items\" / item.item_id / \"summary.json\")\n", + " for item in pipeline_items\n", + "}\n", + "api_arrays = decode_device_pipeline_evidence(api_result.scientific_evidence, config=pipeline_config)\n", + "cli_arrays = decode_device_pipeline_evidence(\n", + " local_by_id[api_result.item_id][\"scientific_evidence\"], config=pipeline_config,\n", + ")\n", + "for name in api_arrays:\n", + " compare_arrays(api_arrays[name], cli_arrays[name], rtol=1e-8, atol=1e-10,\n", + " label=f\"Direct API / local CLI: {name}\")\n", + "\n", + "print(f\"Direct API / local CLI: {len(api_arrays)} scientific arrays passed\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-36", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/pipeline-dragon.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/pipeline-dragon.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-dragon \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\" >\"$RUNS/pipeline-dragon.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-37", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-dragon \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-dragon \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-38", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"pipeline-dragon: launcher-to-exit seconds:\", float((RUNS / \"pipeline-dragon.elapsed\").read_text()))\n", + "_ = summarize_run(RUNS / \"pipeline-dragon\")" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-39", + "metadata": {}, + "source": [ + "### Verify the science and explain the timers\n", + "\n", + "Compare the compact scientific evidence for every item, rather than accepting\n", + "only a successful launcher exit. The root execution summary also verifies worker\n", + "placement, output records and shutdown. Small images can spend more time in\n", + "launch and I/O than in arithmetic; the notebook does not promise a speedup.\n", + "\n", + "The one-item Python illustration excludes process startup and result-file\n", + "publication. Use the full-corpus CLI runs for timing comparisons. Their batch\n", + "timers include dispatch, input reads, computation and worker outputs;\n", + "validation/merging follows. Show those boundaries and actual worker counts\n", + "alongside launcher-to-exit time. Do not divide unmatched timers to claim\n", + "acceleration.\n", + "\n", + "GNU `time` writes each complete command's elapsed seconds to `.elapsed`;\n", + "stdout/stderr go to `.log`. The Python cell reads that timing separately\n", + "from the worker and batch timers reported by cuPhoton. The final local round is\n", + "the baseline; skip its eight self-comparisons, leaving 40 item-round comparisons." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-40", + "metadata": {}, + "outputs": [], + "source": [ + "parity = []\n", + "for treatment in (\"pipeline-local\", \"pipeline-dragon\"):\n", + " round_dirs = sorted((RUNS / treatment / \"rounds\").glob(\"measure-*\"))\n", + " assert len(round_dirs) == 3\n", + " for round_dir in round_dirs:\n", + " if treatment == \"pipeline-local\" and round_dir == local_rounds[-1]:\n", + " continue # This is the reference round, not an independent comparison.\n", + " for item in pipeline_items:\n", + " observed_payload = load_json(round_dir / \"items\" / item.item_id / \"summary.json\")\n", + " local = local_by_id[item.item_id]\n", + " assert observed_payload[\"item_id\"] == local[\"item_id\"]\n", + " expected = decode_device_pipeline_evidence(local[\"scientific_evidence\"],\n", + " config=pipeline_config)\n", + " observed = decode_device_pipeline_evidence(observed_payload[\"scientific_evidence\"],\n", + " config=pipeline_config)\n", + " for name in expected:\n", + " compare_arrays(expected[name], observed[name], rtol=1e-8, atol=1e-10,\n", + " label=f\"{item.item_id}: {name}\")\n", + " parity.append({\"treatment\": treatment, \"round\": round_dir.name,\n", + " \"item\": item.item_id, \"arrays_checked\": len(expected), \"parity\": \"passed\"})\n", + "display(pd.DataFrame(parity))\n", + "(RUNS / \"pipeline-parity.json\").write_text(json.dumps(parity, indent=2) + chr(10))" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-41", + "metadata": {}, + "source": [ + "### Saved features versus features recomputed by the resident pipeline\n", + "\n", + "This is a diagnostic across the two feature-generation batch shapes described\n", + "above, separate from the executor-parity checks. Report score differences and\n", + "decision flips without requiring the two feature bundles to be identical." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-42", + "metadata": {}, + "outputs": [], + "source": [ + "resident_scores = {}\n", + "for item in pipeline_items:\n", + " arrays = decode_device_pipeline_evidence(\n", + " local_by_id[item.item_id][\"scientific_evidence\"], config=pipeline_config,\n", + " )\n", + " for candidate, score in zip(item.candidates, arrays[\"xscan.probabilities\"], strict=True):\n", + " resident_scores[str(candidate.candidate_id)] = float(score)\n", + "score_comparison = pd.DataFrame({\n", + " \"candidate_id\": [row[\"candidate_id\"] for row in prediction_rows],\n", + " \"kind\": [row[\"kind\"] for row in prediction_rows],\n", + " \"saved_feature_score\": python_predictions[\"probabilities\"],\n", + " \"resident_score\": [resident_scores[row[\"candidate_id\"]] for row in prediction_rows],\n", + "})\n", + "score_comparison[\"absolute_delta\"] = abs(score_comparison.saved_feature_score - score_comparison.resident_score)\n", + "score_comparison[\"decision_flip\"] = ((score_comparison.saved_feature_score >= threshold)\n", + " != (score_comparison.resident_score >= threshold))\n", + "display(score_comparison.groupby(\"kind\").agg(\n", + " candidates=(\"candidate_id\", \"size\"), maximum_score_delta=(\"absolute_delta\", \"max\"),\n", + " decision_flips=(\"decision_flip\", \"sum\"),\n", + "))\n", + "score_comparison.to_csv(RUNS / \"saved-vs-resident-scores.csv\", index=False)" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-43", + "metadata": {}, + "source": [ + "## 7. Select native CUDA C++ implementations · 31–35 minutes\n", + "\n", + "The numerical backend and the process executor are independent choices. Native\n", + "implementations use precompiled CUDA kernels and CUDA libraries while retaining\n", + "CuPy-owned buffers. `auto` does not select these paths automatically.\n", + "\n", + "| Operation | Explicit selection | Boundary |\n", + "|---|---|---|\n", + "| xDR compressed-pixel restoration | `postprocess=\"native\"` | GZIP_1/GZIP_2 tiles; reader selection is separate |\n", + "| xRep mapping expansion and reprojection | `backend=\"native\"` | Coarse WCS evaluation remains on CPU |\n", + "| xPois source selection and constant-kernel solve | `backend=\"native\"` | Local CLI/device API; batch CLI currently accepts CuPy, Numba-CUDA or cuTile |\n", + "| xFit Gaussian preparation, LM, and covariance | `backend=\"native\"` | Analytic Gaussian models; also works inside Dragon workers |\n", + "\n", + "The existing resident pipeline can select **native xFit**; its xPois stage remains\n", + "CuPy. No claim is made that the entire pipeline, plotting, or all dependencies\n", + "avoid JIT compilation. Compare the same model, precision, solver policy and data.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-44", + "metadata": {}, + "outputs": [], + "source": [ + "compressed_target = Path(data_manifest[\"pairs\"][0][\"target_compressed\"])\n", + "gzip_cupy = read_fits_images(compressed_target, [HDU], reader=\"xdr\", device=True,\n", + " xdr_options={\"postprocess\": \"fused\"})\n", + "gzip_native = read_fits_images(compressed_target, [HDU], reader=\"xdr\", device=True,\n", + " xdr_options={\"postprocess\": \"native\"})\n", + "print(compare_arrays(cp.asnumpy(gzip_cupy.arrays[0]), cp.asnumpy(gzip_native.arrays[0]),\n", + " rtol=0, atol=0, label=\"xDR lossless native restoration\"))\n", + "print(compare_arrays(target, cp.asnumpy(gzip_native.arrays[0]),\n", + " rtol=0, atol=0, label=\"xDR native decoded / original uncompressed pixels\"))\n", + "\n", + "os.environ[\"COMPRESSED_TARGET\"] = str(compressed_target)" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-45", + "metadata": {}, + "source": [ + "The xDR CLI counterpart is a decoding benchmark; it does not export pixel\n", + "arrays. The API cell above checks exact decoded values. `--skip-gds-read` skips\n", + "only the separate raw-storage phase, not FITS decoding." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-46", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xdr-native.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xdr-native.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " cuphoton xdr benchmark-fits \"$COMPRESSED_TARGET\" --hdu-indices 1 \\\n", + " --iterations 1 --native-plan-threads 1 --native-read-threads 1 \\\n", + " --xdr-postprocess native --skip-gds-read --output-json \"$RUNS/xdr-native.json\" >\"$RUNS/xdr-native.log\" 2>&1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-47", + "metadata": {}, + "outputs": [], + "source": [ + "native_aligned = reproject_stack(sources, stack_spec, backend=\"native\")\n", + "print(compare_arrays(aligned.images, native_aligned.images, label=\"xRep CuPy / native\"))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-48", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xrep-native.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xrep-native.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " cuphoton xrep reproject-stack \\\n", + " --inputs \"$REFERENCE,$RAW_TARGET\" --hdu 1 \\\n", + " --target-wcs \"$REFERENCE\" --target-hdu 1 \\\n", + " --backend native --fits-reader xdr --interpolation lanczos3 \\\n", + " --mapping-grid-step 1 --write-fits \\\n", + " --output-dir \"$RUNS\" --name xrep-native >\"$RUNS/xrep-native.log\" 2>&1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-49", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xrep-native: launcher-to-exit seconds:\", float((RUNS / \"xrep-native.elapsed\").read_text()))\n", + "native_cli_aligned = np.load(RUNS / \"xrep-native\" / \"artifacts\" / \"reprojected_stack.npy\")\n", + "print(compare_arrays(native_aligned.images, native_cli_aligned,\n", + " label=\"xRep native Python / CLI\"))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-50", + "metadata": {}, + "outputs": [], + "source": [ + "native_subtraction = solve_constant_kernel_device(\n", + " reference_device, target_device, components, backend=\"native\", **solve_options\n", + ")\n", + "print(compare_arrays(difference, cp.asnumpy(native_subtraction.residual),\n", + " label=\"xPois CuPy / native\"))\n", + "_ = image_grid([difference, cp.asnumpy(native_subtraction.residual),\n", + " difference - cp.asnumpy(native_subtraction.residual)],\n", + " [\"CuPy difference\", \"Native difference\", \"Numerical difference\"], diverging=True)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-51", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xpois-native-cli.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xpois-native-cli.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " cuphoton xpois fit-kernel --backend native --solver constant \\\n", + " --reference \"$REFERENCE\" --reference-hdu 1 --target \"$TARGET\" --target-hdu 1 \\\n", + " --variance \"$VARIANCE\" --variance-hdu 1 --fit-mask \"$FIT_MASK\" \\\n", + " --kernel-height 15 --kernel-width 15 --basis-sigmas 1.5 --basis-degrees 0 \\\n", + " --background-degree 0 --fits-reader xdr --no-review \\\n", + " --output-dir \"$RUNS\" --name xpois-native-cli >\"$RUNS/xpois-native-cli.log\" 2>&1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-52", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xpois-native-cli: launcher-to-exit seconds:\", float((RUNS / \"xpois-native-cli.elapsed\").read_text()))\n", + "native_cli_difference = np.load(RUNS / \"xpois-native-cli\" / \"artifacts\" / \"residual.npy\")\n", + "print(compare_arrays(cp.asnumpy(native_subtraction.residual), native_cli_difference,\n", + " label=\"xPois native Python / CLI\"))" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-53", + "metadata": {}, + "source": [ + "Native xPois also moves compact-source filtering and scoring into precompiled\n", + "CUDA. Compare its selected centers and mask with the ordinary GPU-array route.\n", + "This diagnostic mask is independent of the fixed injection-exclusion mask used\n", + "for all subtraction comparisons above; we do not change the scientific inputs.\n", + "This source-selection call has no exact standalone CLI equivalent; the\n", + "`fit-kernel` command above compares the same fixed-mask solve as the Python API.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-54", + "metadata": {}, + "outputs": [], + "source": [ + "from cuphoton.xpois import build_compact_source_stamp_mask\n", + "\n", + "selection_options = dict(valid_mask=solve_options[\"fit_mask\"], stamp_size=31, max_stamps=8)\n", + "# CuPy input selects the GPU-array route; this API names its modes auto/native.\n", + "selection_gpu = build_compact_source_stamp_mask(reference_device, **selection_options)\n", + "selection_native = build_compact_source_stamp_mask(\n", + " reference_device, backend=\"native\", **selection_options\n", + ")\n", + "assert selection_native.centers == selection_gpu.centers\n", + "np.testing.assert_array_equal(cp.asnumpy(selection_native.mask), cp.asnumpy(selection_gpu.mask))\n", + "print(\"Identical selected centers:\", selection_native.centers)\n", + "_ = image_grid([reference, cp.asnumpy(selection_native.mask)],\n", + " [\"Reference: compact-source selection\", \"Native selected fit regions\"], columns=2)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-55", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/xfit-native.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/xfit-native.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend native --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-native\" --max-workers \"$WORKERS\" >\"$RUNS/xfit-native.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-56", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend native --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-native\" --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xfit fit-dipoles \\\n", + " --executor dragon --input \"$XFIT_INPUT\" --model gaussian --mode difference \\\n", + " --backend native --compute-dtype float64 --solver-version v0.2 \\\n", + " --chunk-size 32 --max-evaluations 400 \\\n", + " --output-dir \"$RUNS/xfit-native\" --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-57", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"xfit-native: launcher-to-exit seconds:\", float((RUNS / \"xfit-native.elapsed\").read_text()))\n", + "native_fits = pd.read_parquet(RUNS / \"xfit-native\" / \"scientific\" / \"fits.parquet\")\n", + "assert native_fits[\"candidate_id\"].tolist() == fit_table[\"candidate_id\"].tolist()\n", + "candidate_kind = {row[\"candidate_id\"]: row[\"kind\"]\n", + " for item in data_manifest[\"pairs\"] for row in item[\"candidates\"]}\n", + "fit_comparison = pd.DataFrame({\n", + " \"candidate_id\": fit_table[\"candidate_id\"],\n", + " \"kind\": [candidate_kind[c] for c in fit_table[\"candidate_id\"]],\n", + " \"cupy_converged\": fit_table[\"converged\"],\n", + " \"native_converged\": native_fits[\"converged\"],\n", + " \"cupy_uncertainty_valid\": fit_table[\"uncertainty_valid\"],\n", + " \"native_uncertainty_valid\": native_fits[\"uncertainty_valid\"],\n", + "})\n", + "centers = [\"x_pos\", \"y_pos\", \"x_neg\", \"y_neg\"]\n", + "cupy_centers = fit_table[centers].to_numpy()\n", + "native_centers = native_fits[centers].to_numpy()\n", + "fit_comparison[\"center_max_abs_delta_px\"] = np.max(np.abs(cupy_centers - native_centers), axis=1)\n", + "fit_comparison[\"strict_center_failure\"] = ~np.isclose(\n", + " cupy_centers, native_centers, rtol=1e-5, atol=1e-7, equal_nan=True,\n", + ").all(axis=1)\n", + "center_errors = [f\"{name}_standard_error\" for name in centers]\n", + "fit_comparison[\"cupy_center_standard_error_max_px\"] = fit_table[center_errors].max(axis=1)\n", + "fit_comparison[\"native_center_standard_error_max_px\"] = native_fits[center_errors].max(axis=1)\n", + "print(\"All-candidate center comparison: rtol=1e-5, atol=1e-7 pixel\")\n", + "display(fit_comparison.groupby(\"kind\").agg(\n", + " candidates=(\"candidate_id\", \"size\"),\n", + " cupy_converged=(\"cupy_converged\", \"sum\"), native_converged=(\"native_converged\", \"sum\"),\n", + " cupy_valid=(\"cupy_uncertainty_valid\", \"sum\"), native_valid=(\"native_uncertainty_valid\", \"sum\"),\n", + " center_max_abs_delta_px=(\"center_max_abs_delta_px\", \"max\"),\n", + " strict_center_failures=(\"strict_center_failure\", \"sum\"),\n", + "))\n", + "display(fit_comparison.nlargest(6, \"center_max_abs_delta_px\")[[\n", + " \"candidate_id\", \"kind\", \"center_max_abs_delta_px\", \"strict_center_failure\",\n", + " \"cupy_center_standard_error_max_px\", \"native_center_standard_error_max_px\",\n", + "]])\n", + "identified = ((fit_comparison[\"kind\"] == \"moving_source\")\n", + " & fit_comparison[\"cupy_converged\"] & fit_comparison[\"native_converged\"]\n", + " & fit_comparison[\"cupy_uncertainty_valid\"] & fit_comparison[\"native_uncertainty_valid\"])\n", + "assert identified.any(), \"No identified moving-source fits to compare\"\n", + "print(compare_arrays(fit_table.loc[identified, centers].to_numpy(),\n", + " native_fits.loc[identified, centers].to_numpy(),\n", + " rtol=1e-4, atol=1e-5, label=\"xFit CuPy / native identified centers\"))\n", + "fit_comparison.to_json(RUNS / \"native-fit-comparison.json\", orient=\"records\", indent=2)\n", + "_ = summarize_run(RUNS / \"xfit-native\")" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-58", + "metadata": {}, + "source": [ + "The all-candidate table reports center differences and strict comparison failure counts\n", + "for every injected class. A Gaussian dipole is a mismatched model for sharp or\n", + "track-like detector artifacts; poorly constrained fits can have large formal\n", + "center errors even when their convergence and uncertainty-valid flags are true.\n", + "Inspect those rows separately from the identified moving-source assertion.\n", + "Numerical agreement does not make a mismatched artifact fit scientifically identifiable.\n", + "\n", + "To place native xFit inside the resident pipeline, only its backend field changes.\n", + "The replay below retains\n", + "solver policy v0.2 and the exact checkpoint, pair inputs, and other settings.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-59", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "trap 'tail -n 40 \"$RUNS/pipeline-native.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/pipeline-native.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$NATIVE_PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-native \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\" >\"$RUNS/pipeline-native.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-60", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$NATIVE_PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-native \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp cuphoton xscan run-pipeline \\\n", + " --executor dragon --manifest \"$NATIVE_PIPELINE_MANIFEST\" \\\n", + " --output-dir \"$RUNS\" --name pipeline-native \\\n", + " --warmup-rounds 1 --measure-rounds 3 --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-61", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"pipeline-native: launcher-to-exit seconds:\", float((RUNS / \"pipeline-native.elapsed\").read_text()))\n", + "native_pipeline_path = Path(prepared[\"native_pipeline_manifest\"])\n", + "_ = summarize_run(RUNS / \"pipeline-native\")\n", + "native_config, _ = load_pipeline_manifest(native_pipeline_path)\n", + "native_round = sorted((RUNS / \"pipeline-native\" / \"rounds\").glob(\"measure-*\"))[-1]\n", + "native_scores = []\n", + "for item in pipeline_items:\n", + " payload = load_json(native_round / \"items\" / item.item_id / \"summary.json\")\n", + " observed = decode_device_pipeline_evidence(payload[\"scientific_evidence\"], config=native_config)\n", + " expected = decode_device_pipeline_evidence(local_by_id[item.item_id][\"scientific_evidence\"],\n", + " config=pipeline_config)\n", + " native_scores.append(compare_arrays(\n", + " expected[\"xscan.probabilities\"], observed[\"xscan.probabilities\"],\n", + " rtol=1e-4, atol=1e-5, label=f\"{item.item_id}: CuPy/native pipeline scores\"\n", + " ))\n", + "display(pd.DataFrame(native_scores))" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-62", + "metadata": {}, + "source": [ + "## Optional: discover candidates and build a richer catalog · 5–7 minutes\n", + "\n", + "The existing [transient-chain example](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/examples/transient-chain/README.md) starts from\n", + "FITS pairs, aligns them, constructs a fit mask, detects residual peaks,\n", + "and fits candidates with mask/variance handling, quality flags, propagated\n", + "uncertainties, and single-source refits. It also supports a compatible image-only\n", + "checkpoint. Our injection-trained **fusion** checkpoint is a different model\n", + "contract, so this appendix intentionally ends after xFit. The example also offers\n", + "star-based registration refinement; the prepared injection demo disables it so\n", + "fitting an affine transform cannot absorb the deliberately injected motion.\n", + "\n", + "The chain is an example application with its own `launch.py`, not an invented\n", + "`cuphoton` subcommand. Its present numerical path is CuPy; it does not expose the\n", + "native selectors demonstrated above. The uncertainty budget describes the\n", + "supplied noise model, and correlated or reused background pixels still limit\n", + "interpretation. The inputs here remain controlled injections on real backgrounds.\n", + "\n", + "Set `CUPHOTON_DEMO_TRANSIENT_CHAIN=1` before starting Jupyter to execute the\n", + "optional Bash cell. The separate `examples/transient-chain` package must be\n", + "installed in the shared environment, as shown in the README." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-63", + "metadata": {}, + "outputs": [], + "source": [ + "chain = prepared.get(\"transient_chain\")\n", + "RUN_TRANSIENT_CHAIN = os.environ.get(\"CUPHOTON_DEMO_TRANSIENT_CHAIN\", \"0\") == \"1\"\n", + "if RUN_TRANSIENT_CHAIN and chain is None:\n", + " raise RuntimeError(\"Optional chain inputs are missing; rerun prepare.py in a new output directory.\")\n", + "if chain is not None:\n", + " os.environ.update({\n", + " \"TRANSIENT_LAUNCHER\": str(REPO / \"examples\" / \"transient-chain\" / \"launch.py\"),\n", + " \"TRANSIENT_CONFIG\": chain[\"config\"], \"TRANSIENT_SELECTIONS\": chain[\"selections\"],\n", + " })" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-64", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "set -euxo pipefail\n", + "if [[ ${CUPHOTON_DEMO_TRANSIENT_CHAIN:-0} != 1 ]]; then\n", + " echo \"Optional transient-chain skipped; set CUPHOTON_DEMO_TRANSIENT_CHAIN=1 to run.\"\n", + " exit 0\n", + "fi\n", + "trap 'tail -n 40 \"$RUNS/transient-chain.log\"' EXIT\n", + "/usr/bin/time -f '%e' -o \"$RUNS/transient-chain.elapsed\" \\\n", + " timeout --signal=TERM --kill-after=10s 600s \\\n", + " dragon -s -t tcp -o tcp python \"$TRANSIENT_LAUNCHER\" \\\n", + " --executor dragon --config \"$TRANSIENT_CONFIG\" --selections \"$TRANSIENT_SELECTIONS\" \\\n", + " --output-dir \"$RUNS\" --name transient-chain --max-workers \"$WORKERS\" >\"$RUNS/transient-chain.log\" 2>&1" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-65", + "metadata": {}, + "source": [ + "For an allocated cluster, replace the `dragon -s ...` command above with\n", + "one of these complete commands; retain the Bash cell's timing and log redirection.\n", + "Shared absolute paths must exist on every host.\n", + "\n", + "```bash\n", + "# Slurm: run inside an interactive allocation.\n", + "dragon --wlm slurm --nodes \"$NODES\" -t tcp -o tcp python \"$TRANSIENT_LAUNCHER\" \\\n", + " --executor dragon --config \"$TRANSIENT_CONFIG\" --selections \"$TRANSIENT_SELECTIONS\" \\\n", + " --output-dir \"$RUNS\" --name transient-chain --max-workers \"$WORKERS\"\n", + "\n", + "# SSH: use the host file for dedicated, authorized hosts.\n", + "dragon --wlm ssh --nodes \"$NODES\" --hostfile \"$HOSTFILE\" -t tcp -o tcp python \"$TRANSIENT_LAUNCHER\" \\\n", + " --executor dragon --config \"$TRANSIENT_CONFIG\" --selections \"$TRANSIENT_SELECTIONS\" \\\n", + " --output-dir \"$RUNS\" --name transient-chain --max-workers \"$WORKERS\"\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sc26-66", + "metadata": {}, + "outputs": [], + "source": [ + "if RUN_TRANSIENT_CHAIN:\n", + " print(\"transient-chain: launcher-to-exit seconds:\", float((RUNS / \"transient-chain.elapsed\").read_text()))\n", + " _ = summarize_run(RUNS / \"transient-chain\")\n", + " catalogs = list((RUNS / \"transient-chain\").glob(\"**/catalog.parquet\"))\n", + " catalog = pd.read_parquet(catalogs[0])\n", + " display(catalog[[\"candidate_id\", \"x\", \"y\", \"peak_abs_sigma\", \"amplitude\", \"amplitude_err\",\n", + " \"probability\", \"decision\"]].head(12))" + ] + }, + { + "cell_type": "markdown", + "id": "sc26-67", + "metadata": {}, + "source": [ + "The completed run leaves source provenance, manifests, model evaluation, launch\n", + "logs, scientific comparisons, and worker records beside the data. The plots show\n", + "what each stage calculated; the recorded host/GPU identities show where it ran.\n", + "For larger observations, retain the same contracts and increase the number of\n", + "independent image pairs before changing worker sharing or transport settings.\n", + "\n", + "Further reading: [Dragon architecture](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/docs/dragon.md),\n", + "[launch guide](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/docs/distributed-execution.md),\n", + "[transport and timing](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/docs/dragon-performance.md),\n", + "[native xFit](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/docs/components/xfit.md#optional-native-cuda-backend), and\n", + "[xImage reconstruction](https://github.com/NVIDIA/cuPhoton/blob/b17cf8a30fa64c466c3e9b62ad98eccbb8f87cab/docs/components/ximage.md) for the separate problem\n", + "of reconstructing a common scene from registered exposures with supplied PSFs.\n" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/syllabi/sc26__8_hours.ipynb b/notebooks/syllabi/sc26__8_hours.ipynb index 70c6e77..50e8148 100644 --- a/notebooks/syllabi/sc26__8_hours.ipynb +++ b/notebooks/syllabi/sc26__8_hours.ipynb @@ -23,7 +23,8 @@ "| 10 | [Mini-App: CppJIT - CUDA C++ Kernel](../gpu-mini-app/10__cosmic_web__cppjit__cuda_kernel.ipynb) | - | CppJIT, CUDA |\n", "| 11 | [Mini-App: Synthesis](../gpu-mini-app/11__cosmic_web__synthesis.ipynb) | - | - |\n", "| - | Domain Science Examples | [Notebooks](https://compiler-research.org/live-cpp-tutorials/) | - |\n", - "| 12 | [DragonHPC/cuPhoton](../distributed/12__dragonhpc_cuphoton__workflow.ipynb) (placeholder) | - | DragonHPC, cuPhoton |\n", + "| 12 | [DragonHPC](../distributed/12__dragonhpc_cuphoton__intro.ipynb) | [Click](../distributed/solutions/12__dragonhpc_cuphoton__intro__SOLUTION.ipynb) | DragonHPC |\n", + "| 13 | [DragonHPC/cuPhoton](../distributed/13__dragonhpc_cuphoton__workflow.ipynb) | [Click](../distributed/solutions/13__dragonhpc_cuphoton__workflow__SOLUTION.ipynb) | DragonHPC, cuPhoton |\n", "\n", "### Materials\n", "\n", @@ -44,7 +45,8 @@ "- Programming models in Python for GPU applications: array expressions, fusion, library iterators, kernel authoring, and interoperability with CUDA C++ kernels.\n", "- Reading an application's profile with Nsight Systems and Nsight Compute.\n", "- Case studies of horizontal Python and C++ integration in domain sciences such as cancer research and high energy physics.\n", - "- cuPhoton and DragonHPC (placeholder).\n", + "- DragonHPC fundamentals enabling `multiprocessing` to scale across HPC multi-node resources.\n", + "- cuPhoton astronomical image processing workflows orchestrated using DragonHPC.\n", "\n", "Attendees are expected to have general knowledge of HPC and Python programming. Some familiarity with NumPy and C/C++ is helpful but not required." ] diff --git a/test/test_packages.py b/test/test_packages.py index a667608..6195c74 100644 --- a/test/test_packages.py +++ b/test/test_packages.py @@ -344,7 +344,7 @@ def test_dragon_and_cuphoton(): assert torch.cuda.is_available() assert Path(os.environ["CUPHOTON_DEMO_REPO"], "examples", "dragon-imaging-pipeline", "demo.py").is_file() result = subprocess.run(["dragon", "--version"], capture_output=True, text=True, check=True) - assert "Dragon Version 0.14.2" in result.stdout, result.stdout + result.stderr + assert "Dragon Version 0.15.0" in result.stdout, result.stdout + result.stderr def test_mpi4py():