diff --git a/.claude/skills/openbot-screen-layout/SKILL.md b/.claude/skills/openbot-screen-layout/SKILL.md index 20cd104d6f..39442b1160 100644 --- a/.claude/skills/openbot-screen-layout/SKILL.md +++ b/.claude/skills/openbot-screen-layout/SKILL.md @@ -188,11 +188,11 @@ itself when the dialog changes something. ## Units -The root is the browser's 16px, so Tailwind's spacing, sizes and widths are their documented values -(`p-4` is 16px, `max-w-3xl` is 768px) and land on its 4px grid. Text is the exception: `styles.css` -sets every `--text-*` to Tailwind's default times 15/16, so `text-sm` is 13.125px and body text is -15px. That is the scale this app has always read at; do not size text in px or rem by hand to match -the docs. +The root is the browser's 16px, so Tailwind's spacing, sizes, widths and text sizes are their +documented values (`p-4` is 16px, `max-w-3xl` is 768px, `text-sm` is 14px) and land on its 4px grid. +Text used to be the exception, on a 15px scale of its own; it is not any more. Where a screen needs one +of the named sizes, use the five `type-*` utilities in `theme.css`; do not size text in px or rem by +hand. Even so, never hardcode a pixel constant derived from a Tailwind token, and never compute a ratio from one. Measure the element — `offsetWidth`, `clientWidth`, a `ResizeObserver` — and scale from what diff --git a/CHANGELOG.md b/CHANGELOG.md index 27745c6f84..87c9c74a26 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged. ## Unreleased -**Before upgrading.** Sixteen things change for an existing deployment: +**Before upgrading.** Seventeen things change for an existing deployment: - Bots now share one computer per person instead of each having its own. Migration `0069_isolated_bots` adds `agent_profiles.isolated`, off for every existing Bot. With the supervisor (laptop, Desktop, Compose), a person's computer is created the first time one of their @@ -64,6 +64,9 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged. research run replaces them. - Migration `0062_action_policy_labels` adds a `labels` column to `action_policies`, defaulting to empty. Existing boundary rules keep working and show as "Custom rule" until somebody names them. +- Migration `0070_tasks` adds one table, `tasks`, naming each task's conversation (below), and one + index to `work_items`. The table starts empty. A suggestion started before the upgrade and still + queued opens a task when it runs, rather than running in the conversation it was suggested in. - On Helm's `shared` computer and the docker-compose `agent-computer`, each Bot's files now live in `computers/` on the workspace volume (below). Files saved before the upgrade stay at the root of the volume, where no Bot sees them; move them into a Bot's folder to give them back. @@ -78,6 +81,31 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged. `COMPUTER_MEMORY_HIGH_PERCENT=0` and `COMPUTER_TAB_IDLE_MS=0` (Helm: `computers.memoryHighPercent` and `computers.tabIdleMs`) keep the old behaviour. +### Started work gets a conversation of its own + +Start on a suggestion now opens a task: a new conversation with the Bot where the work runs, so the +conversation the suggestion came from stays as it was. A task's status is worked out from the work +queue and the approval and question inboxes on every read. A turn that fails is tried once more, then +reads as failed with the reason in plain words and can be tried again; replying in a task's +conversation steers it, in order, even across a restart. The new `/api/tasks` routes serve it. + +A Bot OpenBot runs itself now hands anything bigger than a quick answer to a task of its own +(`start_task`) and answers "On it" where it was asked. A Bot behind a remote agent framework is not +offered the tool; its tasks start from its suggestions, or from `POST /api/tasks`, which starts one +with any of the person's Bots. `GET /api/tasks/:id/report` reads a task's conversation back as its +report: the person's decisions and replies, and the Bot's update as each turn ended, in order. + +Tasks sits above Bots in the sidebar, with how many tasks need you. It lists every task by what it +needs (Needs you, Working, Failed, Ready for review, then Done, folded), each with its Bot and one line +of what comes next, and opens the task beside the list, or after it on a narrow screen. ↓ and ↑, or +j and k, move through the list, Enter opens a task and Escape closes it. A task's page is its report +above its conversation, with Mark done, Reopen, and Try again when it couldn't finish; a reply there +goes to the server and waits its turn, and a reply on a failed task tries it again with your words. +The + starts a task with any of your Bots, and Up next under the list starts or dismisses your +Bots' suggestions. In a conversation, a task a Bot started is a card under its "On it" that keeps +its status and opens the task. A task's own conversation is left out of the conversation list, and +the OpenDuet skin's Tasks tab now opens this Tasks and counts the tasks that need you. + ### A computer closes browsers nobody is using when its memory runs high A computer runs one browser for each Bot that is browsing, and the only limits were a count and half @@ -133,6 +161,67 @@ again when they come back to the end or press the arrow. A rail beside a convers per turn: hovering one previews what was asked and answered, and choosing it goes there. Nothing changes for a deployment. +### First run shows what OpenBot does, one step at a time + +- The looping intro is replaced by steps that show what OpenBot does (a browser and files of its + own, a team of Bots, and asking before it acts), then Connect your apps, Get notified, Check your + setup, and the name step, whose Get started opens the app. On a desktop the steps are listed on the + left beside a picture of each; on a phone they come one at a time. A reload, browser Back and a + connection's return all keep the person's place, which is now saved as they go. +- Check your setup asks the new `GET /api/setup` whether Intelligence, the model key, the Bot computer, + more apps and Slack and Teams work, row by row. An admin, or somebody running OpenBot alone, is + shown the page or the variables that close a gap; everybody else only that their admin sets it up. + Check again asks Intelligence and the model provider again, at most every 10 seconds. Reading the + checklist does the same once an answer has failed or is five minutes old, without waiting for it, + so the next read has the new answer. To check the Bot computer, the server sends each running + computer its network policy whenever it asks, and sees that it answers; it wakes none. +- An app connected from Connect your apps comes back to that step from the app's own sign-in page, + whether it connected, failed or was cancelled there. A connection started from Connections or an + app's admin page comes back where it did before. + +### OpenBot has a new look + +- Every screen has new colours: cool neutral greys, a near-black primary button that turns light + in dark mode, a purple focus ring, slightly rounder corners and quicker easing. Text now follows + the browser's own size, so it is a little larger than before. +- Dark mode follows the system by default. Settings → General → Appearance offers System, Light and + Dark, kept in this browser. Someone who had turned the old dark switch on stays dark. +- Every Bot, and OpenBot's own mark, is an orb with eyes that blinks and follows the pointer. It + holds still with reduced motion. +- Each Bot's orb takes its colour and pattern from the Bot, so two Bots in one colour still look + different. Violet is kept for OpenBot's own mark. +- The sidebar takes the same look, with OpenBot's orb beside its name. A row lifts to a card under + the pointer and while it is open, and you can always see where the keyboard is. While a Bot is + working, its orb in the list shows it, and so does a still blue dot. Each conversation has a "…" + button with Pin and Delete, which used to be on right-click alone. ⌘K, or Ctrl+K, opens the + sidebar and puts you in its search box. +- A conversation reads like a chat. A Bot's reply arrives one bubble per thought, and a bubble holding + a code block or a table takes the full width on a wide screen; your messages sit in a tinted bubble, + and three dots show a Bot thinking. A step is marked done only when it worked: one that was refused, + failed or declined by you is named for what it would have done ("Send your email"), and one held on + its approval card says it is waiting for you. Mail and calendar steps, through any app, are named + for what happened to yours ("Searched your email"). +- A conversation with one Bot shows that Bot's orb in its header. The orb works while the Bot does, + watches while you type, and smiles when an answer lands. Under the conversation's name, "Working · + Searching the web" says which step the Bot is on, and "Needs you" says an approval is waiting there. + A run that goes on after you allow something shows as working too. The message box is a rounded + pill with larger buttons, and it says "Message" and the Bot's name until you type. +- A Bot's computer opens in a wider pane beside the conversation. Its header shows the Bot's orb, + whose computer it is and Take control, with a dot while the conversation needs you. Under the + screen, every step the Bot took in its browser, approved clicks included, sits among the commands + it ran, newest last, marked done, waiting for you or didn't work. The pane opens by itself only + when it fits beside the conversation; on a narrow screen the computer button lights up instead. + The box for a password or a code says the value goes straight to the page and the Bot never sees + it. +- When a Bot asks before acting, its card says in plain words what it will do, in which app or + where, to whom, and what it will send, with everything else under Details. Decline and Allow once + answer it, and a quieter Always allow here saves a rule for that Bot, action and place; a step + handed to you asks "I did it myself" or "Don't do it". The same card is in the conversation, on + the Bot's page and in the list of what needs you. Once you answer, it folds to how it went + (Allowed, Sent, Done, Declined, Didn't work) and says who decided. A request for a step that is + alone in its turn now shows in the conversation too, where before it waited only on the Bot's + page. Under the message box, every conversation says that OpenBot is AI and can make mistakes. + ### Generated interfaces use A2UI 1.0 With generative UI on, a Bot builds an interface from the A2UI 1.0 basic components instead of the diff --git a/app/index.html b/app/index.html index a839d57c50..b35c822cb2 100644 --- a/app/index.html +++ b/app/index.html @@ -5,15 +5,27 @@ OpenBot + diff --git a/app/package.json b/app/package.json index b5be1f7ac8..406d4f0a40 100644 --- a/app/package.json +++ b/app/package.json @@ -28,7 +28,6 @@ "@tanstack/react-router": "^1.170.27", "@tanstack/react-table": "^9.3.1", "better-auth": "^1.6.27", - "boring-avatars": "^2.0.4", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "embla-carousel-react": "^8.6.0", diff --git a/app/src/components/agents/abstract-avatar.tsx b/app/src/components/agents/abstract-avatar.tsx index 9ca9845070..d746511206 100644 --- a/app/src/components/agents/abstract-avatar.tsx +++ b/app/src/components/agents/abstract-avatar.tsx @@ -1,5 +1,14 @@ -import Avatar from "boring-avatars"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +/** + * A Bot's face: its orb, with the look its seed picks. The seed is the Bot's id unless a tenant + * package set one or the Bot was copied from another, so `ChannelAvatar`, which goes by id, matches it + * for every other Bot. + * The orb hides itself from assistive tech, so the Bot is announced once, by name. + * + * Clipped to its circle, so its glow never spills into the row beside it or makes a list scroll. + * At rest the glow is too faint that far out to be missed. + */ export function AbstractAvatar({ name, seed, @@ -16,10 +25,7 @@ export function AbstractAvatar({ className="inline-flex shrink-0 overflow-hidden rounded-full" style={{ height: size, width: size }} > - {/* The drawing carries its own role; hidden so the coworker is announced once, by name. */} - + ); } diff --git a/app/src/components/agents/orb/agent-orb.tsx b/app/src/components/agents/orb/agent-orb.tsx index 9fc2c7f393..4f953f922b 100644 --- a/app/src/components/agents/orb/agent-orb.tsx +++ b/app/src/components/agents/orb/agent-orb.tsx @@ -1,437 +1,52 @@ -import { - type MotionStyle, - motion, - type TargetAndTransition, - type Transition, - useReducedMotion, - useTransform, -} from "motion/react"; import { cn } from "@/lib/utils"; -import { - type AIAmplitude, - type AIState, - getAIStateMotion, - useAmplitudeValue, -} from "./ai-core"; - -const SIZE_THRESHOLD_SMALL = 50; -const SIZE_THRESHOLD_TINY = 30; -const SIZE_THRESHOLD_MEDIUM = 100; -const BLUR_MULTIPLIER_SMALL = 0.008; -const BLUR_MIN_SMALL = 1; -const BLUR_MULTIPLIER_LARGE = 0.015; -const BLUR_MIN_LARGE = 4; -const CONTRAST_MULTIPLIER_SMALL = 0.004; -const CONTRAST_MIN_SMALL = 1.2; -const CONTRAST_MULTIPLIER_LARGE = 0.008; -const CONTRAST_MIN_LARGE = 1.5; -const DOT_SIZE_MULTIPLIER_SMALL = 0.004; -const DOT_SIZE_MIN_SMALL = 0.05; -const DOT_SIZE_MULTIPLIER_LARGE = 0.008; -const DOT_SIZE_MIN_LARGE = 0.1; -const SHADOW_MULTIPLIER_SMALL = 0.004; -const SHADOW_MIN_SMALL = 0.5; -const SHADOW_MULTIPLIER_LARGE = 0.008; -const SHADOW_MIN_LARGE = 2; -const MASK_RADIUS_TINY = "0%"; -const MASK_RADIUS_SMALL = "5%"; -const MASK_RADIUS_MEDIUM = "15%"; -const MASK_RADIUS_LARGE = "25%"; -const CONTRAST_TINY = 1.1; -const CONTRAST_MULTIPLIER_FINAL = 1.2; -const CONTRAST_MIN_FINAL = 1.3; - -/** Loud audio tightens the gradient, which reads as the orb "focusing". */ -const AMPLITUDE_BLUR_FALLOFF = 0.45; -/** Amplitude adds at most 12% of extra size on top of the state scale. */ -const AMPLITUDE_SCALE_GAIN = 0.12; -/** Single lateral nudge used to signal `error`, well under 200ms. */ -const ERROR_SHAKE_KEYFRAMES = [0, -3, 3, 0]; -const ERROR_SHAKE_DURATION = 0.18; -const EASE_IN_OUT = [0.645, 0.045, 0.355, 1] as const; -const GLOW_BLUR_RATIO = 0.28; -const GLOW_MAX_OPACITY = 0.7; -/** Depth rim, as a fraction of the orb size. */ -const RIM_RATIO = 0.06; -const RIM_MIN = 1.5; -/** Sheen drift period at rest, before the state speed divides it. */ -const DRIFT_BASE_SECONDS = 12; -/** `idle` breathes: a slow, shallow scale cycle that never draws attention. */ -const BREATHE_SCALE = [1, 1.035, 1]; -const BREATHE_SECONDS = 5.5; -const SPRING_DEFAULT: Transition = { - bounce: 0.1, - duration: 0.25, - type: "spring", +import { BotOrb, type BotOrbState } from "./bot-orb"; + +/** What OpenBot is doing, wherever its own mark stands for it. */ +export type AIState = + | "idle" + | "listening" + | "thinking" + | "streaming" + | "done" + | "error"; + +/** + * Each state as the orb shows it. Streaming is still working. An error rests, and the words beside + * the mark say what went wrong. + */ +const ORB_STATE: Record = { + idle: "idle", + listening: "listening", + thinking: "thinking", + streaming: "thinking", + done: "done", + error: "idle", }; export interface AgentOrbProps { + className?: string; /** - * Live audio level, 0–1. Pass the `MotionValue` from `useAudioAmplitude` so - * the signal never re-renders React. + * Its footprint, in px, as "64px". Only px: the orb is drawn on a canvas at that size, so a share + * of its container, as "100%", has nothing to be a share of. */ - amplitude?: AIAmplitude; - /** Ambient rotation period in seconds, before the state speed multiplier. */ - animationDuration?: number; - className?: string; - colors?: { - bg?: string; - c1?: string; - c2?: string; - c3?: string; - /** Fourth mesh stop. More stops means fewer visible repeats per rotation. */ - c4?: string; - }; - size?: string; - /** Shared AI state driving speed, scale, saturation and reactivity. */ + size?: `${number}px`; state?: AIState; } -const AgentOrb: React.FC = ({ - size = "192px", +/** OpenBot's own mark: the orb, in OpenBot's violet. */ +export default function AgentOrb({ className, - colors, - animationDuration = 20, - amplitude, + size = "192px", state = "idle", -}) => { - const shouldReduceMotion = useReducedMotion(); - const amplitudeValue = useAmplitudeValue(amplitude); - const stateMotion = getAIStateMotion(state); - - /** - * Higher chroma than the original pastels. - * - * The first palette sat at chroma 0.12–0.15, which washes out once the mesh is - * blurred and the dot pattern is overlaid on top. Pushing to ~0.2 and adding a - * saturate() pass to the mesh is what makes the colour survive all that. - */ - const defaultColors = { - bg: "oklch(92% 0.03 300)", - c1: "oklch(68% 0.21 350)", // Pink - c2: "oklch(70% 0.18 210)", // Blue - c3: "oklch(66% 0.2 285)", // Violet - c4: "oklch(72% 0.19 325)", // Magenta, the fourth stop for variety - }; - - const finalColors = { ...defaultColors, ...colors }; - // The bloom stays in the orb's own palette — see the note in ai-orb-aura. - const glowColor = finalColors.c2; - - // Extract numeric value from size for calculations - const sizeValue = Number.parseInt(size.replace("px", ""), 10); - - // Responsive calculations based on size - const blurAmount = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * BLUR_MULTIPLIER_SMALL, BLUR_MIN_SMALL) // Reduced blur for small sizes - : Math.max(sizeValue * BLUR_MULTIPLIER_LARGE, BLUR_MIN_LARGE); - - const contrastAmount = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * CONTRAST_MULTIPLIER_SMALL, CONTRAST_MIN_SMALL) // Reduced contrast for small sizes - : Math.max(sizeValue * CONTRAST_MULTIPLIER_LARGE, CONTRAST_MIN_LARGE); - - const dotSize = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * DOT_SIZE_MULTIPLIER_SMALL, DOT_SIZE_MIN_SMALL) // Smaller dots for small sizes - : Math.max(sizeValue * DOT_SIZE_MULTIPLIER_LARGE, DOT_SIZE_MIN_LARGE); - - const shadowSpread = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * SHADOW_MULTIPLIER_SMALL, SHADOW_MIN_SMALL) // Reduced shadow for small sizes - : Math.max(sizeValue * SHADOW_MULTIPLIER_LARGE, SHADOW_MIN_LARGE); - - // Adjust mask radius based on size to reduce black center in small sizes - const getMaskRadius = (value: number) => { - if (value < SIZE_THRESHOLD_TINY) { - return MASK_RADIUS_TINY; - } - if (value < SIZE_THRESHOLD_SMALL) { - return MASK_RADIUS_SMALL; - } - if (value < SIZE_THRESHOLD_MEDIUM) { - return MASK_RADIUS_MEDIUM; - } - return MASK_RADIUS_LARGE; - }; - - const maskRadius = getMaskRadius(sizeValue); - - // Use more subtle contrast for very small sizes - const getFinalContrast = (value: number) => { - if (value < SIZE_THRESHOLD_TINY) { - return CONTRAST_TINY; // Very subtle contrast for tiny sizes - } - if (value < SIZE_THRESHOLD_SMALL) { - return Math.max( - contrastAmount * CONTRAST_MULTIPLIER_FINAL, - CONTRAST_MIN_FINAL, - ); // Reduced contrast for small sizes - } - return contrastAmount; - }; - - const finalContrast = getFinalContrast(sizeValue); - - // Reactivity is gated by the state preset: `thinking` barely listens so the - // orb keeps churning internally instead of throbbing with room noise. - const reactivity = shouldReduceMotion ? 0 : stateMotion.reactivity; - - const reactiveBlur = useTransform(amplitudeValue, (level) => { - const focus = 1 - level * reactivity * AMPLITUDE_BLUR_FALLOFF; - return `${blurAmount * focus}px`; - }); - - const reactiveScale = useTransform( - amplitudeValue, - (level) => stateMotion.scale + level * reactivity * AMPLITUDE_SCALE_GAIN, - ); - - const loopDuration = shouldReduceMotion - ? animationDuration - : animationDuration / stateMotion.speed; - - // Rim thickness scales with the orb, so the lit edge reads the same at 24px - // and at 240px instead of swallowing the small one. - const rim = Math.max(sizeValue * RIM_RATIO, RIM_MIN); - // The sheen drifts slower than the mesh rotates; a busier state speeds it up - // a touch without matching the rotation, which would look mechanical. - const driftDuration = (DRIFT_BASE_SECONDS / (1 + stateMotion.speed)) * 2; - - const getRootAnimate = (): TargetAndTransition => { - if (shouldReduceMotion) { - return { scale: 1, x: 0 }; - } - if (state === "error") { - return { scale: 1, x: ERROR_SHAKE_KEYFRAMES }; - } - if (stateMotion.motif === "breathe") { - return { scale: BREATHE_SCALE, x: 0 }; - } - return { scale: 1, x: 0 }; - }; - - const getRootTransition = (): Transition => { - if (shouldReduceMotion) { - return { duration: 0 }; - } - if (state === "error") { - return { duration: ERROR_SHAKE_DURATION, ease: EASE_IN_OUT }; - } - if (stateMotion.motif === "breathe") { - return { - duration: BREATHE_SECONDS, - ease: EASE_IN_OUT, - repeat: Number.POSITIVE_INFINITY, - }; - } - return SPRING_DEFAULT; - }; - +}: AgentOrbProps) { + if (!/^\d+(\.\d+)?px$/.test(size)) { + throw new Error(`AgentOrb's size is in px, as "64px", not "${size}".`); + } return ( - // The gradient disc clips its own overflow, so the bloom and the status - // motif have to live in a wrapper rather than inside it. - - {/* Bloom tinted by the state's semantic accent. */} - - - - {/* Specular sheen and depth rim. The mesh alone reads as a flat blurred - disc; a drifting highlight plus a lit top edge and shaded bottom are - what make it read as a sphere. */} - - + ); -}; - -export default AgentOrb; +} diff --git a/app/src/components/agents/orb/ai-core.tsx b/app/src/components/agents/orb/ai-core.tsx deleted file mode 100644 index a1f13a6361..0000000000 --- a/app/src/components/agents/orb/ai-core.tsx +++ /dev/null @@ -1,395 +0,0 @@ -"use client"; - -import { - type MotionValue, - useAnimationFrame, - useMotionValue, -} from "motion/react"; -import { useCallback, useEffect, useRef, useState } from "react"; - -/** - * The single state contract shared by every SmoothUI AI component. - * - * Passing the same value to an orb, a prompt input and a tool card makes the - * whole surface move as one organism instead of a set of independent widgets. - */ -export type AIState = - | "idle" - | "listening" - | "thinking" - | "streaming" - | "done" - | "error"; - -/** - * A behavioural hint each component expresses **in its own material**. - * - * Deliberately not an overlay. Bolting one shared graphic onto every orb makes - * four different materials look like the same widget wearing a costume, and it - * pushes literal iconography (a checkmark, a warning ring) onto pieces that are - * decorative by nature. Instead: a shader warps, a canvas ring loses harmonics, - * a character changes expression. Same vocabulary, different flesh. - */ -export type AIStateMotif = - | "breathe" - | "receive" - | "scan" - | "pulse" - | "ping" - | "fault"; - -/** Semantic accent applied on top of the component's own palette. */ -export type AIStateAccent = "success" | "danger" | null; - -/** Motion parameters a component reads to render a given {@link AIState}. */ -export type AIStateMotion = { - /** Semantic colour override, so status is carried by hue and not only motion. */ - accent: AIStateAccent; - /** Outer bloom strength, 0–1. */ - glow: number; - /** Palette hue rotation in degrees. Small shifts read as a mood change. */ - hueRotate: number; - /** Overall motion energy, 0–1. Scales ambient loops and displacement. */ - intensity: number; - /** Behavioural hint; each component expresses it in its own material. */ - motif: AIStateMotif; - /** Seconds between discrete pulses, for rhythmic behaviour. */ - pulseSeconds: number; - /** - * How far a shader's domain warp pushes the field, 0–1. Low values read as a - * calm surface; high values churn without the silhouette growing. - */ - turbulence: number; - /** Revolutions per second of the noise field — the slow tumble. */ - tumble: number; - /** How much external amplitude reaches the surface, 0 = ignore it. */ - reactivity: number; - /** Chroma multiplier. Below 1 desaturates. */ - saturation: number; - /** Resting scale of the surface, 1 = no change. */ - scale: number; - /** Ambient loop speed multiplier, 1 = the component's base tempo. */ - speed: number; -}; - -/** - * Per-state presets. - * - * Deliberate choices worth keeping: - * - `thinking` has `scale: 1` — internal churn only, so layout stays calm while - * the model works. Growing the surface here makes pages feel unstable. - * - `error` desaturates instead of growing, so it reads as a state change - * rather than an attention grab. - * - `done` overshoots once; components are expected to settle back to `idle`. - * - Each state owns a distinct `motif`, so the six states are told apart by - * what moves, not by how fast it moves. - */ -export const AI_STATE_MOTION: Record = { - done: { - accent: "success", - glow: 0.7, - hueRotate: 0, - intensity: 0.4, - motif: "ping", - pulseSeconds: 0.65, - reactivity: 0, - saturation: 1, - scale: 1.1, - speed: 0.8, - // The field settles almost flat — stillness is what reads as "finished". - tumble: 0.02, - turbulence: 0.08, - }, - error: { - accent: "danger", - glow: 0.25, - hueRotate: 0, - intensity: 0.5, - motif: "fault", - pulseSeconds: 0.9, - reactivity: 0, - saturation: 0.3, - scale: 0.96, - speed: 1, - // Tumble stalls while turbulence stays mid: the surface twitches in place - // instead of flowing, which is what a fault feels like. - tumble: 0, - turbulence: 0.55, - }, - idle: { - accent: null, - glow: 0.15, - hueRotate: 0, - intensity: 0.3, - motif: "breathe", - pulseSeconds: 4.5, - reactivity: 0, - saturation: 0.75, - scale: 0.94, - speed: 0.6, - tumble: 0.012, - turbulence: 0.14, - }, - listening: { - accent: null, - glow: 0.6, - hueRotate: 0, - intensity: 0.75, - motif: "receive", - pulseSeconds: 1.6, - reactivity: 1, - saturation: 1.05, - scale: 1.06, - speed: 1, - tumble: 0.03, - turbulence: 0.42, - }, - streaming: { - accent: null, - glow: 0.45, - hueRotate: -10, - intensity: 0.6, - motif: "pulse", - pulseSeconds: 1.25, - reactivity: 0.6, - saturation: 1, - scale: 1.02, - speed: 1.4, - tumble: 0.06, - turbulence: 0.5, - }, - thinking: { - accent: null, - glow: 0.35, - hueRotate: 18, - intensity: 1, - motif: "scan", - pulseSeconds: 1.1, - reactivity: 0.15, - saturation: 1, - scale: 1, - speed: 2.4, - // High churn, unchanged silhouette: the work is visible without the orb - // growing and unsettling the layout around it. - tumble: 0.14, - turbulence: 0.95, - }, -}; - -/** Semantic accents. Deliberately not tokens — orbs render outside a theme. */ -export const AI_ACCENT_COLORS: Record<"success" | "danger", string> = { - danger: "oklch(63% 0.21 25)", - success: "oklch(72% 0.17 150)", -}; - -/** - * Colour used by a state's overlay motif: the semantic accent when the state - * has one, otherwise the component's own secondary colour. - */ -export const getAIStateAccentColor = ( - state: AIState | undefined, - fallback: string, -): string => { - const accent = AI_STATE_MOTION[state ?? "idle"]?.accent; - return accent ? AI_ACCENT_COLORS[accent] : fallback; -}; - -/** Motion preset for a state, falling back to `idle` for unknown values. */ -export const getAIStateMotion = (state: AIState | undefined): AIStateMotion => - AI_STATE_MOTION[state ?? "idle"] ?? AI_STATE_MOTION.idle; - -/** - * Amplitude accepted by every reactive AI component. - * - * A `MotionValue` is the preferred form: it updates outside React, so a 60fps - * audio signal never triggers a re-render. - */ -export type AIAmplitude = number | MotionValue | undefined; - -const isMotionValue = (value: AIAmplitude): value is MotionValue => - typeof value === "object" && value !== null && "get" in value; - -/** - * Normalises the `amplitude` prop into a stable `MotionValue` so - * component internals only deal with one shape. - */ -export const useAmplitudeValue = ( - amplitude: AIAmplitude, -): MotionValue => { - const fallback = useMotionValue(0); - const numeric = typeof amplitude === "number" ? amplitude : null; - - useEffect(() => { - if (numeric !== null) { - fallback.set(numeric); - } - }, [numeric, fallback]); - - return isMotionValue(amplitude) ? amplitude : fallback; -}; - -export type AudioAmplitudeStatus = - | "idle" - | "requesting" - | "active" - | "denied" - | "unsupported"; - -export type UseAudioAmplitudeOptions = { - /** Request microphone access as soon as the hook mounts. */ - autoStart?: boolean; - /** - * Envelope smoothing, 0–1. Higher is smoother and lazier; the default keeps - * attack snappy so an orb reacts on the first syllable. - */ - smoothing?: number; - /** FFT size handed to the analyser node. Must be a power of two. */ - fftSize?: number; -}; - -export type UseAudioAmplitudeResult = { - /** Smoothed RMS level, 0–1, as a `MotionValue`. */ - amplitude: MotionValue; - status: AudioAmplitudeStatus; - start: () => Promise; - stop: () => void; -}; - -const DEFAULT_SMOOTHING = 0.55; -const DEFAULT_FFT_SIZE = 512; -/** Raw RMS rarely exceeds ~0.3 for speech, so normalise into a usable 0–1. */ -const RMS_TO_UNIT = 3.2; -/** Attack is faster than release so peaks land immediately and decay gently. */ -const ATTACK_FACTOR = 0.35; - -/** - * Reads microphone loudness as a 0–1 `MotionValue`. - * - * SSR-safe, permission-aware, and silent on failure — a denied prompt leaves - * the amplitude at 0 so the consuming component simply falls back to its - * ambient animation. - */ -export const useAudioAmplitude = ( - options: UseAudioAmplitudeOptions = {}, -): UseAudioAmplitudeResult => { - const { - autoStart = false, - smoothing = DEFAULT_SMOOTHING, - fftSize = DEFAULT_FFT_SIZE, - } = options; - - const amplitude = useMotionValue(0); - const [status, setStatus] = useState("idle"); - - const contextRef = useRef(null); - const streamRef = useRef(null); - const analyserRef = useRef(null); - const bufferRef = useRef | null>(null); - - const stop = useCallback(() => { - for (const track of streamRef.current?.getTracks() ?? []) { - track.stop(); - } - streamRef.current = null; - analyserRef.current = null; - bufferRef.current = null; - contextRef.current?.close(); - contextRef.current = null; - amplitude.set(0); - setStatus("idle"); - }, [amplitude]); - - const start = useCallback(async () => { - if (analyserRef.current) { - return; - } - - const AudioContextCtor = - typeof window === "undefined" - ? undefined - : (window.AudioContext ?? - (window as unknown as { webkitAudioContext?: typeof AudioContext }) - .webkitAudioContext); - - if (!(AudioContextCtor && navigator.mediaDevices?.getUserMedia)) { - setStatus("unsupported"); - return; - } - - setStatus("requesting"); - - try { - const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); - const context = new AudioContextCtor(); - const analyser = context.createAnalyser(); - analyser.fftSize = fftSize; - context.createMediaStreamSource(stream).connect(analyser); - - streamRef.current = stream; - contextRef.current = context; - analyserRef.current = analyser; - bufferRef.current = new Float32Array(analyser.fftSize); - setStatus("active"); - } catch { - setStatus("denied"); - } - }, [fftSize]); - - useEffect(() => { - if (autoStart) { - start(); - } - return stop; - }, [autoStart, start, stop]); - - useAnimationFrame(() => { - const analyser = analyserRef.current; - const buffer = bufferRef.current; - if (!(analyser && buffer)) { - return; - } - - analyser.getFloatTimeDomainData(buffer); - - let sumOfSquares = 0; - for (const sample of buffer) { - sumOfSquares += sample * sample; - } - const rms = Math.sqrt(sumOfSquares / buffer.length); - const target = Math.min(1, rms * RMS_TO_UNIT); - - const previous = amplitude.get(); - const factor = target > previous ? smoothing * ATTACK_FACTOR : smoothing; - amplitude.set(previous + (target - previous) * (1 - factor)); - }); - - return { amplitude, start, status, stop }; -}; - -/** - * Amplitude generator for demos, docs and previews — no microphone involved. - * - * Produces a plausible speech-like envelope whose energy follows the current - * {@link AIState}, so every example can show the reactive behaviour without - * asking the visitor for permissions. - */ -export const useSimulatedAmplitude = ( - state: AIState = "idle", -): MotionValue => { - const amplitude = useMotionValue(0); - const motion = getAIStateMotion(state); - - useAnimationFrame((time) => { - const t = time / 1000; - // Three detuned sines read as organic; a single sine reads as a metronome. - const envelope = - 0.5 + - 0.3 * Math.sin(t * 2.1 * motion.speed) + - 0.14 * Math.sin(t * 5.3 * motion.speed + 1.7) + - 0.06 * Math.sin(t * 11.7 * motion.speed + 0.4); - - amplitude.set(Math.min(1, Math.max(0, envelope * motion.intensity))); - }); - - return amplitude; -}; diff --git a/app/src/components/agents/orb/bot-orb-draw.ts b/app/src/components/agents/orb/bot-orb-draw.ts new file mode 100644 index 0000000000..f3e3f2b370 --- /dev/null +++ b/app/src/components/agents/orb/bot-orb-draw.ts @@ -0,0 +1,782 @@ +/** + * The orb, every Bot's face and OpenBot's own mark, drawn on a 2D canvas: a glass sphere of + * drifting colour with two eye-lights that look, blink and smile. It comes in nine colours. A Bot's + * look, its colour and the few small ways two Bots in one colour still differ, is picked from its + * seed; OpenBot's own is violet, which no Bot is given. It also plays a finish: the check and the + * burst of rays the orb shows when something the person allowed is done. + * + * Every frame is a pure function of an `OrbPose`, so the component that animates it owns time and + * this file owns only the picture. + */ + +const TAU = Math.PI * 2; +const PI = Math.PI; + +export const clamp = (x: number, a = 0, b = 1) => (x < a ? a : x > b ? b : x); +const lerp = (a: number, b: number, u: number) => a + (b - a) * u; +const eio = (u: number) => { + const v = clamp(u); + return v < 0.5 ? 4 * v * v * v : 1 - (-2 * v + 2) ** 3 / 2; +}; +const eout = (u: number) => 1 - (1 - clamp(u)) ** 3; +const smooth = (u: number) => { + const v = clamp(u); + return v * v * (3 - 2 * v); +}; +const bump = (u: number) => (u <= 0 || u >= 1 ? 0 : Math.sin(PI * u) ** 2); + +// ---------- motion: keyed tracks with eases and closed-form springs ---------- + +/** One move of a track: to `to` from time `at`, over an ease of `ease` seconds or on a spring of [stiffness, damping]. */ +export type TrackKey = + | { at: number; to: number; ease: number } + | { at: number; to: number; spring: readonly [number, number] }; + +/** A value that starts at `from` and moves through its keys in order. */ +export type Track = { from: number; keys: TrackKey[] }; + +/** Step response of a damped spring: closed form, so it is a pure function of elapsed time. */ +function springStep(tau: number, w: number, z: number) { + if (tau <= 0) return 0; + if (z >= 1) return 1 - Math.exp(-w * tau) * (1 + w * tau); + const wd = w * Math.sqrt(1 - z * z); + return ( + 1 - + Math.exp(-z * w * tau) * + (Math.cos(wd * tau) + ((z * w) / wd) * Math.sin(wd * tau)) + ); +} + +/** + * The track's value at time t. Each key moves the value from the previous key's value to its own, + * and the responses are summed, which for springs is exactly a spring chasing a stepped target: + * anticipation, overshoot and settle all fall out of the same maths. + */ +export function track(tr: Track, t: number) { + let v = tr.from; + let prev = tr.from; + for (const k of tr.keys) { + if (t < k.at) break; + const tau = t - k.at; + const r = + "ease" in k + ? eio(tau / k.ease) + : springStep(tau, k.spring[0], k.spring[1]); + v += (k.to - prev) * r; + prev = k.to; + } + return v; +} + +/** A blink at `tau` seconds after it starts: shut fast, open slower, 0.24 s in all. */ +export function blinkCurve(tau: number) { + if (tau < 0 || tau > 0.24) return 0; + return tau < 0.07 ? eout(tau / 0.07) : 1 - eio((tau - 0.07) / 0.17); +} + +/** A nod that rings down, `tau` seconds after it starts. */ +export function nodCurve(tau: number) { + return tau >= 0 && tau < 2.2 ? Math.exp(-3 * tau) * Math.sin(8 * tau) : 0; +} + +export function mulberry32(seed: number) { + let a = seed; + return () => { + a |= 0; + a = (a + 0x6d2b79f5) | 0; + let t = Math.imul(a ^ (a >>> 15), 1 | a); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +/** + * Where it looks while nothing needs it: mostly at you, now and then a glance aside and back. + * Each look snaps over 0.14 s and then holds, so the pauses are part of the motion. + */ +export function restLook(ta: number, period: number, seed: number) { + const k = Math.floor(ta / period); + const f = ta - k * period; + const at = (n: number): [number, number, number] => { + const r = mulberry32((seed * 7919 + n * 104729) | 0); + return r() < 0.55 + ? [0, 0, 0] + : [(r() * 2 - 1) * 0.7, (r() * 2 - 1) * 0.4, (r() * 2 - 1) * 0.24]; + }; + const a = at(k - 1); + const b = at(k); + const u = eout(f / 0.14); + return [lerp(a[0], b[0], u), lerp(a[1], b[1], u), lerp(a[2], b[2], u)]; +} + +// ---------- colour ---------- + +export type Rgb = [number, number, number]; + +function ok2rgb(L: number, C: number, h: number): Rgb | null { + const hr = (h * PI) / 180; + const a = C * Math.cos(hr); + const b = C * Math.sin(hr); + const l = (L + 0.3963377774 * a + 0.2158037573 * b) ** 3; + const m = (L - 0.1055613458 * a - 0.0638541728 * b) ** 3; + const s = (L - 0.0894841775 * a - 1.291485548 * b) ** 3; + const linear = [ + 4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s, + -1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s, + -0.0041960863 * l - 0.7034186147 * m + 1.707614701 * s, + ]; + const out: number[] = []; + for (const x of linear) { + if (x < -0.002 || x > 1.002) return null; + const c = clamp(x); + out.push( + Math.round( + 255 * (c <= 0.0031308 ? 12.92 * c : 1.055 * c ** (1 / 2.4) - 0.055), + ), + ); + } + return out as Rgb; +} + +/** An oklch colour as sRGB, losing chroma until it fits rather than clipping its hue. */ +export function oklch(L: number, C: number, h: number): Rgb { + let chroma = C; + for (let i = 0; i < 40; i++) { + const c = ok2rgb(L, chroma, h); + if (c) return c; + chroma *= 0.95; + } + return ok2rgb(L, 0, h) ?? [128, 128, 128]; +} + +const rgba = (c: Rgb, a: number) => + `rgba(${c[0]},${c[1]},${c[2]},${a <= 0 ? 0 : a >= 1 ? 1 : a.toFixed(3)})`; +const mix = (a: Rgb, b: Rgb, u: number): Rgb => [ + Math.round(lerp(a[0], b[0], u)), + Math.round(lerp(a[1], b[1], u)), + Math.round(lerp(a[2], b[2], u)), +]; +const WHITE: Rgb = [255, 255, 255]; + +type Oklch = readonly [L: number, C: number, h: number]; + +/** + * One of the orb's colours, as oklch [L, C, h]: four tones for the sphere's drifting colour and its + * pale base. `main` also tints its glow and, in the dark, the landing's ping and a finish's rays; + * `deep` draws those in the light. `shade` is the tint of the sphere's shaded rim, as sRGB. + */ +type PaletteSpec = { + main: Oklch; + side: Oklch; + deep: Oklch; + pale: Oklch; + base: Oklch; + shade: readonly [r: number, g: number, b: number]; +}; + +/** + * The nine colours: violet, OpenBot's own, and the eight a Bot's look is picked from by position. + * Adding, removing or reordering one changes the colour of most Bots. + */ +const PALETTES = { + violet: { + main: [0.7, 0.14, 288], + side: [0.72, 0.15, 318], + deep: [0.47, 0.2, 296], + pale: [0.86, 0.08, 308], + base: [0.93, 0.03, 300], + shade: [40, 16, 78], + }, + ember: { + main: [0.79, 0.13, 62], + side: [0.75, 0.15, 35], + deep: [0.53, 0.17, 30], + pale: [0.89, 0.08, 52], + base: [0.94, 0.03, 50], + shade: [84, 30, 14], + }, + moss: { + main: [0.77, 0.11, 182], + side: [0.78, 0.13, 152], + deep: [0.5, 0.12, 165], + pale: [0.89, 0.07, 160], + base: [0.94, 0.03, 160], + shade: [14, 60, 40], + }, + sun: { + main: [0.81, 0.15, 72], + side: [0.85, 0.14, 92], + deep: [0.57, 0.14, 58], + pale: [0.93, 0.08, 92], + base: [0.95, 0.03, 90], + shade: [92, 58, 10], + }, + berry: { + main: [0.7, 0.15, 328], + side: [0.72, 0.16, 352], + deep: [0.48, 0.18, 345], + pale: [0.86, 0.08, 350], + base: [0.93, 0.03, 350], + shade: [80, 14, 48], + }, + ocean: { + main: [0.74, 0.13, 205], + side: [0.75, 0.13, 232], + deep: [0.53, 0.2, 262], + pale: [0.86, 0.08, 220], + base: [0.93, 0.025, 235], + shade: [14, 30, 80], + }, + // Beside ocean on the wheel, so lighter and softer: a clear sky against ocean's deep water. + sky: { + main: [0.79, 0.11, 220], + side: [0.8, 0.1, 242], + deep: [0.55, 0.15, 245], + pale: [0.9, 0.06, 228], + base: [0.95, 0.02, 230], + shade: [10, 40, 76], + }, + coral: { + main: [0.78, 0.13, 32], + side: [0.74, 0.15, 8], + deep: [0.52, 0.17, 18], + pale: [0.88, 0.08, 22], + base: [0.94, 0.03, 20], + shade: [84, 20, 24], + }, + // Nearly grey: for a team big enough to want a quiet one. + slate: { + main: [0.72, 0.045, 232], + side: [0.76, 0.04, 250], + deep: [0.48, 0.05, 265], + pale: [0.88, 0.02, 255], + base: [0.93, 0.012, 258], + shade: [30, 38, 54], + }, +} as const satisfies Record; + +export type OrbPaletteName = keyof typeof PALETTES; + +/** Every colour, OpenBot's violet first. */ +export const ORB_PALETTES = Object.keys(PALETTES) as readonly OrbPaletteName[]; + +/** The eight colours a Bot can be: all but violet, so a Bot is never mistaken for OpenBot. */ +const BOT_PALETTES = ORB_PALETTES.filter((name) => name !== "violet"); + +/** + * How one orb is drawn apart from its motion: its colour, how far round its colour masses are + * turned, in radians, a nudge to its hue in degrees, and how far apart its eyes are, as a share of + * the usual spacing. + */ +export type OrbLook = { + palette: OrbPaletteName; + turn: number; + hue: number; + eyes: number; +}; + +/** OpenBot's own look: violet, untouched. */ +export const OPENBOT_LOOK: OrbLook = { + palette: "violet", + turn: 0, + hue: 0, + eyes: 1, +}; + +/** The most a Bot's hue is nudged either way, in degrees: a touch, never enough to pass for another colour. */ +const HUE_NUDGE = 4; +/** The most a Bot's eyes are moved apart or together, as a share of the usual spacing. */ +const EYE_NUDGE = 0.08; + +/** A 32-bit FNV-1a hash of the seed, mixed with murmur3's finaliser so every character moves every bit. */ +function hashSeed(seed: string): number { + let h = 0x811c9dc5; + for (let i = 0; i < seed.length; i++) { + h ^= seed.charCodeAt(i); + h = Math.imul(h, 0x01000193); + } + h ^= h >>> 16; + h = Math.imul(h, 0x85ebca6b); + h ^= h >>> 13; + h = Math.imul(h, 0xc2b2ae35); + h ^= h >>> 16; + return h >>> 0; +} + +/** + * The look for a Bot's seed. Eight colours repeat soon in a team, so the same hash also turns the + * colour masses, nudges the hue and spaces the eyes, and two Bots in one colour still look apart. + * Stable across reloads and releases, so a Bot keeps its face. + */ +export function orbLookFor(seed: string): OrbLook { + const h = hashSeed(seed); + // The eight colours take the low three bits, and each of the rest the next eight. + const share = (from: number) => ((h >>> from) & 0xff) / 256; + return { + palette: BOT_PALETTES[h % BOT_PALETTES.length] as OrbPaletteName, + turn: share(3) * TAU, + hue: Math.round((share(11) * 2 - 1) * HUE_NUDGE), + eyes: 1 + (share(19) * 2 - 1) * EYE_NUDGE, + }; +} + +/** A colour's tones as sRGB, ready to draw. */ +export type Palette = Record; +const paletteMemo = new Map(); + +/** A colour with its saturation and hue nudged, as the orb livens when it listens and cools when it works. */ +export function palette( + name: OrbPaletteName, + sat: number, + hue: number, +): Palette { + const ks = Math.round(clamp(sat, 0.5, 1.3) * 50) / 50; + const kh = Math.round(hue * 2) / 2; + const key = `${name}|${ks}|${kh}`; + let p = paletteMemo.get(key); + if (!p) { + const spec: PaletteSpec = PALETTES[name]; + const c = ([L, C, h]: Oklch, sk = ks, hk = kh) => oklch(L, C * sk, h + hk); + p = { + main: c(spec.main), + side: c(spec.side), + deep: c(spec.deep), + pale: c(spec.pale), + base: c(spec.base, 1, kh * 0.5), + shade: [...spec.shade], + }; + paletteMemo.set(key, p); + } + return p; +} + +// ---------- the pose ---------- + +/** Everything one frame depends on. `ta` is seconds on the orb's own clock; ages are -1 when nothing happened yet. */ +export type OrbPose = { + ta: number; + /** 0 at rest, 1 listening (can overshoot a little). */ + listen: number; + think: number; + /** The landing's glow, 0 to 1. */ + done: number; + smile: number; + tilt: number; + /** Where it looks: right and down, about -1 to 1. */ + ax: number; + ay: number; + blink: number; + nod: number; + doneAge: number; + noticeAge: number; + /** Seconds into a finish (a check and a burst of rays, `finishPhase`), or -1 when none is playing. */ + finishAge: number; + /** -1 to 1, a slow breath. */ + breath: number; +}; + +// ---------- the finish ---------- + +/** How long a finish plays, in seconds, from the eyes giving way to the check until they are back. */ +export const FINISH_S = 1.2; +/** A moment in a finish when the check is whole and the rays have gone: the finish as a still. */ +export const FINISH_HELD = 0.8; +/** The short strokes that burst out round the orb when it finishes. */ +export const RAYS = 10; + +/** Where a finish is at one moment. Every part of it is a pure function of its age. */ +export type FinishPhase = { + /** How much of the check is drawn along its stroke, 0 to 1. */ + check: number; + /** The check's opacity: whole while it holds, gone as the eyes come back. */ + checkAlpha: number; + /** How far the rays have flown, 0 to 1, or -1 before they leave and once they have faded. */ + rays: number; + /** The eye-lights' opacity: they give way to the check and come back after it. */ + eyes: number; + /** The sphere's pop as the check lands, 0 to 1. */ + pop: number; +}; + +const NO_FINISH: FinishPhase = { + check: 0, + checkAlpha: 0, + rays: -1, + eyes: 1, + pop: 0, +}; + +/** + * A finish `age` seconds in: the eyes fade as the sphere pops and the check strokes on, the rays + * burst outward and fade, the check holds, then fades as the eyes come back. -1 is no finish. + */ +export function finishPhase(age: number): FinishPhase { + if (!(age >= 0 && age < FINISH_S)) return NO_FINISH; + const back = smooth((age - 0.85) / 0.3); + return { + check: eout((age - 0.06) / 0.3), + checkAlpha: 1 - back, + rays: age >= 0.1 && age < 0.75 ? (age - 0.1) / 0.65 : -1, + eyes: age < 0.85 ? 1 - smooth(age / 0.12) : back, + pop: bump(age / 0.5), + }; +} + +// ---------- the orb's pieces ---------- + +type Ctx = CanvasRenderingContext2D; + +function blob(ctx: Ctx, x: number, y: number, r: number, c: Rgb, a: number) { + if (a <= 0.003 || r <= 0) return; + const g = ctx.createRadialGradient(x, y, 0, x, y, r); + g.addColorStop(0, rgba(c, a)); + g.addColorStop(0.42, rgba(c, a * 0.78)); + g.addColorStop(0.76, rgba(c, a * 0.24)); + g.addColorStop(1, rgba(c, 0)); + ctx.fillStyle = g; + ctx.fillRect(x - r, y - r, 2 * r, 2 * r); +} + +function light(ctx: Ctx, x: number, y: number, r: number, a: number) { + if (a <= 0.003 || r <= 0) return; + const g = ctx.createRadialGradient(x, y, 0, x, y, r); + g.addColorStop(0, `rgba(255,255,255,${clamp(a).toFixed(3)})`); + g.addColorStop(0.45, `rgba(255,255,255,${clamp(a * 0.42).toFixed(3)})`); + g.addColorStop(1, "rgba(255,255,255,0)"); + ctx.fillStyle = g; + ctx.fillRect(x - r, y - r, 2 * r, 2 * r); +} + +function circle(ctx: Ctx, r: number) { + ctx.beginPath(); + ctx.arc(0, 0, r, 0, TAU); +} + +// angle, orbit, radius of the mesh's four colour masses +const MESH = [ + [3.95, 0.43, 0.88], + [5.5, 0.45, 0.74], + [0.78, 0.45, 0.86], + [2.36, 0.45, 0.88], +] as const; + +function meshPos(ta: number, fast: boolean, turn: number) { + const rot = turn + (fast ? 1.45 * ta + 1.1 : 0.22 * ta); + return MESH.map((b, i) => { + const a = + b[0] + + rot + + (fast + ? 0.5 * Math.sin(1.1 * ta + i * 1.93) + : 0.42 * Math.sin(0.29 * ta + i * 1.93)); + return [Math.cos(a) * b[1], Math.sin(a) * b[1], b[2]] as const; + }); +} + +/** + * Four soft colour masses drifting round a pale base, starting `turn` radians round; a faster swirl + * joins them while it works. + */ +function mesh( + ctx: Ctx, + R: number, + S: OrbPose, + pal: Palette, + turn: number, + lx: number, + ly: number, +) { + ctx.fillStyle = rgba(pal.base, 1); + ctx.fillRect(-R * 1.7, -R * 1.7, R * 3.4, R * 3.4); + const cols = [pal.main, pal.pale, pal.side, pal.deep]; + meshPos(S.ta, false, turn).forEach((p, i) => { + blob(ctx, p[0] * R + lx * R, p[1] * R + ly * R, p[2] * R, cols[i], 0.94); + }); + const swirl = S.think * 0.55; + if (swirl > 0.01) { + meshPos(S.ta, true, turn).forEach((p, i) => { + blob( + ctx, + p[0] * R + lx * R * 0.6, + p[1] * R + ly * R * 0.6, + p[2] * R * 0.92, + cols[i], + 0.9 * swirl, + ); + }); + } +} + +/** Lit top edge, shaded bottom: what makes the disc read as a sphere. Call inside the clip. */ +function rim(ctx: Ctx, R: number, shade: Rgb, dark: boolean) { + const g = ctx.createRadialGradient( + 0, + -R * 0.3, + R * 0.55, + 0, + -R * 0.12, + R * 1.16, + ); + g.addColorStop(0, rgba(shade, 0)); + g.addColorStop(0.72, rgba(shade, 0)); + g.addColorStop(1, rgba(shade, dark ? 0.3 : 0.24)); + ctx.fillStyle = g; + ctx.fillRect(-R * 1.3, -R * 1.3, R * 2.6, R * 2.6); + const lw = Math.max(1, R * 0.05); + const lg = ctx.createLinearGradient(0, -R, 0, R * 0.25); + lg.addColorStop(0, "rgba(255,255,255,.7)"); + lg.addColorStop(1, "rgba(255,255,255,0)"); + ctx.strokeStyle = lg; + ctx.lineWidth = lw; + circle(ctx, R - lw * 0.5); + ctx.stroke(); +} + +function glow(ctx: Ctx, R: number, amt: number, c: Rgb, dark: boolean) { + if (amt < 0.005) return; + const r = R * 1.85; + const g = ctx.createRadialGradient(0, 0, R * 0.55, 0, 0, r); + g.addColorStop(0, rgba(c, amt * (dark ? 0.8 : 0.5))); + g.addColorStop(0.38, rgba(c, amt * (dark ? 0.34 : 0.18))); + g.addColorStop(1, rgba(c, 0)); + ctx.fillStyle = g; + ctx.fillRect(-r, -r, 2 * r, 2 * r); +} + +/** How long the landing's ping rings, in seconds. */ +const PING_S = 1; + +/** One ring that leaves the orb and fades: the landing's ping. */ +function ping(ctx: Ctx, r0: number, age: number, pal: Palette, dark: boolean) { + if (!(age >= 0 && age < PING_S)) return; + const u = age / PING_S; + const r = r0 * (1.02 + (1.5 - 1.02) * eout(u)); + const a = (1 - u) ** 1.5; + ctx.strokeStyle = rgba( + dark ? mix(pal.main, WHITE, 0.2) : pal.deep, + a * (dark ? 0.75 : 0.62), + ); + ctx.lineWidth = Math.max(1, r0 * 0.07 * (1 - u * 0.6)); + circle(ctx, r); + ctx.stroke(); +} + +/** + * The finish's burst: a ring of short strokes, one straight up, that fly out round the sphere, + * shortening and fading as they go. Every other one is longer, so the ring sparkles rather than ticks. + */ +function rays(ctx: Ctx, r0: number, u: number, pal: Palette, dark: boolean) { + if (!(u >= 0 && u < 1)) return; + const travel = eout(u); + const a = (1 - u) ** 1.4; + ctx.save(); + ctx.lineCap = "round"; + ctx.strokeStyle = rgba( + dark ? mix(pal.main, WHITE, 0.25) : pal.deep, + a * (dark ? 0.9 : 0.8), + ); + ctx.lineWidth = Math.max(1, r0 * 0.09 * (1 - 0.5 * u)); + for (let i = 0; i < RAYS; i++) { + const angle = -PI / 2 + (TAU * i) / RAYS; + const length = r0 * (i % 2 === 0 ? 0.3 : 0.2) * (1 - 0.55 * u); + const inner = r0 * (1.3 + 0.32 * travel); + const dx = Math.cos(angle); + const dy = Math.sin(angle); + ctx.beginPath(); + ctx.moveTo(dx * inner, dy * inner); + ctx.lineTo(dx * (inner + length), dy * (inner + length)); + ctx.stroke(); + } + ctx.restore(); +} + +// The check's three points, as shares of the sphere's radius: in at the left, down to the elbow, up +// to the right. Its two legs are about 0.37 and 0.75 of the radius long. +const CHECK = [ + [-0.38, 0.02], + [-0.12, 0.28], + [0.42, -0.26], +] as const; + +/** The check, stroked on along its length to `drawn` (0 to 1), in the eye-lights' white. */ +function check( + ctx: Ctx, + R: number, + drawn: number, + alpha: number, + dark: boolean, +) { + if (drawn <= 0 || alpha <= 0.003) return; + const [a, b, c] = CHECK.map(([x, y]) => [x * R, y * R] as const); + const first = Math.hypot(b[0] - a[0], b[1] - a[1]); + const second = Math.hypot(c[0] - b[0], c[1] - b[1]); + const along = drawn * (first + second); + ctx.save(); + ctx.globalAlpha = clamp(alpha); + ctx.lineCap = "round"; + ctx.lineJoin = "round"; + ctx.lineWidth = Math.max(1.6, R * 0.17); + ctx.strokeStyle = "rgba(255,255,255,.97)"; + ctx.shadowColor = dark ? "rgba(8,6,20,.4)" : "rgba(20,16,33,.28)"; + ctx.shadowBlur = Math.max(1, R * 0.07); + ctx.beginPath(); + ctx.moveTo(a[0], a[1]); + if (along <= first) { + const u = along / first; + ctx.lineTo(lerp(a[0], b[0], u), lerp(a[1], b[1], u)); + } else { + const u = (along - first) / second; + ctx.lineTo(b[0], b[1]); + ctx.lineTo(lerp(b[0], c[0], u), lerp(b[1], c[1], u)); + } + ctx.stroke(); + ctx.restore(); +} + +/** The sphere's scale: it breathes, swells a little to listen and to land, and pops as a check lands. */ +function scaleOf(S: OrbPose, finish: FinishPhase) { + const L = clamp(S.listen, 0, 1.25); + return ( + 1 + + 0.015 * S.breath * (1 - S.think) + + 0.025 * L + + 0.04 * S.done + + 0.06 * finish.pop + ); +} + +function eyePath(ctx: Ctx, w: number, h: number, smile: number) { + const rx = (w / 2) * (1 + 0.12 * smile); + const ry = h / 2; + const e = 0.32 * ry * smile; + const top = -ry; + const bot = lerp(ry, -0.36 * ry, smile); + const ct = (top - 0.25 * e) / 0.75; + const cb = (bot - 0.25 * e) / 0.75; + ctx.beginPath(); + ctx.moveTo(-rx, e); + ctx.bezierCurveTo(-rx, ct, rx, ct, rx, e); + ctx.bezierCurveTo(rx, cb, -rx, cb, -rx, e); + ctx.closePath(); +} + +/** How long an orb stays perked up after the pointer comes close, in seconds. */ +const PERK_S = 1.6; + +/** The sphere: the mesh turned toward where it looks, its rim, and two eye-lights that blink and smile. */ +function sphere( + ctx: Ctx, + R: number, + S: OrbPose, + pal: Palette, + look: OrbLook, + dark: boolean, + size: number, +) { + const na = S.noticeAge; + const perk = + na >= 0 && na < PERK_S ? Math.exp(-2.6 * na) * smooth(na / 0.1) : 0; + const finish = finishPhase(S.finishAge); + const sc = scaleOf(S, finish); + ctx.save(); + ctx.translate(0, S.nod * R * 0.11); + ctx.scale(sc, sc); + circle(ctx, R); + ctx.clip(); + const yaw = clamp(S.ax, -1.1, 1.1) * 0.62; + const pitch = -clamp(S.ay, -1.1, 1.1) * 0.5; + const roll = S.tilt; + mesh(ctx, R, S, pal, look.turn, Math.sin(yaw) * 0.2, -Math.sin(pitch) * 0.16); + light(ctx, -0.42 * R, -0.5 * R, 0.5 * R, 0.26); + light(ctx, 0, R * 0.05, R * 0.9, 0.28 * S.done); + rim(ctx, R, pal.shade, dark); + const small = size < 40; + const widen = 1 + 0.16 * perk; + const open = clamp(1 - 0.94 * S.blink) * (1 - 0.3 * S.think) * widen; + const cr = Math.cos(roll); + const sr = Math.sin(roll); + const ew = Math.max(small ? 2.4 : 1.8, R * 0.16 * widen); + const eh = Math.max(small ? 3.4 : 2.4, R * 0.3); + for (const side of [-1, 1]) { + if (finish.eyes <= 0.003) break; + const lon = side * 0.35 * look.eyes; + const lat = 0.05 + 0.04 * S.smile; + const x = Math.sin(lon) * Math.cos(lat); + const y = Math.sin(lat); + const z = Math.cos(lon) * Math.cos(lat); + const y1 = y * Math.cos(pitch) + z * Math.sin(pitch); + const z1 = -y * Math.sin(pitch) + z * Math.cos(pitch); + const x2 = x * Math.cos(yaw) + z1 * Math.sin(yaw); + const sx = x2 * cr - y1 * sr; + const sy = x2 * sr + y1 * cr; + const fx = Math.sqrt(clamp(1 - sx * sx, 0.12, 1)); + const fy = Math.sqrt(clamp(1 - sy * sy, 0.2, 1)); + const h = Math.max(small ? 1.1 : 0.9, eh * open * fy); + ctx.save(); + ctx.globalAlpha = finish.eyes; + ctx.translate(sx * R * 0.84, -sy * R * 0.84 + (eh - h) * 0.28); + ctx.rotate(-roll); + ctx.shadowColor = dark ? "rgba(8,6,20,.4)" : "rgba(20,16,33,.28)"; + ctx.shadowBlur = Math.max(1, R * 0.07); + ctx.fillStyle = "rgba(255,255,255,.97)"; + eyePath(ctx, ew * fx, h, clamp(S.smile) * (1 - S.blink)); + ctx.fill(); + ctx.restore(); + } + check(ctx, R, finish.check, finish.checkAlpha, dark); + ctx.restore(); +} + +/** + * The whole orb with `look`, centred on the context's origin, for a footprint of `size` CSS px. + * Its glow, the landing's ping and a finish's rays spill past the footprint, so draw it on a canvas + * twice its size. + */ +export function drawBotOrb( + ctx: Ctx, + size: number, + S: OrbPose, + { dark, look }: { dark: boolean; look: OrbLook }, +) { + const finish = finishPhase(S.finishAge); + const pal = palette( + look.palette, + 0.9 + 0.14 * clamp(S.listen) + 0.08 * S.think + 0.12 * clamp(S.done), + look.hue - 6 * S.think, + ); + const r = (size / 2) * 0.72; + const sc = scaleOf(S, finish); + const glowAmt = clamp( + 0.18 + + 0.42 * clamp(S.listen) + + 0.22 * S.think + + 0.55 * S.done + + 0.45 * finish.pop, + ); + glow(ctx, r * sc, glowAmt * 0.85, mix(pal.main, pal.deep, 0.4), dark); + ping(ctx, r, S.doneAge, pal, dark); + rays(ctx, r, finish.rays, pal, dark); + sphere(ctx, r, S, pal, look, dark, size * 0.72); +} + +/** How long each age in a pose shows, in seconds: after that the orb draws as if it never happened. */ +const SHOWS_FOR: Partial> = { + doneAge: PING_S, + noticeAge: PERK_S, + finishAge: FINISH_S, +}; + +/** + * Whether two poses draw the same picture, every value within a thousandth of the other's. An age + * counts only while what it times still shows, so an orb that has settled poses the same picture + * frame after frame and need not draw it again. + */ +export function samePicture(a: OrbPose, b: OrbPose): boolean { + for (const key of Object.keys(a) as (keyof OrbPose)[]) { + const shows = SHOWS_FOR[key]; + const seen = (v: number) => + shows === undefined || (v >= 0 && v < shows) ? v : -1; + if (Math.abs(seen(a[key]) - seen(b[key])) >= 1e-3) return false; + } + return true; +} diff --git a/app/src/components/agents/orb/bot-orb.tsx b/app/src/components/agents/orb/bot-orb.tsx new file mode 100644 index 0000000000..d628b9cd45 --- /dev/null +++ b/app/src/components/agents/orb/bot-orb.tsx @@ -0,0 +1,542 @@ +import { useEffect, useMemo, useRef, useSyncExternalStore } from "react"; +import { cn } from "@/lib/utils"; +import { + blinkCurve, + clamp, + drawBotOrb, + FINISH_HELD, + FINISH_S, + mulberry32, + nodCurve, + OPENBOT_LOOK, + type OrbLook, + type OrbPose, + orbLookFor, + restLook, + samePicture, + type Track, + type TrackKey, + track, +} from "./bot-orb-draw"; +import { pointer, subscribeFrame, subscribePointer } from "./orb-clock"; + +/** + * What the orb is doing: resting, listening to you type, working, or landing a finished thing. + * `finished` is a moment rather than a state: something the person allowed is done, and the orb + * plays a check with a burst of rays once, over whatever it was doing, which carries on under it. + * Hold it for `FINISH_MS`, then pass the state it was in before. + */ +export type BotOrbState = + | "idle" + | "listening" + | "thinking" + | "done" + | "finished"; + +/** How long a finish plays: the check, its rays, and the eyes coming back. */ +export const FINISH_MS = FINISH_S * 1000; + +type Look = readonly [number, number]; +const LOOK_AT_YOU: Look = [0, 0.04]; +/** Up and aside, as if thinking it over. */ +const LOOK_UP: Look = [-0.36, -0.5]; +/** Down at what you are typing. */ +const LOOK_DOWN: Look = [0.1, 0.6]; +const LOOKS: Record = { + idle: LOOK_AT_YOU, + listening: LOOK_DOWN, + thinking: LOOK_UP, + done: LOOK_AT_YOU, + finished: LOOK_AT_YOU, +}; +/** Seconds between blinks, before each one's jitter. */ +const BLINK_EVERY = 4.2; +/** How far the canvas draws past the footprint on every side, as a share of the size. */ +const SPILL = 0.5; +/** Backing pixels per CSS pixel, at most: sharp on a 3× screen without a huge canvas. */ +const MAX_DPR = 3; +/** Below this footprint, in CSS px, an orb only blinks and watches: no glances, no breath, no drift. */ +const SMALL = 40; +/** + * The least time between two frames of a small orb, in ms: about thirty a second, plenty for eyes a + * few px across, where a fast screen would otherwise draw every face in a list 120 times a second. + */ +const SMALL_FRAME_MS = 30; +/** The moment on its clock a still is composed at, and where a small orb's colour holds. */ +const STILL_T = 1.6; +/** How much of its look an orb gives the pointer: all of it at rest, some while it listens or works. */ +const WATCH: Record = { + idle: 1, + listening: 0.5, + thinking: 0.35, + done: 1, + finished: 1, +}; +/** The pointer perks it up within this share of its size from its centre… */ +const NOTICE_NEAR = 0.9; +/** …and has to go past this one before it can perk it up again. */ +const NOTICE_FAR = 1.3; + +/** Every track the state drives, and when the things that play once last happened. */ +type Motion = { + listen: Track; + think: Track; + done: Track; + smile: Track; + ax: Track; + ay: Track; + doneAt: number; + noticedAt: number; + finishedAt: number; + nods: number[]; +}; + +const still = (v: number): Track => ({ from: v, keys: [] }); + +/** The motion at rest in `state`, as it is on the first frame. */ +function settled(state: BotOrbState): Motion { + const look = LOOKS[state]; + return { + listen: still(state === "listening" ? 1 : 0), + think: still(state === "thinking" ? 1 : 0), + done: still(0), + smile: still(state === "done" ? 1 : 0), + ax: still(look[0]), + ay: still(look[1]), + doneAt: -1, + noticedAt: -1, + finishedAt: state === "finished" ? 0 : -1, + nods: [], + }; +} + +/** The track restarted from its value at `now`, heading for the new key. */ +const retarget = (tr: Track, now: number, key: TrackKey): Track => ({ + from: track(tr, now), + keys: [key], +}); + +/** Moves every track toward `state` from time `now`, each on its own spring or ease. */ +function enter(m: Motion, state: BotOrbState, now: number): Motion { + const look = LOOKS[state]; + const gaze = + state === "thinking" + ? ([6.5, 0.85] as const) + : state === "listening" + ? ([15, 0.6] as const) + : ([12, 0.8] as const); + const next: Motion = { + ...m, + listen: retarget( + m.listen, + now, + state === "listening" + ? { at: now, to: 1, spring: [14, 0.5] } + : { at: now, to: 0, ease: 0.45 }, + ), + think: retarget(m.think, now, { + at: now, + to: state === "thinking" ? 1 : 0, + ease: state === "thinking" ? 0.4 : 0.45, + }), + smile: retarget(m.smile, now, { + at: now + (state === "done" ? 0.08 : 0), + to: state === "done" ? 1 : 0, + ease: state === "done" ? 0.4 : 0.8, + }), + ax: retarget(m.ax, now, { at: now, to: look[0], spring: gaze }), + ay: retarget(m.ay, now, { at: now, to: look[1], spring: gaze }), + done: retarget(m.done, now, { at: now, to: 0, ease: 0.45 }), + }; + if (state === "listening") next.noticedAt = now; + if (state === "done") { + // It lands: a glow that springs up and settles away, a ping and a nod. + next.done = { + from: track(m.done, now), + keys: [ + { at: now, to: 1, spring: [11, 0.42] }, + { at: now + 1.4, to: 0, ease: 1.2 }, + ], + }; + next.doneAt = now; + next.nods = [now + 0.1]; + } + return next; +} + +/** + * The pose at `t` seconds on the orb's clock. `seed` keeps two orbs from blinking and glancing in + * step. A small orb keeps still apart from its blinks, its colour included, so a list of them + * doesn't fidget and, once its eyes settle, has nothing new to draw. + */ +function poseAt( + m: Motion, + t: number, + seed: number, + { reduced, small }: { reduced: boolean; small: boolean }, +): OrbPose { + const listen = track(m.listen, t); + const think = clamp(track(m.think, t)); + // Left alone it glances about now and then; while it listens or works it keeps its eyes on that. + const calm = reduced || small ? 0 : (1 - clamp(listen)) * (1 - think); + const glance = restLook(t, 3.6, seed); + let blink = 0; + if (!reduced) { + const k = Math.floor(t / BLINK_EVERY); + for (const n of [k - 1, k]) { + const at = n * BLINK_EVERY + 1.3 + mulberry32(seed + n)() * 1.6; + blink = Math.max(blink, blinkCurve(t - at)); + } + } + return { + ta: small ? STILL_T : t, + listen, + think, + done: Math.max(0, track(m.done, t)), + smile: clamp(track(m.smile, t)), + tilt: 0.5 * glance[2] * calm, + ax: track(m.ax, t) + 0.6 * glance[0] * calm, + ay: track(m.ay, t) + 0.6 * glance[1] * calm, + blink, + nod: m.nods.reduce((n, at) => n + nodCurve(t - at), 0), + doneAge: m.doneAt < 0 ? -1 : t - m.doneAt, + noticeAge: m.noticedAt < 0 ? -1 : t - m.noticedAt, + // Still, a finish is its check without the burst. + finishAge: m.finishedAt < 0 ? -1 : reduced ? FINISH_HELD : t - m.finishedAt, + breath: small ? 0 : Math.sin((Math.PI * 2 * t) / 5.5), + }; +} + +/** + * An orb watching the pointer: the look it is heading for (`tx`, `ty`), where its eyes are on a + * spring (`x`, `y`, with their speeds), how much of its look it gives the pointer (`w`), whether it + * can see the pointer at all, and whether the pointer is close enough to have perked it up. `epoch` + * and `last` are the pointer's epoch and the time when it last looked. + */ +export type Gaze = { + tx: number; + ty: number; + x: number; + y: number; + vx: number; + vy: number; + w: number; + seen: boolean; + near: boolean; + epoch: number; + last: number; +}; + +export const restingGaze = (): Gaze => ({ + tx: 0, + ty: 0, + x: 0, + y: 0, + vx: 0, + vy: 0, + w: 0, + seen: false, + near: false, + epoch: -1, + last: -1, +}); + +type Box = Pick; + +/** + * Moves `gaze` on to time `t` for an orb in `state`, measuring where the orb is on screen only when + * the pointer, the scroll or the window has changed. True on the frame the pointer comes close, + * which perks the orb up; a listening orb has already noticed you, so it never perks. + */ +export function followPointer( + gaze: Gaze, + t: number, + state: BotOrbState, + measure: () => Box, +): boolean { + const dt = gaze.last < 0 ? 0 : Math.min(1 / 30, Math.max(0, t - gaze.last)); + gaze.last = t; + let perked = false; + if (gaze.epoch !== pointer.epoch) { + gaze.epoch = pointer.epoch; + const box = measure(); + const dx = pointer.x - (box.left + box.width / 2); + const dy = pointer.y - (box.top + box.height / 2); + const d = Math.hypot(dx, dy) || 1; + // Right over it, it looks straight out at you; the further away, the further round it turns. + const reach = d / (d + Math.max(48, box.width * 1.6)); + gaze.tx = (dx / d) * reach * 0.95; + gaze.ty = (dy / d) * reach; + gaze.seen = pointer.on && box.width > 0; + const near = + gaze.seen && d < box.width * (gaze.near ? NOTICE_FAR : NOTICE_NEAR); + perked = near && !gaze.near && state !== "listening"; + gaze.near = near; + } + gaze.vx += ((gaze.tx - gaze.x) * 150 - gaze.vx * 19) * dt; + gaze.vy += ((gaze.ty - gaze.y) * 150 - gaze.vy * 19) * dt; + gaze.x += gaze.vx * dt; + gaze.y += gaze.vy * dt; + gaze.w += ((gaze.seen ? WATCH[state] : 0) - gaze.w) * (1 - Math.exp(-dt * 5)); + return perked; +} + +/** Its own look, glances and all, gives way to the pointer by as much as it is watching. */ +function lookAt(pose: OrbPose, gaze: Gaze): OrbPose { + return { + ...pose, + ax: pose.ax + (gaze.x - pose.ax) * gaze.w, + ay: pose.ay + (gaze.y - pose.ay) * gaze.w, + tilt: pose.tilt + (-0.12 * gaze.x - pose.tilt) * gaze.w, + }; +} + +/** + * Sizes the canvas for a footprint of `size` and returns how to draw a pose on it with `look`, or + * null where there is no 2D context to draw with. A pose that would draw what the canvas already + * shows, in the same theme and at the same density, is skipped. + */ +function surface(canvas: HTMLCanvasElement, size: number, look: OrbLook) { + const ctx = canvas.getContext("2d"); + if (!ctx) return null; + const box = size * (1 + 2 * SPILL); + let dpr = 0; + let shown: { pose: OrbPose; dark: boolean } | null = null; + return (pose: OrbPose) => { + // Read on every draw: a window dragged to another screen, or a zoom, changes it under an orb + // that stays mounted, and an orb drawn for the old one goes soft. + const next = Math.min(MAX_DPR, window.devicePixelRatio || 1); + const dark = document.documentElement.classList.contains("dark"); + if (next === dpr && shown?.dark === dark && samePicture(shown.pose, pose)) { + return; + } + shown = { pose, dark }; + if (next !== dpr) { + dpr = next; + canvas.width = Math.round(box * dpr); + canvas.height = Math.round(box * dpr); + } + ctx.setTransform(dpr, 0, 0, dpr, 0, 0); + ctx.clearRect(0, 0, box, box); + ctx.translate(box / 2, box / 2); + drawBotOrb(ctx, size, pose, { dark, look }); + }; +} + +/** + * Calls `onChange` whenever the screen's pixel density changes, for a still orb that nothing else + * redraws. A resolution query matches only the density it names, so it is asked again after each change. + */ +function watchResolution(onChange: () => void): () => void { + if (typeof window.matchMedia !== "function") return () => {}; + let query: MediaQueryList | null = null; + const arm = () => { + query?.removeEventListener("change", fire); + query = window.matchMedia( + `(resolution: ${window.devicePixelRatio || 1}dppx)`, + ); + query.addEventListener("change", fire); + }; + const fire = () => { + arm(); + onChange(); + }; + arm(); + return () => query?.removeEventListener("change", fire); +} + +const REDUCE_MOTION = "(prefers-reduced-motion: reduce)"; + +/** + * Whether the person asks for less motion, kept current: turning it on while the app is open stills + * every orb already on screen, not only the ones drawn afterwards. + */ +function usePrefersReducedMotion(): boolean { + return useSyncExternalStore( + (onChange) => { + if (typeof window.matchMedia !== "function") return () => {}; + const query = window.matchMedia(REDUCE_MOTION); + query.addEventListener("change", onChange); + return () => query.removeEventListener("change", onChange); + }, + () => + typeof window.matchMedia === "function" && + window.matchMedia(REDUCE_MOTION).matches, + () => false, + ); +} + +let staggers = 0; +const secondsSince = (start: number) => (performance.now() - start) / 1000; + +/** + * A Bot's orb, with the look its `seed` picks; with no seed, OpenBot's own, in violet. `size` is its + * footprint in CSS px; its glow spills past it. It breathes and glances about when idle, watches + * what you type when listening, swirls while thinking, and lands with a ping and a smile when done. + * Passed `finished`, its eyes give way to a check while short rays burst out round it, and then it + * is back as it was. Wherever the pointer is, its eyes follow it: fully at rest, partly while it + * listens or works, with a small perk when the pointer comes close. An orb under 40 px only blinks + * and watches, at about thirty frames a second. + * + * Every orb on the page draws on one shared animation frame, and only while it is on screen, the + * tab is shown and its picture has changed. With reduced motion, or `still`, it is a still in the + * state's pose, drawn again only when the state or the theme changes, and its eyes stay put. + */ +export function BotOrb({ + className, + seed, + size, + state = "idle", + still = false, +}: { + className?: string; + seed?: string; + size: number; + state?: BotOrbState; + still?: boolean; +}) { + const boxRef = useRef(null); + const canvasRef = useRef(null); + const reduced = usePrefersReducedMotion() || still; + const look = useMemo( + () => (seed === undefined ? OPENBOT_LOOK : orbLookFor(seed)), + [seed], + ); + // Each orb's own, so two orbs on a page, even two of one Bot, don't blink or glance in step. + const clock = useRef<{ start: number; stagger: number } | null>(null); + clock.current ??= { start: performance.now(), stagger: ++staggers * 7 }; + const motion = useRef(null); + const shown = useRef(null); + const current = useRef(state); + const gaze = useRef(null); + gaze.current ??= restingGaze(); + + // It starts at rest in its first state, except that a first "done" still lands. After that a + // change moves every track from where it is, so a change mid-motion never jumps. A finish moves + // no track: it plays over the state it was in, and coming back to that state changes nothing. + useEffect(() => { + current.current = state; + if (!clock.current) return; + const t = secondsSince(clock.current.start); + if (state === "finished") { + motion.current = { + ...(motion.current ?? settled("idle")), + finishedAt: t, + }; + return; + } + if (shown.current === state) return; + const before = motion.current; + motion.current = + shown.current === null && state !== "done" + ? { ...settled(state), finishedAt: before?.finishedAt ?? -1 } + : enter(before ?? settled("idle"), state, t); + shown.current = state; + }, [state]); + + // A still: a composed picture in the state's pose, eyes open, its look held, no ping; finished, + // the check in place of the eyes, with no burst, for as long as it is held. Nothing draws it + // again on its own, so it is redrawn when the theme flips the root's `dark` class. + useEffect(() => { + const canvas = canvasRef.current; + const draw = reduced && canvas ? surface(canvas, size, look) : null; + if (!draw || !clock.current) return; + const { stagger } = clock.current; + const held = + state === "finished" + ? { ...settled(shown.current ?? "idle"), finishedAt: 0 } + : settled(state); + const paint = () => + draw( + poseAt(held, STILL_T, stagger, { reduced: true, small: size < SMALL }), + ); + paint(); + const theme = new MutationObserver(paint); + theme.observe(document.documentElement, { + attributes: true, + attributeFilter: ["class"], + }); + const unwatch = watchResolution(paint); + return () => { + theme.disconnect(); + unwatch(); + }; + }, [reduced, size, look, state]); + + // Alive: drawn on the shared clock while it is on screen. It counts as on screen until an + // IntersectionObserver says otherwise, and always where there is none to ask. It reads the theme + // on every frame, so a flip shows on the next one; a frame that would draw what is already there + // is skipped, so a small orb at rest with the pointer still draws nothing between blinks. + useEffect(() => { + const canvas = canvasRef.current; + const box = boxRef.current; + const draw = !reduced && canvas ? surface(canvas, size, look) : null; + if (!draw || !box || !clock.current || !gaze.current) return; + const { start, stagger } = clock.current; + const eyes = gaze.current; + const small = size < SMALL; + const measure = () => box.getBoundingClientRect(); + let ticked = Number.NEGATIVE_INFINITY; + const frame = (now: number) => { + if (small && now - ticked < SMALL_FRAME_MS) return; + ticked = now; + const t = Math.max(0, (now - start) / 1000); + const state = current.current; + if (followPointer(eyes, t, state, measure) && motion.current) { + motion.current = { ...motion.current, noticedAt: t }; + } + const pose = poseAt(motion.current ?? settled(state), t, stagger, { + reduced: false, + small, + }); + draw(lookAt(pose, eyes)); + }; + let unsubscribe: (() => void) | null = null; + const onScreen = (on: boolean) => { + if (on && !unsubscribe) unsubscribe = subscribeFrame(frame); + if (!on && unsubscribe) { + unsubscribe(); + unsubscribe = null; + } + }; + const sight = + "IntersectionObserver" in window + ? new IntersectionObserver((entries) => + // The newest entry is where it is now. + onScreen(entries.at(-1)?.isIntersecting ?? false), + ) + : null; + sight?.observe(box); + onScreen(true); + const unwatch = subscribePointer(); + return () => { + sight?.disconnect(); + onScreen(false); + unwatch(); + }; + }, [reduced, size, look]); + + const spill = size * SPILL; + return ( + + + + ); +} diff --git a/app/src/components/agents/orb/orb-clock.ts b/app/src/components/agents/orb/orb-clock.ts new file mode 100644 index 0000000000..176196bbfb --- /dev/null +++ b/app/src/components/agents/orb/orb-clock.ts @@ -0,0 +1,140 @@ +/** + * What every orb on the page shares, so a list of fifty Bots costs one animation loop and one set of + * listeners rather than fifty of each: a clock that asks the browser for one frame and draws every + * orb in it, and where the pointer is, for the orbs whose eyes follow it. + * + * Both start with their first subscriber and stop with their last, so nothing is left running or + * listening on a page with no orb on it. + */ + +type FrameCallback = (now: number) => void; + +const subscribers = new Set(); +let frame: number | null = null; +/** The document whose visibility the clock follows, while it has subscribers. */ +let watched: Document | null = null; + +function run(now: number) { + frame = null; + // A Set visits what is added during the loop and skips what is removed, so an orb that unmounts + // mid-frame is not drawn after it has gone. + for (const callback of subscribers) callback(now); + schedule(); +} + +function schedule() { + if (frame !== null || subscribers.size === 0 || document.hidden) return; + frame = window.requestAnimationFrame(run); +} + +function halt() { + if (frame !== null) window.cancelAnimationFrame(frame); + frame = null; +} + +/** A hidden tab draws nothing, and picks up again where its orbs are now once it is shown. */ +function onVisibility() { + if (document.hidden) halt(); + else schedule(); +} + +/** + * Calls `callback` once on every animation frame, with the frame's timestamp, until the returned + * function is called. Every subscriber is drawn in the same frame; no frame is asked for while the + * tab is hidden or nobody is subscribed. + */ +export function subscribeFrame(callback: FrameCallback): () => void { + subscribers.add(callback); + if (!watched) { + watched = document; + watched.addEventListener("visibilitychange", onVisibility); + } + schedule(); + return () => { + if (!subscribers.delete(callback) || subscribers.size > 0) return; + halt(); + watched?.removeEventListener("visibilitychange", onVisibility); + watched = null; + }; +} + +// ---------- the pointer ---------- + +type PointerState = { x: number; y: number; on: boolean; epoch: number }; + +const state: PointerState = { x: 0, y: 0, on: false, epoch: 0 }; + +/** + * Where the pointer is, in client px, and whether it is on the page at all. `epoch` counts every + * move, scroll and resize, so an orb measures itself again only when something has changed. Kept + * up to date only while something is subscribed with `subscribePointer`. + */ +export const pointer: Readonly = state; + +let watchers = 0; +let lift: ReturnType | undefined; +let listening: Window | null = null; + +const aim = (event: PointerEvent) => { + state.x = event.clientX; + state.y = event.clientY; + state.on = true; + state.epoch++; + clearTimeout(lift); +}; +const away = () => { + state.on = false; + state.epoch++; +}; +// A finger lifts off the glass; a mouse stays where it was. Give a tap a moment of being watched. +const lifted = (event: PointerEvent) => { + if (event.pointerType !== "mouse") lift = setTimeout(away, 1600); +}; +const left = (event: PointerEvent) => { + if (!event.relatedTarget) away(); +}; +const moved = () => { + state.epoch++; +}; + +function listen(target: Window) { + target.addEventListener("pointermove", aim, { passive: true }); + target.addEventListener("pointerdown", aim, { passive: true }); + target.addEventListener("pointerup", lifted, { passive: true }); + target.document.addEventListener("pointerout", left, { passive: true }); + target.addEventListener("blur", away); + target.addEventListener("scroll", moved, { capture: true, passive: true }); + target.addEventListener("resize", moved); +} + +function unlisten(target: Window) { + target.removeEventListener("pointermove", aim); + target.removeEventListener("pointerdown", aim); + target.removeEventListener("pointerup", lifted); + target.document.removeEventListener("pointerout", left); + target.removeEventListener("blur", away); + target.removeEventListener("scroll", moved, { capture: true }); + target.removeEventListener("resize", moved); + clearTimeout(lift); + // Nobody is following it now, so what it last knew is stale for whoever subscribes next. + away(); +} + +/** + * Keeps `pointer` up to date until the returned function is called: one set of window listeners for + * every orb that watches. + */ +export function subscribePointer(): () => void { + if (watchers++ === 0) { + listening = window; + listen(listening); + } + let subscribed = true; + return () => { + if (!subscribed) return; + subscribed = false; + if (--watchers > 0 || !listening) return; + unlisten(listening); + listening = null; + }; +} diff --git a/app/src/components/app-sidebar/app-sidebar.tsx b/app/src/components/app-sidebar/app-sidebar.tsx index 27fe657b18..c4420d4139 100644 --- a/app/src/components/app-sidebar/app-sidebar.tsx +++ b/app/src/components/app-sidebar/app-sidebar.tsx @@ -3,7 +3,8 @@ import { useInfiniteQuery, useQuery } from "@tanstack/react-query"; import { Link, type LinkOptions, useParams } from "@tanstack/react-router"; import { AnimatePresence, motion, useReducedMotion } from "motion/react"; import type * as React from "react"; -import { useRef, useState } from "react"; +import { type RefObject, useRef, useState } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; import { DropdownMenu, DropdownMenuContent, @@ -25,16 +26,25 @@ import { SidebarMenuButton, SidebarMenuItem, SidebarRail, + useSidebar, } from "@/components/ui/sidebar"; +import { + Tooltip, + TooltipContent, + TooltipTrigger, +} from "@/components/ui/tooltip"; import { currentUserQueryOptions } from "@/lib/auth/queries"; import { type ChannelSummary, channelListQueryOptions, + conversationsOnly, } from "@/lib/channels/queries"; import { useChannelEvents } from "@/lib/channels/use-channel-events"; import { appConfig } from "@/lib/generated/application-config"; +import { getHotkey, hotkeyLabel } from "@/lib/hotkeys/hotkeys"; +import { useHotkey } from "@/lib/hotkeys/use-hotkey"; import { EASE_OUT, ENTRANCE_SECONDS } from "@/lib/motion"; -import { navigationFor } from "@/lib/navigation"; +import { type NavigationItem, useNavigationItems } from "@/lib/navigation"; import { relativeTime } from "@/lib/relative-time"; import { type MessageListEmphasis, @@ -44,7 +54,7 @@ import { BotsNavBadge } from "../bot-profile/attention"; import { BrowserNavItem } from "../browser/browser-nav-item"; import { Button } from "../ui/button"; import { Empty, EmptyDescription, EmptyHeader, EmptyTitle } from "../ui/empty"; -import { Channel } from "./channel"; +import { Channel, LIT } from "./channel"; import { ChannelPagination } from "./channel-pagination"; import { useSignOut } from "./use-sign-out"; @@ -52,6 +62,11 @@ const appLinkOptions = { to: "/" } satisfies LinkOptions; const userMenuItemClassName = "gap-2 px-2 py-1.5"; +/** A control in the sidebar lifts to the card's fill and shadow under the pointer, as a row does. */ +const LIT_ON_HOVER = "hover:bg-sidebar-accent hover:shadow-(--shadow-card)"; + +const NEW_CHAT_KEYS = hotkeyLabel(getHotkey("new-chat").combo); + function UserAvatar() { const { data: currentUser } = useQuery(currentUserQueryOptions()); const initials = @@ -198,17 +213,86 @@ function ChannelRow({ ); } +/** + * A count beside a destination, drawn as the Bots count is so the two read as one kind of thing. + * Nothing for zero. + */ +function NavigationBadge({ count }: { count: number }) { + if (count <= 0) return null; + return ( + + {count} + + ); +} + +/** One destination in the sidebar's footer, from the shared list, with its count when it has one. */ +export function NavigationEntry({ item }: { item: NavigationItem }) { + return ( + + ( + + )} + > +
+ +
+ {item.label} + {/* What waits on you across every Bot; Bots lists them first. */} + {item.id === "bots" ? : null} + {item.badge ? : null} + + + ); +} + +/** + * Puts the cursor in the search box as soon as it is on the page, with what it holds selected so + * typing replaces the last search. On a narrow window the sidebar is a sheet that only mounts once + * it opens, and a frame later moves focus to its first control; trying again on each of the next + * few frames lands after both. + */ +function focusWhenDrawn(box: RefObject, frames = 10) { + const input = box.current; + if (input?.isConnected) { + input.focus(); + input.select(); + if (document.activeElement === input) return; + } + if (frames > 0) requestAnimationFrame(() => focusWhenDrawn(box, frames - 1)); +} + export function AppSidebar({ ...props }: React.ComponentProps) { const emphasis = useMessageListEmphasis(); const { data: currentUser } = useQuery(currentUserQueryOptions()); const signOut = useSignOut(); const channels = useInfiniteQuery(channelListQueryOptions()); + // A task's own conversation is listed under Tasks, never here. + const conversations = channels.data && conversationsOnly(channels.data); // One socket for the app, opened where the roster is kept live. useChannelEvents(); const [search, setSearch] = useState(""); + const searchBox = useRef(null); const scrollRoot = useRef(null); + const { isMobile, setOpen, setOpenMobile } = useSidebar(); + /* + * ⌘K opens the sidebar, or its sheet on a narrow window, with the cursor in its search box. Bound + * here, so it works wherever this sidebar is: not on Settings or Admin, which have sidebars of + * their own, and not under a skin, which draws its own frame. + */ + useHotkey("search", () => { + if (isMobile) setOpenMobile(true); + else setOpen(true); + focusWhenDrawn(searchBox); + }); const searching = search.trim().length > 0; - const visibleChannels = pinnedFirst(matchingChannels(channels.data, search)); + const visibleChannels = pinnedFirst(matchingChannels(conversations, search)); /* * FILTERING DOES NOT ANIMATE. Rows exit and relayout on every keystroke otherwise, which is a * list thrashing under somebody who is still typing — and the moving target is the very thing @@ -216,8 +300,8 @@ export function AppSidebar({ ...props }: React.ComponentProps) { * occasional; this is not. */ const animateOrder = - !searching && (channels.data?.length ?? 0) <= MAX_ANIMATED_ROWS; - const navigation = navigationFor(currentUser?.role); + !searching && (conversations?.length ?? 0) <= MAX_ANIMATED_ROWS; + const navigation = useNavigationItems(); return ( @@ -225,29 +309,44 @@ export function AppSidebar({ ...props }: React.ComponentProps) { ( - {appConfig.brand.productName} + {/* OpenBot's own mark, clipped to its circle as every face below it is. */} + + + + {appConfig.brand.productName} )} /> - + + ( + + )} + /> + } + > + + + {/* An accelerator nobody is told about is not a feature. */} + + New conversation + {NEW_CHAT_KEYS} + + @@ -255,9 +354,10 @@ export function AppSidebar({ ...props }: React.ComponentProps) { - + setSearch(event.target.value)} placeholder="Search…" value={search} @@ -292,7 +392,7 @@ export function AppSidebar({ ...props }: React.ComponentProps) { ) : null} - {!searching && channels.data?.length === 0 ? ( + {!searching && conversations?.length === 0 ? (
@@ -324,28 +424,11 @@ export function AppSidebar({ ...props }: React.ComponentProps) { + {/* In the list's own order, so a destination goes where `lib/navigation.ts` puts it. */} {navigation .filter((item) => item.section === "primary") .map((item) => ( - - ( - - )} - > -
- -
- {item.label} - {/* What waits on you across every Bot; Bots lists them first. */} - {item.id === "bots" ? : null} - - + ))} {/* Your own browser, with the Recorder in it. Drawn only where OpenBot can run one for you. */} @@ -353,11 +436,13 @@ export function AppSidebar({ ...props }: React.ComponentProps) { + } > - + {currentUser?.name || currentUser?.email} diff --git a/app/src/components/app-sidebar/channel-item-content.tsx b/app/src/components/app-sidebar/channel-item-content.tsx index ece35115c2..1a8538b810 100644 --- a/app/src/components/app-sidebar/channel-item-content.tsx +++ b/app/src/components/app-sidebar/channel-item-content.tsx @@ -26,8 +26,9 @@ export function ChannelItemContent({ pinned?: boolean; revealing?: boolean; }) { - const primaryText = `text-[0.9rem] leading-5 tracking-[-1%] ${unread ? "font-semibold" : "font-medium"}`; - const secondaryText = "text-[12px] leading-4 text-muted-foreground"; + // On the type scale: the line that leads at the sub size, the other a caption. + const primaryText = `type-sub ${unread ? "font-semibold" : "font-medium"}`; + const secondaryText = "type-caption text-muted-foreground"; return (
@@ -46,15 +47,23 @@ export function ChannelItemContent({ > {name} - - {lastMessageAt} + {/* + * The sidebar row's "…" button sits over this end of the line while it shows: under the + * pointer, with focus in the row, or with its menu open. The time and marks step aside + * for it. Under 768px the button always shows and the row keeps room for it instead. The + * preview in Settings is in no menu item, so none of this reaches it. + */} + + + {lastMessageAt} + + {unread ? ( + + ) : null} + {pinned ? ( + + ) : null} - {unread ? ( - - ) : null} - {pinned ? ( - - ) : null}
void; + onDelete: () => void; +}) { + return ( + <> + + {pinned ? : } + {pinned ? "Unpin conversation" : "Pin conversation"} + + + + Delete conversation… + + + ); +} + /** * Memoized roster row. `use-channel-events` preserves unchanged row identity, and * `content-visibility` keeps off-screen rows cheap without virtualization. * - * Right-click opens Pin and Delete. Deleting is confirmed in a dialog that names the channel, - * because the row it was invoked on is one of several identical-looking rows. + * Pin and Delete open from right-click, and from the "…" button over the row's end. Right-click + * alone was found by nobody who was not told, and reached by no keyboard. The button shows under + * the pointer and while focus is in the row, and comes straight after it in tab order. Deleting is + * confirmed in a dialog that names the channel, because the row it was invoked on is one of several + * identical-looking rows. */ export const Channel = memo(function Channel({ channelId, @@ -70,8 +128,18 @@ export const Channel = memo(function Channel({ */ const [pinProblem, setPinProblem] = useState(null); + const title = summary || lastMessage || "New conversation"; + const pin = () => { + setPinProblem(null); + setPinned.mutate( + { channelId, pinned: !pinned }, + { onError: (thrown) => setPinProblem(thrown.message) }, + ); + }; + const askToDelete = () => setConfirming(true); + return ( - <> + - { - setPinProblem(null); - setPinned.mutate( - { channelId, pinned: !pinned }, - { onError: (thrown) => setPinProblem(thrown.message) }, - ); - }} - > - {pinned ? : } - {pinned ? "Unpin conversation" : "Pin conversation"} - - setConfirming(true)} - > - - Delete conversation… - + + + + } + > + + + + + + {pinProblem ? (

{pinProblem} @@ -137,6 +212,6 @@ export const Channel = memo(function Channel({ onOpenChange={setConfirming} open={confirming} /> - + ); }); diff --git a/app/src/components/approvals/approval-card.tsx b/app/src/components/approvals/approval-card.tsx new file mode 100644 index 0000000000..f3f55adfe3 --- /dev/null +++ b/app/src/components/approvals/approval-card.tsx @@ -0,0 +1,301 @@ +import { + IconCheck, + IconChevronRight, + IconCircleCheck, + IconMail, + IconShieldCheck, + IconTerminal2, + IconWorld, + IconX, +} from "@tabler/icons-react"; +import { Fragment, type ReactNode, useId } from "react"; +import { Button } from "@/components/ui/button"; +import { + type ApprovalView, + alwaysAllowReach, + approvalOutcome, + approvalPlace, + clockTime, + decisionReceipt, + describeApproval, + type OutcomeTone, +} from "@/lib/approval-view"; +import type { ApprovalDecision, ApprovalRequest } from "@/lib/approvals"; +import { cn } from "@/lib/utils"; + +const KIND_ICON = { + mail: IconMail, + browser: IconWorld, + command: IconTerminal2, + other: IconShieldCheck, +} as const; + +const PILL_TONES: Record = { + done: "bg-status-done-soft text-status-done", + working: "bg-status-working-soft text-status-working", + needs: "bg-status-needs-soft text-status-needs", + problem: "bg-status-problem-soft text-status-problem", + neutral: "bg-muted text-muted-foreground", +}; + +/** Decline and Allow once: half the card wide each, a finger's height tall. */ +const CHOICE = + "h-(--target) min-w-0 flex-1 basis-28 rounded-full px-4 type-sub font-semibold"; + +/** + * One action a Bot wants to take, and the person's answer to it: what it is, where, who it goes to + * and what it says, then Decline and Allow once. "Always allow here" is a quiet button under the + * two, because it saves a rule for this Bot, this action and this place rather than answering only + * this request; the line under it says how far that rule reaches. A hand-off is the person's to do, + * so it asks "I did it myself" or "Don't do it". + * + * Once decided the card folds: the title, a pill saying how it ended (Allowed, Sent, Done, + * Declined, Didn't work), and a line saying who decided and when. While it waits, everything the + * request carries is one tap away under Details. + * + * `frame="card"` draws it as a card of its own, for a conversation. `frame="row"` leaves the frame + * to the list it sits in, a row of a page's `PageRows` card. + */ +export function ApprovalCard({ + request, + botName, + busy = false, + error, + onDecide, + frame = "card", + className, +}: { + request: ApprovalRequest; + /** Named where the card is shown away from the Bot's own conversation. */ + botName?: string; + busy?: boolean; + error?: string | null; + onDecide?: (choice: ApprovalDecision) => void; + frame?: "card" | "row"; + className?: string; +}) { + const view = describeApproval(request); + const waiting = request.status === "pending"; + const handOff = request.action.policy?.behaviour === "hand_off"; + const outcome = approvalOutcome(request); + const receipt = waiting ? null : decisionReceipt(request); + const place = approvalPlace(request); + const caption = [botName, view.app].filter(Boolean).join(" · "); + const Icon = KIND_ICON[view.kind]; + const reachId = useId(); + + return ( +

+
+ + + +
+

+ {handOff && waiting ? `This needs you: ${view.title}` : view.title} +

+ {caption || place ? ( +

+ {caption} + {caption && place ? " · " : null} + {/* Shown whole, however long: the person is deciding about this exact place. */} + {place ? ( + {place} + ) : null} +

+ ) : null} +
+ {outcome ? ( + + {outcome.tone === "done" ? ( + + ) : null} + {outcome.label} + + ) : null} +
+ {waiting ? ( + + ) : null} + {error ? ( +

+ {error} +

+ ) : null} + {waiting && onDecide ? ( +
+
+ + +
+ {handOff ? null : ( + <> + +

+ {alwaysAllowReach(request)} +

+ + )} +
+ ) : null} + {receipt ? ( + + {receipt} + + ) : null} +
+ ); +} + +/** The fields, the body, why the Bot asked, and everything else it would send. */ +function ApprovalContent({ + request, + view, + handOff, +}: { + request: ApprovalRequest; + view: ApprovalView; + handOff: boolean; +}) { + const detailsId = useId(); + return ( + <> + {view.fields.map((field) => ( + + +
+ + {field.label} + + + {field.value} + +
+ + ))} + {view.body ? ( + <> + + {/* Whole, not clipped: this is what the person is agreeing to send or type. */} +

+ {view.body} +

+ + ) : null} + {request.action.policy?.reason ? ( +

+ {handOff ? "Handed to you: " : ""} + {request.action.policy.reason} +

+ ) : null} + {/* Open by itself when the face above leaves out something the request would send. */} +
0 || undefined}> + + Details + + + 
+          {JSON.stringify(request.action.args, null, 2)}
+
+
+ + ); +} + +/** The line between a card's parts, short of its edges like the rows of a list. */ +function Hairline() { + return
; +} + +/** Who decided and when, under a decided card. */ +function DecidedLine({ + children, + at, + declined, +}: { + children: ReactNode; + at: string | null; + declined: boolean; +}) { + const Mark = declined ? IconX : IconCircleCheck; + return ( + <> + +
+ + {children} + {at ? ( + + ) : null} +
+ + ); +} diff --git a/app/src/components/approvals/inline-approval.tsx b/app/src/components/approvals/inline-approval.tsx index ab179e1f8b..dd0bfdecd9 100644 --- a/app/src/components/approvals/inline-approval.tsx +++ b/app/src/components/approvals/inline-approval.tsx @@ -1,18 +1,27 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import type { ReactNode } from "react"; -import { Button } from "@/components/ui/button"; +import { personWasAsked } from "@/lib/approval-view"; import { - type ApprovalDecision, - approvalInboxOptions, - decideApproval, + type ApprovalRequest, + approvalInboxCacheOptions, + decideApprovalMutationOptions, } from "@/lib/approvals"; +import { ApprovalCard } from "./approval-card"; + +/** A request the person decided on a card, which keeps its folded card in the conversation. */ +const decidedByPerson = (request: ApprovalRequest) => + request.status !== "pending" && + Boolean(request.decision) && + personWasAsked(request); /** - * A pending approval, drawn in the conversation where the action it holds would have been. + * An approval, drawn in the conversation where the action it holds would have been. * - * The same request as the Approvals page, read from the same owner-scoped inbox, so a decision made - * in either place is the one decision. Once decided the server runs the action exactly once and - * continues the conversation, and this gives the line back to the tool's own renderer. + * The same request as the Bot's page, read from the same owner-scoped inbox, so a decision made in + * either place is the one decision. Once decided the server runs the action exactly once and + * continues the conversation, and this gives the line back to the tool's own renderer, under the + * folded card saying how it ended and who decided. A request the person was never asked about (a + * saved rule allowed it) leaves no card behind. */ export function InlineApproval({ toolCallId, @@ -22,89 +31,59 @@ export function InlineApproval({ fallback: ReactNode; }) { const cache = useQueryClient(); - const inbox = useQuery(approvalInboxOptions()); - const decision = useMutation({ - mutationFn: (choice: ApprovalDecision) => - decideApproval(request?.id ?? "", choice), - onSuccess: () => cache.invalidateQueries({ queryKey: ["approvals"] }), - }); + const inbox = useQuery(approvalInboxCacheOptions()); + const decision = useMutation(decideApprovalMutationOptions(cache)); const request = toolCallId ? inbox.data?.requests.find( (candidate) => candidate.action.toolCallId === toolCallId, ) : undefined; - if (request?.status !== "pending") return <>{fallback}; - const handOff = request.action.policy?.behaviour === "hand_off"; - const choices: [ApprovalDecision, string][] = handOff - ? [ - ["handled", "I did it myself"], - ["deny", "Don't do it"], - ] - : [ - ["allow_once", "Allow once"], - ["allow_always", "Always allow here"], - ["deny", "Deny"], - ]; - return ( -
-

- {handOff ? "This needs you" : "Waiting for your approval"}:{" "} - {request.action.toolRef - .replace(/^computer_|^host\//, "") - .replaceAll("_", " ")}{" "} - on {request.action.scope} -

- {request.action.policy ? ( -

{request.action.policy.reason}

- ) : null} - {decision.error ? ( -

- {decision.error.message} -

- ) : null} -
- {choices.map(([choice, label], index) => ( - - ))} + if (!request) return <>{fallback}; + if (request.status !== "pending") { + if (!decidedByPerson(request)) return <>{fallback}; + return ( +
+ + {fallback}
-
+ ); + } + return ( + decision.mutate({ id: request.id, choice })} + request={request} + /> ); } /** - * The pending approvals for a group of tool calls, for a transcript that draws those calls itself - * rather than through each tool's renderer: the browser-activity group folds every click and field - * into one collapsed line, so a card inside a step would never be seen. Drawn beside the group, - * outside the fold, one card per waiting call; nothing at all when none is waiting. + * The approval cards for tool calls a transcript draws itself rather than through each tool's + * renderer: a run of steps folded into one line, whose cards would never be seen inside the fold, + * and a server call alone in its turn, which has no renderer to draw one. Drawn beside the line, one + * card per call the person was asked about: waiting to be decided, or folded once it was. Nothing + * at all for calls nobody asked about. */ -export function PendingApprovals({ +export function ToolCallApprovals({ toolCallIds, }: { toolCallIds: readonly string[]; }) { - const inbox = useQuery(approvalInboxOptions()); - const waiting = new Set( + const inbox = useQuery(approvalInboxCacheOptions()); + const asked = new Set( (inbox.data?.requests ?? []) - .filter((request) => request.status === "pending") + .filter( + (request) => request.status === "pending" || decidedByPerson(request), + ) .flatMap((request) => request.action.toolCallId ? [request.action.toolCallId] : [], ), ); - const ids = toolCallIds.filter((id) => waiting.has(id)); + const ids = toolCallIds.filter((id) => asked.has(id)); if (ids.length === 0) return null; return ( -
+
{ids.map((id) => ( ))} diff --git a/app/src/components/approvals/waiting.tsx b/app/src/components/approvals/waiting.tsx index b101dddd51..b9b209f23e 100644 --- a/app/src/components/approvals/waiting.tsx +++ b/app/src/components/approvals/waiting.tsx @@ -1,4 +1,4 @@ -import { IconHandStop, IconMessageQuestion } from "@tabler/icons-react"; +import { IconMessageQuestion } from "@tabler/icons-react"; import { useMutation, useQuery } from "@tanstack/react-query"; import { Fragment, useState } from "react"; import { PageRows, PageSection } from "@/components/layout/page-shell"; @@ -20,37 +20,54 @@ import { decideApprovalMutationOptions, } from "@/lib/approvals"; import { queryClient } from "@/query-client"; +import { ApprovalCard } from "./approval-card"; /** - * Questions and actions waiting on the person, one row each, decided where it is shown: the answer - * box and the decision buttons sit on the row's own footer line. Given a Bot, only that Bot's. The - * same requests the conversation draws inline, so a decision made in either place is the one - * decision. + * Questions and actions waiting on the person, one row each, decided where it is shown: a question's + * answer box sits on its row's footer line, and an action is the same approval card the + * conversation draws, so a decision made in either place is the one decision. An action answered + * here stays, folded to how it went, until the page is left. Given a Bot, only that Bot's. + * + * Given a `title`, it is its own section of a page, drawn only while something is listed. */ -export function WaitingForYou({ agentId }: { agentId?: string }) { +export function WaitingForYou({ + agentId, + title, +}: { + agentId?: string; + title?: string; +}) { const inbox = useQuery(approvalInboxOptions()); const agents = useQuery(agentListQueryOptions()); const botName = (id: string) => agents.data?.find((agent) => agent.id === id)?.name ?? id; const decision = useMutation(decideApprovalMutationOptions(queryClient)); + const [decidedHere, setDecidedHere] = useState>( + () => new Set(), + ); const [answers, setAnswers] = useState>({}); const answer = useMutation(answerPersonQuestionMutationOptions(queryClient)); const ours = (botId: string) => agentId === undefined || botId === agentId; const questions = (inbox.data?.questions ?? []).filter((question) => ours(question.conversationBotId ?? question.botId), ); - const pending = (inbox.data?.requests ?? []).filter( - (request) => request.status === "pending" && ours(request.action.botId), + const requests = (inbox.data?.requests ?? []).filter( + (request) => + ours(request.action.botId) && + (request.status === "pending" || decidedHere.has(request.id)), ); - const error = inbox.error ?? decision.error ?? answer.error; - return ( + // A decision's failure is said on its own card. + const error = inbox.error ?? answer.error; + const listed = questions.length + requests.length > 0; + if (title && !listed) return null; + const list = ( <> {error ? (

{error.message}

) : null} - {questions.length + pending.length > 0 ? ( + {listed ? ( {questions.map((question, index) => ( @@ -97,103 +114,32 @@ export function WaitingForYou({ agentId }: { agentId?: string }) { ))} - {pending.map((request, index) => ( + {requests.map((request, index) => ( {questions.length + index > 0 ? : null} - - - - - - - {botName(request.action.botId)} wants to{" "} - {request.action.toolRef - .replace(/^computer_|^host\//, "") - .replaceAll("_", " ")} - - - {request.action.scope} - - {request.action.policy ? ( -

- {request.action.policy.behaviour === "hand_off" - ? "Handed to you: " - : ""} - {request.action.policy.reason} -

- ) : null} - 
-                    {JSON.stringify(request.action.args, null, 2)}
-
- - {request.action.policy?.behaviour === "hand_off" ? ( - - - - - ) : ( - - - - - - )} - + { + // Kept here, folded, once answered: the person sees how it went. + setDecidedHere((prior) => new Set(prior).add(request.id)); + decision.mutate({ id: request.id, choice }); + }} + request={request} + /> ))} ) : null} ); + return title ? {list} : list; } /** @@ -201,19 +147,5 @@ export function WaitingForYou({ agentId }: { agentId?: string }) { * waits: the sidebar's badge already said whether to look. */ export function BotNeedsYou({ agentId }: { agentId: string }) { - const inbox = useQuery(approvalInboxOptions()); - const waiting = - (inbox.data?.questions ?? []).some( - (question) => (question.conversationBotId ?? question.botId) === agentId, - ) || - (inbox.data?.requests ?? []).some( - (request) => - request.status === "pending" && request.action.botId === agentId, - ); - if (!waiting) return null; - return ( - - - - ); + return ; } diff --git a/app/src/components/bot-profile/access.tsx b/app/src/components/bot-profile/access.tsx index 6b53a54dce..755791f57d 100644 --- a/app/src/components/bot-profile/access.tsx +++ b/app/src/components/bot-profile/access.tsx @@ -76,7 +76,8 @@ function connectorTools(plugins: GrantedPlugins): Map { const connectors = new Map(); for (const tool of plugins.tools) { const key = tool.ref.split("/")[0] ?? tool.ref; - let label = readToolName(tool.toolName).label; + // What the tool does, not what it did: "Send your email", never "Sent your email". + let label = readToolName(tool.toolName, "plain").label; const prefix = `${key.toLowerCase()} `; if (label.toLowerCase().startsWith(prefix)) { const rest = label.slice(prefix.length); diff --git a/app/src/components/bot-profile/activity.tsx b/app/src/components/bot-profile/activity.tsx index 91c0fad637..d30e4a1ec3 100644 --- a/app/src/components/bot-profile/activity.tsx +++ b/app/src/components/bot-profile/activity.tsx @@ -38,6 +38,7 @@ import { botActivityQueryOptions, } from "@/lib/bot-lifecycle/queries"; import { relativeTime } from "@/lib/relative-time"; +import { taskListQueryOptions } from "@/lib/tasks/queries"; import { cn } from "@/lib/utils"; import { queryClient } from "@/query-client"; @@ -59,8 +60,16 @@ const KIND_LABEL: Record = { follow_up: "Follow-up", }; -/** Where an item opens: its conversation when it has one, otherwise the screen that owns its kind. */ -function linkFor(item: ActivityItem, agentId: string) { +/** + * Where an item opens: the task it was raised in, its conversation when it has one, otherwise the + * screen that owns its kind. + */ +function linkFor( + item: ActivityItem, + agentId: string, + taskId: string | undefined, +) { + if (taskId) return ; if (item.channelId) return ( @@ -90,8 +99,13 @@ function ActivityRow({ }) { const stop = useMutation(stopHandoffMutationOptions(queryClient)); const cancel = useMutation(cancelFollowUpMutationOptions(queryClient)); + // A task's conversation is the task's: what was raised there is answered on its page. + const tasks = useQuery(taskListQueryOptions()); + const taskId = item.channelId + ? tasks.data?.find((task) => task.channelId === item.channelId)?.id + : undefined; const Icon = ICONS[item.kind]; - const link = linkFor(item, agentId); + const link = linkFor(item, agentId, taskId); const when = relativeTime(item.at); const control = item.stoppable ? (
); })} @@ -106,25 +114,15 @@ function GroupFaces({ } /** - * Three bouncing dots in a small badge, ringed in the sidebar's own colour so it sits on the - * avatar as a badge rather than floating over it. The staggered negative delays start each dot at - * a different point in the same bounce, which is what makes the three read as one wave. + * A still dot in the working colour, ringed so it sits on the avatar as a badge rather than floating + * over it. Still on purpose: an orb this small only blinks, and reduced motion stills it entirely, so + * the faces working is too slight to read as busy from across the roster. The ring is the ground the + * avatar sits on, the sidebar's own unless a row lit behind it sets `--avatar-ground`. */ -function TypingBadge() { +function BusyDot() { return ( -
+ Working… - - - -
- ); -} - -function Dot({ className }: { className?: string }) { - return ( - + ); } diff --git a/app/src/components/channels/browser-activity.tsx b/app/src/components/channels/browser-activity.tsx index 4995fdf2d8..9476433d66 100644 --- a/app/src/components/channels/browser-activity.tsx +++ b/app/src/components/channels/browser-activity.tsx @@ -1,17 +1,12 @@ +import { + BROWSER_STEP_DONE as labels, + BROWSER_STEP_PLAIN as plainLabels, +} from "@/lib/computers/step-names"; +import { toolResultHeldBack } from "@/lib/plugins/tool-result"; import type { VisibleChatItem } from "./chat-messages"; type BrowserStep = Extract; -const labels: Record = { - computer_navigate: "Opened page", - computer_read: "Read page", - computer_snapshot: "Inspected page", - computer_click: "Clicked on page", - computer_type: "Filled in field", - computer_key: "Pressed key", - computer_scroll: "Scrolled page", -}; - /** Whether a tool call is one of the browser steps a tool group draws with its page link. */ export function isBrowserStep(item: BrowserStep): boolean { return Object.hasOwn(labels, item.toolCall.function.name); @@ -49,6 +44,7 @@ export function browserStepDetails(step: BrowserStep) { step.result?.startsWith("Error:") === true; return { label: labels[step.toolCall.function.name], + plain: plainLabels[step.toolCall.function.name], href: url?.href, title: navigation && typeof result.title === "string" && result.title.trim() @@ -60,3 +56,41 @@ export function browserStepDetails(step: BrowserStep) { pending: step.result === undefined, }; } + +/** + * How a browser step came out, from its result alone. + * + * ONE READING FOR BOTH PLACES A STEP IS SHOWN. The conversation's tool line (`stepDetails`) and the + * computer pane's step log (`step-log.tsx`) both read a step here, so a click the person declined + * never reads as done in one of them and declined in the other. + * + * A step the person held back is read before any of the browser's own fields, with + * `toolResultHeldBack`: declined on its card, or not asked again because the same request was still + * waiting (`alreadyWaiting`). The server says either in words for the model, sometimes inside the + * browser's JSON, and it would otherwise read as a step that worked or one that failed. `pending` is + * a step with no result yet, or only the note that it went to its approval card. Whether that is + * running, waiting for the person or cut off is for the caller to say, from the turn and the + * approvals inbox, and so is whether a request `alreadyWaiting` repeated is still waiting. + */ +export type BrowserOutcome = + | "pending" + | "declined" + | "alreadyWaiting" + | "stopped" + | "blocked" + | "notYet" + | "failed" + | "done"; + +export function browserOutcome(step: BrowserStep): BrowserOutcome { + if (step.result === undefined) return "pending"; + const result = objectOf(step.result); + if (result.waiting === true) return "pending"; + const heldBack = toolResultHeldBack(step.result); + if (heldBack) return heldBack; + // Cut off, like a call that never got its result: nothing went wrong with the page. + if (result.stopped === true) return "stopped"; + if (result.refused === true) return "blocked"; + if (result.ok === false && result.starting === true) return "notYet"; + return browserStepDetails(step).failed ? "failed" : "done"; +} diff --git a/app/src/components/channels/channel-chat.tsx b/app/src/components/channels/channel-chat.tsx index 317647c970..876a4fa211 100644 --- a/app/src/components/channels/channel-chat.tsx +++ b/app/src/components/channels/channel-chat.tsx @@ -309,6 +309,31 @@ function describeAttachments(attachments: readonly Attachment[]): string { return `Sent ${attachments.length} attachments`; } +/** + * A task's own conversation, open on its page (components/tasks/task-page.tsx). + * + * The task's turns run on the server, so nothing typed here runs in this tab: `send` hands it to the + * server, which queues it behind whatever the task is doing and records it under `messageId`. The + * person's bubble is drawn under that id at once and kept until the transcript has it. + */ +export type ChannelTaskMode = { + /** A turn runs or waits on the server for this task: the Bot shows as working. */ + busy: boolean; + /** Replies the server took and has not started yet, drawn as Queued until their turn. */ + queued: readonly { messageId: string; text: string }[]; + /** Only words go: no attachments, no skills (a reply that tries a failed task again). */ + wordsOnly: boolean; + /** Throws when the server did not take it, which puts the words back in the composer. */ + send: (input: { + text: string; + messageId: string; + attachmentIds?: string[]; + skillIds?: string[]; + }) => Promise; + /** Changes whenever the task moves on, which reads the conversation's history again. */ + version: string; +}; + /** * One channel's conversation with one coworker. * @@ -318,9 +343,12 @@ function describeAttachments(attachments: readonly Attachment[]): string { export function ChannelChat({ channel, runtimeAgentId, + task, }: { channel: AgentChannel; runtimeAgentId: string; + /** This conversation is a task's own, and what is sent goes through the task. */ + task?: ChannelTaskMode; }) { // The core attaches the frontend tool registry; direct agent runs do not. const { copilotkit } = useCopilotKit(); @@ -381,6 +409,13 @@ export function ChannelChat({ */ const agentRef = useRef(agent); agentRef.current = agent; + /** Reads the durable history again, from the refresh below; null until it is set up. */ + const pullHistory = useRef<(() => void) | null>(null); + /** + * What the person sent to a task that the transcript does not hold yet, drawn as their bubbles + * until it does: the server records a reply when its turn starts, and reads it back after. + */ + const [taskSends, setTaskSends] = useState([]); const voiceArchive = useVoiceArchive(channel.id); const [voiceContextNotice, setVoiceContextNotice] = useState( null, @@ -557,6 +592,7 @@ export function ChannelChat({ })(); }; + pullHistory.current = pull; const unsubscribe = queryClient.getQueryCache().subscribe(() => { const activity = authoredActivity(); if (activity && !sameActivity(activity, lastSeen)) { @@ -582,11 +618,33 @@ export function ChannelChat({ })(); return () => { cancelled = true; + pullHistory.current = null; unsubscribe(); offFinishedAway(); }; }, [channel.id, channel.threadId, joinGatePromise, runtimeAgentId]); + /* + * A TASK MOVING ON IS NEWS HERE TOO. Its turns run on the server, and the Bot's line arriving is + * not the only moment its history changes: a reply's turn starting records the person's words, + * and a stop for approval records the step it stopped on. So history is read again whenever the + * task does, after the join, never in place of the read on mount. + */ + const taskVersion = task?.version; + const lastTaskVersion = useRef(taskVersion); + useEffect(() => { + if (taskVersion === undefined || taskVersion === lastTaskVersion.current) + return; + lastTaskVersion.current = taskVersion; + let cancelled = false; + void joinGatePromise.then(() => { + if (!cancelled) pullHistory.current?.(); + }); + return () => { + cancelled = true; + }; + }, [taskVersion, joinGatePromise]); + // Tool calls from this conversation act on this coworker's own computer. useActiveBot(runtimeAgentId); @@ -949,11 +1007,58 @@ export function ChannelChat({ /** Stable reference for effects and component callbacks. */ const sayRef = useRef(say); sayRef.current = say; + + /** + * A send into a task: drawn at once under the id it is sent with, so the turn that records it is + * the same message, and taken back off if the server does not take it, so the composer can put + * the words back. + */ + const sendToTask = task + ? async (draft: { + text: string; + attachments: Attachment[]; + commandIds: string[]; + }) => { + const text = draft.text.trim(); + if (!text && draft.attachments.length === 0) return; + const messageId = newId(); + setTaskSends((sent) => [ + ...sent, + { + id: messageId, + role: "user", + content: toMessageContent(text, draft.attachments), + }, + ]); + try { + await task.send({ + text, + messageId, + ...(draft.attachments.length + ? { + attachmentIds: draft.attachments.map( + (attachment) => uploadedAttachment(attachment).attachmentId, + ), + } + : {}), + ...(draft.commandIds.length ? { skillIds: draft.commandIds } : {}), + }); + } catch (error) { + setTaskSends((sent) => + sent.filter((message) => message.id !== messageId), + ); + throw error; + } + } + : null; + const taskSendRef = useRef(sendToTask); + taskSendRef.current = sendToTask; const stopEpoch = useRef(0); const call = useVoiceCall({ channelId: channel.id, - disabled: !channel.active, + // A task's turns run on the server, and a call would talk to the Bot here, beside the task. + disabled: !channel.active || task !== undefined, context: async () => [ voiceArchiveContext(await loadVoiceArchive(channel.id)).slice(-6000), @@ -990,7 +1095,12 @@ export function ChannelChat({ * is still holding, and the failed turn is already reported by `runError` under the transcript. */ const askFromComponent = useCallback((text: string) => { - void sayRef.current(text).catch(() => undefined); + const send = taskSendRef.current; + void ( + send + ? send({ text, attachments: [], commandIds: [] }) + : sayRef.current(text) + ).catch(() => undefined); }, []); /** @@ -1012,6 +1122,30 @@ export function ChannelChat({ // Keep `seed` in state; transcriptMessages gives it up once the agent holds a user turn. }, []); + /** The Bot's own name, once its profile is in. The conversation's name may be its title instead. */ + const botName = agentProfiles?.find( + (profile) => profile.id === runtimeAgentId, + )?.name; + + const transcript = transcriptMessages(agent.messages, seed); + /* + * What was sent to the task and is not in the transcript yet. One the server holds is drawn as + * Queued instead, until its turn starts; one whose turn has started is the person's bubble again, + * until the transcript has it. + */ + const recorded = new Set(transcript.map((message) => message.id)); + const held = new Set((task?.queued ?? []).map((reply) => reply.messageId)); + const unrecorded = taskSends.filter( + (message) => !recorded.has(message.id) && !held.has(message.id), + ); + /* + * The server's own turn shows itself as a work row ("Ana is working on it…"), so the transcript's + * Thinking line is only for a task whose turn is waiting to start. + */ + const serverRunning = + channel.work?.some((entry) => entry.kind === "task") ?? false; + const taskBusy = Boolean(task?.busy) && !serverRunning; + return ( // Activity renderers resolve their agent through this SDK context. It must match useAgent's // channel instance so an action continues this thread instead of looking for a default agent. @@ -1021,21 +1155,23 @@ export function ChannelChat({ >
- profile.id === runtimeAgentId) - ?.name ?? channel.name - } - avatarSeed={ - agentProfiles?.find((profile) => profile.id === runtimeAgentId) - ?.avatarSeed ?? runtimeAgentId - } - taskRunning={agent.isRunning || runsInFlight > 0} - /> + {task ? null : ( + profile.id === runtimeAgentId) + ?.name ?? channel.name + } + avatarSeed={ + agentProfiles?.find((profile) => profile.id === runtimeAgentId) + ?.avatarSeed ?? runtimeAgentId + } + taskRunning={agent.isRunning || runsInFlight > 0} + /> + )} 0} + busy={agent.isRunning || turnsInFlight > 0 || taskBusy} // The `/` menu exposes only skills granted to this Bot. - commands={skillCommands} + {...(task?.wordsOnly ? {} : { commands: skillCommands })} + held={task?.queued} // Readiness is handled by `say`; deletion is the only disabled-chat state. disabled={!channel.active} + // Until the Bot's name is in, the composer keeps its own words. + {...(botName + ? { + placeholder: task + ? `Reply to ${botName}` + : `Message ${botName}`, + } + : {})} messages={withVoiceChats( - transcriptMessages(agent.messages, seed), + unrecorded.length ? [...transcript, ...unrecorded] : transcript, voiceArchive.entries, )} notice={ @@ -1111,6 +1257,10 @@ export function ChannelChat({ } onSubmit={async (draft) => { + if (sendToTask) { + await sendToTask(draft); + return; + } // `draft.agentId` carries the @mentioned coworker, but nothing routes on it yet: this // channel is pinned to one `runtimeAgentId` for the life of its thread, so honouring a // per-message mention is a change to that binding, not to the composer. @@ -1147,8 +1297,11 @@ export function ChannelChat({ * correction typed mid-answer is held here, in this tab, and runs as one follow-up turn the * moment this one is over — including when it is over because somebody pressed the button * above. + * + * Not in a task: the server holds a reply until the task's turn is over, durably, so + * holding it here as well would only hold it twice. */ - queueWhileBusy + queueWhileBusy={!task} restoring={historyLoading} /* * The run, not the turn. Stop reaches a run through the core's abort controller, and that diff --git a/app/src/components/channels/channel-work.tsx b/app/src/components/channels/channel-work.tsx index c78531cf81..3ae7e6df43 100644 --- a/app/src/components/channels/channel-work.tsx +++ b/app/src/components/channels/channel-work.tsx @@ -9,8 +9,9 @@ export type KnownBot = { name: string; avatarSeed: string }; * What a conversation is waiting on, one row each, at the end of the transcript. * * Driven by the server's list rather than this tab's run, because the work is not this tab's: a Bot - * the conversation's Bot asked answers out of sight, and a routine fires with nobody's browser - * open. Without these the conversation looks finished while something is still working on it. + * the conversation's Bot asked answers out of sight, a routine fires with nobody's browser open, and + * a task's turn runs on the server. Without these the conversation looks finished while something is + * still working on it. * * Gone when the list empties, which is when the answer is being written into the conversation or * the work is over. The caller hides it while its own run is going, so a turn that has just asked @@ -54,5 +55,9 @@ function labelFor(entry: ChannelWork, name: string | null): string { if (entry.kind === "handoff") { return name ? `Asking ${name}…` : "Asking another Bot…"; } + // A task's own turn, in the task's own conversation: the Bot is doing the work it was handed. + if (entry.kind === "task") { + return name ? `${name} is working on it…` : "Working on it…"; + } return name ? `${name} is running a routine…` : "A routine is running…"; } diff --git a/app/src/components/channels/chat-messages.ts b/app/src/components/channels/chat-messages.ts index 096cc5c3e8..4ea054cbe7 100644 --- a/app/src/components/channels/chat-messages.ts +++ b/app/src/components/channels/chat-messages.ts @@ -1,3 +1,4 @@ +import { readTaskFrame, START_TASK_TOOL } from "@openbot/shared/task-brief"; import type { ActivityMessage, Message, ToolCall } from "@ag-ui/core"; import { classifyAttachment } from "@/lib/channels/attachments"; @@ -373,6 +374,27 @@ export function toVisibleChatItems( if (isToolResult(message)) results.set(message.toolCallId, message.content); } + /* + * A TASK'S TURN THAT RUNS AGAIN OPENS WITH THE SAME WORDS AGAIN: a retry, or Try again, sends the + * ask once more, and a reply that is tried again is sent with it (shared/task-brief.ts). The person + * said it once, so it is drawn once, where they said it. + */ + let briefSeen = false; + const inputsSeen = new Set(); + const rerun = (text: string) => { + const frame = readTaskFrame(text); + if (!frame) return false; + if (frame.inputId) { + if (inputsSeen.has(frame.inputId)) return true; + inputsSeen.add(frame.inputId); + return false; + } + if (frame.kind !== "brief") return false; + if (briefSeen) return true; + briefSeen = true; + return false; + }; + return messages.flatMap((message): VisibleChatItem[] => { if (!isReadableMessage(message)) return []; // Every row built below is keyed on `message.id`, including the one that keys on it by string @@ -455,6 +477,7 @@ export function toVisibleChatItems( if (typeof message.content === "string") { const text = message.content; + if (rerun(text)) return []; return text ? [{ kind: "text", id: message.id, role: "user", text }] : []; } @@ -462,6 +485,7 @@ export function toVisibleChatItems( .map((part) => (isReadablePart(part) ? readText(part) : null)) .filter((part) => part !== null) .join("\n"); + if (rerun(text)) return []; /* * ATTACHMENTS FIRST, THEN THE CAPTION, which is the order they are read in and the order every @@ -567,3 +591,39 @@ export function toVisibleChatItems( return items; }); } + +/** + * The tasks a Bot started in a turn, moved to the end of that turn, under its words. + * + * A Bot hands bigger work to a task of its own with `start_task` and says only "On it." in the + * conversation, and the call comes before the words. Drawn where it was made, the task's card sat + * above the line that introduces it. A turn ends where the person next says something, so the cards + * go just before that, in the order they were started. + */ +export function startedTasksLast( + items: readonly VisibleChatItem[], +): VisibleChatItem[] { + const ordered: VisibleChatItem[] = []; + let started: VisibleChatItem[] = []; + const close = () => { + ordered.push(...started); + started = []; + }; + for (const item of items) { + if ( + item.kind === "tool" && + item.toolCall.function.name === START_TASK_TOOL + ) { + started.push(item); + continue; + } + if ( + item.kind === "attachments" || + (item.kind === "text" && item.role === "user") + ) + close(); + ordered.push(item); + } + close(); + return ordered; +} diff --git a/app/src/components/channels/chat-transcript.tsx b/app/src/components/channels/chat-transcript.tsx index ba7002775d..cc686d2a1c 100644 --- a/app/src/components/channels/chat-transcript.tsx +++ b/app/src/components/channels/chat-transcript.tsx @@ -10,6 +10,7 @@ import { readConnectPrompt, SHOW_APP_ACCESS_TOOL, } from "@openbot/shared/connect-prompt"; +import { readTaskBrief } from "@openbot/shared/task-brief"; import { IconAlertTriangle, IconBox, @@ -31,6 +32,7 @@ import { useState, } from "react"; import { Streamdown } from "streamdown"; +import { ToolCallApprovals } from "@/components/approvals/inline-approval"; import { ArgumentsArrivingContext } from "@/components/gallery/streaming"; import { Bubble, BubbleContent } from "@/components/ui/bubble"; import { @@ -67,6 +69,7 @@ import { forDisplay, REFUSAL_MARKER, toolResultFailed, + toolResultHeldBack, } from "@/lib/plugins/tool-result"; import { readTeamBotConsent } from "@/lib/team-bots"; import { cn } from "@/lib/utils"; @@ -77,13 +80,16 @@ import { isBrowserStep } from "./browser-activity"; import { attachmentModality, type SentAttachment, + startedTasksLast, toVisibleChatItems, } from "./chat-messages"; import { splitChatOrigin } from "./chat-origin"; import type { QueuedMessage } from "./composer"; +import { useConversationWaiting } from "./conversation-presence"; import { FollowScroll, RESUME_PX, useFollowIntent } from "./follow-intent"; import { LateRenderersContext, LateRenderersWatch } from "./late-renderers"; import { usePacedReveal } from "./paced-reveal"; +import { holdsBlock, splitBubbles } from "./split-bubbles"; import { TeamBotConsentCard } from "./team-bot-consent"; import { dropRepeatedAppCards, @@ -92,6 +98,7 @@ import { type ToolGroup, type TranscriptItem, useDrawnTool, + useHeldForPerson, } from "./tool-activity"; import { ToolRenderBoundary, toolCallFinal } from "./tool-boundary"; import { shouldDrawEarlier, useTranscriptPages } from "./transcript-pages"; @@ -118,6 +125,11 @@ type ChatTranscriptProps = { queued?: readonly QueuedMessage[]; /** Take one back before it runs. Without it a queued line is shown but cannot be undone. */ onRemoveQueued?: (id: string) => void; + /** + * Taken by the server and waiting for their turn, after anything queued here: a task's replies. + * Drawn as Queued, and never taken back from here. + */ + held?: readonly { messageId: string; text: string }[]; /** History has been asked for and has not arrived. Drawn only when there is nothing else to draw. */ restoring?: boolean; /** @@ -138,6 +150,34 @@ type ChatTranscriptProps = { /** One shared empty array, so a screen without a queue does not hand down a new one per render. */ const EMPTY_QUEUE: readonly QueuedMessage[] = []; +const EMPTY_HELD: readonly { messageId: string; text: string }[] = []; +const NO_FILES: readonly Attachment[] = []; + +/** + * A conversation's bubbles: body text, one radius on every corner and no tail, the same padding for + * the Bot and the person. Their colours are `--bubble` and `--bubble-user` (theme.css). + */ +const BUBBLE_CONTENT = "type-body rounded-(--radius-bubble) px-4 py-3"; +/** The person's words, in the accent's tint, on the right. */ +const USER_BUBBLE = "max-w-[85%] *:data-[slot=bubble-content]:bg-bubble-user"; +/* + * A Bot's thought, in a bubble of its own. STILL THE GHOST VARIANT, because a skin finds a Bot's + * words by `[data-variant="ghost"][data-align="start"]` (`skins//skin.css`); so these put back + * the fill, the radius and the padding the variant takes off, rather than choose another variant + * and leave every skin's rule matching nothing. + * + * Capped like a chat's bubble, so a short answer reads as a message rather than a page. + */ +const AGENT_BUBBLE = + "data-[variant=ghost]:max-w-[82%] *:data-[slot=bubble-content]:rounded-(--radius-bubble) *:data-[slot=bubble-content]:bg-bubble *:data-[slot=bubble-content]:px-4 *:data-[slot=bubble-content]:py-3"; +/** + * A bubble holding a code block or a table spans the column once there is room. On a phone it keeps + * the cap, and the block scrolls inside it. + */ +const WIDE_BUBBLE = "md:w-full md:data-[variant=ghost]:max-w-full"; +/** Streamdown's prose, tightened to a bubble: closer paragraphs, bullets hung outside the text. */ +const AGENT_PROSE = + "[&>div]:space-y-2.5 [&_ol]:list-outside [&_ol]:pl-5 [&_ul]:list-outside [&_ul]:pl-5 [&_li]:py-0.5"; /** * Split a person's message into the skill they invoked and the rest of what they typed. @@ -205,19 +245,27 @@ function RestoringTranscript() { * A person who has just asked something watches their own message sit there, and a Bot that is * thinking is indistinguishable from a Bot that failed silently — which this app has shipped before. * - * It borrows the shimmer a running tool line uses, so "working on it" reads the same whether the - * work is a tool call or a model that has not spoken yet. + * It is the Bot's bubble with three dots in it, where its first words will land, the way a chat + * shows somebody typing. The dots hold still with reduced motion (`.typing-dot` in styles.css). */ function Thinking() { return ( -

- Thinking -

+ Thinking + {[0, 1, 2].map((dot) => ( + + ))} +
); } @@ -366,8 +414,12 @@ function Queued({ * message sent by mistake rather than as a file waiting its turn. */} {text ? ( - - + + {/* Shown exactly as typed, for the same reason a sent message is. */} {text} @@ -743,32 +795,30 @@ const TranscriptMessage = memo(function TranscriptMessage({ ); } const align = isUser ? "end" : "start"; + /* + * A task's turns open with the person's ask, or their reply, inside a frame the model reads + * (shared/task-brief.ts). They see their own words. + */ + const asked = (isUser ? readTaskBrief(text) : null) ?? text; // Slack/Teams turns carry a model-facing participant line; show a marker instead of the line. - const origin = isUser ? splitChatOrigin(text) : null; - const said = origin ? origin.text : text; + const origin = isUser ? splitChatOrigin(asked) : null; + const said = origin ? origin.text : asked; const invoked = isUser ? splitSkillChip(said, commandNames) : null; return ( - {/* - A Bot's message takes the whole column, not the width of its words: block content - inside it — a fenced code block, a table — should span the transcript rather than - shrink to its own text. A person's bubble keeps fitting what they said. - */} - - - {isUser ? ( - // A person's own message is shown exactly as they typed it. Rendering it as markdown - // would silently reformat what they said, and an asterisk in a sentence is not - // emphasis. The chip is the one exception, and it is not reformatting: it is drawing - // the thing that was already a chip in the composer as a chip here too, so the - // transcript shows a skill was used rather than a slash that was typed. + {isUser ? ( + + + {/* + * A person's own message is shown exactly as they typed it. Rendering it as + * markdown would silently reformat what they said, and an asterisk in a sentence + * is not emphasis. The chip is the one exception, and it is not reformatting: it is + * drawing the thing that was already a chip in the composer as a chip here too, so + * the transcript shows a skill was used rather than a slash that was typed. + */} {origin && ( @@ -792,28 +842,75 @@ const TranscriptMessage = memo(function TranscriptMessage({ said )} - ) : ( - /* - * A Bot's prose is markdown, and it arrives in pieces. - * - * Rendered with a streaming-aware renderer rather than an ordinary one: half a fenced - * code block or an unclosed bold marker is the NORMAL state for most of a run, and a - * plain markdown parser draws that as literal asterisks and backticks until the - * closing token arrives, so the answer visibly rewrites itself as it lands. This - * closes them for the duration. - */ - - {revealed} - - )} - - + + + ) : ( + + )} ); }); +/** + * A Bot's reply as separate bubbles, one per thought and 8px apart, the way a chat sends them + * (`splitBubbles`). + * + * ONE MEMOISED BUBBLE PER PART. A streamed reply changes only its last part on each chunk, so the + * parts before it compare equal and their markdown is not parsed again: the same reasoning as + * `TranscriptMessage`, one level down. The split itself is one pass over the text and runs whenever + * the text changes, which is the only time this draws at all. + */ +function AgentReply({ + mode, + text, +}: { + mode: "static" | "streaming"; + text: string; +}) { + const parts = splitBubbles(text); + return ( +
+ {parts.map((part, index) => ( + // biome-ignore lint/suspicious/noArrayIndexKey: a reply's thoughts keep their order as it streams. + + ))} +
+ ); +} + +const AgentBubble = memo(function AgentBubble({ + mode, + part, +}: { + mode: "static" | "streaming"; + part: string; +}) { + return ( + + + {/* + * A Bot's prose is markdown, and it arrives in pieces. + * + * Rendered with a streaming-aware renderer rather than an ordinary one: half a fenced code + * block or an unclosed bold marker is the NORMAL state for most of a run, and a plain + * markdown parser draws that as literal asterisks and backticks until the closing token + * arrives, so the answer visibly rewrites itself as it lands. This closes them for the + * duration. + */} + + {part} + + + + ); +}); + /** * The one shape an attachment url is allowed to have, taken from the helper that builds it rather * than written out again, so the check cannot drift from the route that serves the file. @@ -1407,6 +1504,9 @@ const ToolCallBody = memo(function ToolCallBody({ spokeAfter: boolean; }) { const renderToolCall = useRenderToolCall(); + // Held on its approval card: the same inbox the card under it reads. + const waiting = useHeldForPerson().has(toolCallId); + const conversationWaiting = useConversationWaiting(); const toolCall = useMemo( () => ({ id: toolCallId, @@ -1450,13 +1550,23 @@ const ToolCallBody = memo(function ToolCallBody({ * * What was called, shimmering until its result arrives, and then the server's own words * drawn the way a Bot's prose is drawn. + * + * WITH ITS APPROVAL CARD UNDER IT when the server held the call for the person. A run of + * steps gets its cards beside the folded line (`ToolActivity`); a call alone in its turn + * keeps this line instead, and without the card here the request waited only on the Bot's + * page. A tool the app draws itself is left alone: its own renderer places its card. */} {drawn ?? ( - + <> + + + )} @@ -1546,14 +1656,27 @@ export function ServerToolLine({ name, result, place, + waiting = false, + conversationWaiting = false, }: { name: string; result?: string; /** Where this call sits in the conversation, for an app card that may carry the task on. */ place?: CardPlace; + /** Held on its approval card for the person's answer, so it is not running yet. */ + waiting?: boolean; + /** An approval in this conversation waits on the person, which a call repeating it waits with. */ + conversationWaiting?: boolean; }) { - const running = result === undefined; - const { label, detail } = readToolName(name, running); + const heldBack = toolResultHeldBack(result); + // Said no on its card: nothing was done, and the line says so rather than naming it as done. + const declined = heldBack === "declined"; + // Not asked again, because the same request already waits on its card. Not done either way. + const alreadyWaiting = heldBack === "alreadyWaiting"; + const held = + (waiting && result === undefined) || + (alreadyWaiting && conversationWaiting); + const running = result === undefined && !held; /* * A refusal is not a result, and must not read like one. * @@ -1562,8 +1685,13 @@ export function ServerToolLine({ * any guess made here. See REFUSAL_MARKER in server/src/plugins/tools.ts. */ const answer = result === undefined ? undefined : asText(result); - const refused = answer?.startsWith(REFUSAL_MARKER) ?? false; - const failed = !refused && toolResultFailed(result); + const refused = !heldBack && (answer?.startsWith(REFUSAL_MARKER) ?? false); + const failed = !refused && !heldBack && toolResultFailed(result); + // A call that did not happen, or waits on its card, is named plainly ("Send your email"). + const { label, detail } = readToolName( + name, + refused || failed || heldBack !== null || held ? "plain" : running, + ); /* * The marker is for this component, not for the reader. Left in, a refusal reads "Blocked" in the * label and then "Refused." again in the first two words of the body, which is the same fact three @@ -1590,16 +1718,21 @@ export function ServerToolLine({ * THAT TOOL'S OTHER ANSWERS ARE FOR THE MODEL. An app nobody can name comes back as plain text * telling the Bot to say so plainly and not to browse to it — an instruction, which read by the * person looks like the Bot being told off in front of them. The line still says the check - * happened; the Bot's own reply says what it found. + * happened; the Bot's own reply says what it found. A call the person held back answers the same + * kind of thing: it tells the Bot not to try again, and the line has already said what happened. */ - const shown = name === SHOW_APP_ACCESS_TOOL ? undefined : body; + const shown = + name === SHOW_APP_ACCESS_TOOL || heldBack !== null ? undefined : body; return ( {shown ? ( @@ -1815,6 +1948,7 @@ export function ChatTranscript({ messages, onRemoveQueued, queued = EMPTY_QUEUE, + held = EMPTY_HELD, restoring = false, stopped, working, @@ -1833,7 +1967,7 @@ export function ChatTranscript({ /* Which row each tool step was drawn in, so a row keeps its key; see `groupToolSteps`. */ const rowIdsRef = useRef(new Map()); const items = groupToolSteps( - dropRepeatedAppCards(toVisibleChatItems(messages)), + startedTasksLast(dropRepeatedAppCards(toVisibleChatItems(messages))), isDrawn, rowIdsRef.current, ); @@ -1891,7 +2025,10 @@ export function ChatTranscript({ ); /* Which tool calls are still the latest thing, for an app card: see `latestFrom`. */ - const latestIndex = latestFrom(items, { busy, queued: queued.length }); + const latestIndex = latestFrom(items, { + busy, + queued: queued.length + held.length, + }); /* The last thing the Bot said, so a call can tell whether its turn went on to say anything: see `ToolPlace`. */ const lastBotWords = items.findLastIndex( (item) => item.kind === "text" && item.role === "assistant", @@ -2117,6 +2254,13 @@ export function ChatTranscript({ text={message.text} /> ))} + {held.map((message) => ( + + ))}
); @@ -2296,6 +2440,13 @@ export function ChatTranscript({ text={message.text} /> ))} + {held.map((message) => ( + + ))}
{items .slice(pages.split) @@ -2310,7 +2461,7 @@ export function ChatTranscript({ }} /> diff --git a/app/src/components/channels/composer/attachment-strip.tsx b/app/src/components/channels/composer/attachment-strip.tsx index 90ba773333..b752533518 100644 --- a/app/src/components/channels/composer/attachment-strip.tsx +++ b/app/src/components/channels/composer/attachment-strip.tsx @@ -144,8 +144,8 @@ export function AttachmentStrip({ * ON THE SIZE LINE, NOT UNDER IT, AND THAT IS A LAYOUT CONSTRAINT RATHER THAN A * PREFERENCE. The tile's height is fixed so that a file and a thumbnail read as * one row. A third line does not fit: with three lines the two in-flow children - * need 65px (a 20px icon and three 15px lines) inside a 62px box. Something - * has to give, and the only choices are the icon shrinking to 17px or the text + * need 68px (a 20px icon and three 16px lines) inside a 62px box. Something + * has to give, and the only choices are the icon shrinking to 14px or the text * spilling past the border. Folding the warning onto this line keeps two. * * `truncate` is the guarantee, not the wording: `formatBytes` is unbounded, so a diff --git a/app/src/components/channels/composer/composer.tsx b/app/src/components/channels/composer/composer.tsx index f33ca00e3a..665560d0e3 100644 --- a/app/src/components/channels/composer/composer.tsx +++ b/app/src/components/channels/composer/composer.tsx @@ -13,6 +13,7 @@ import { isSegmentsEmpty, mergeAdjacentTextSegments, type Segment, + segmentsToPlainText, text, } from "prompt-area/helpers"; import { @@ -115,9 +116,9 @@ function stagedModality(attachment: Attachment) { const MAX_HEIGHT_PX = 220; /** - * Tracks the compact `text-sm` line box so PromptArea stays vertically centered in one row. + * Tracks the compact editor's 23px line box so PromptArea stays vertically centered in one row. */ -const COMPACT_MIN_HEIGHT_PX = 19; +const COMPACT_MIN_HEIGHT_PX = 23; const COMPACT_MAX_HEIGHT_PX = 96; export type ComposerProps = { @@ -243,6 +244,14 @@ export type ComposerProps = { * that can sum this; a caller with no queue leaves it out and nothing changes. */ queuedAttachmentCount?: number; + /** What the empty box says. "Message Ana" in a conversation with one Bot; "Ask anything" elsewhere. */ + placeholder?: string; + /** + * Called with the draft's plain text whenever it changes: typing, dictation, a send clearing the + * box and a failed send putting the words back alike. A conversation's header orb listens + * through it. + */ + onTextChange?: (text: string) => void; }; /** @@ -328,6 +337,8 @@ export function Composer({ droppedAttachments, // Nothing parked is the right answer for every caller without a queue, which is most of them. queuedAttachmentCount = 0, + placeholder = "Ask anything", + onTextChange, }: ComposerProps) { const [value, setValue] = useState( initialValue ? [{ type: "text", text: initialValue }] : [], @@ -1046,6 +1057,9 @@ export function Composer({ [agents, commands], ); const draft = useMemo(() => toDraft(value, staged), [staged, value]); + useEffect(() => { + onTextChange?.(segmentsToPlainText(value)); + }, [onTextChange, value]); /** * MORE FILES ON THIS STRIP THAN ONE MESSAGE MAY CARRY — a state the screening is supposed to make @@ -1337,9 +1351,10 @@ export function Composer({ const sendLabel = parking ? "Queue message" : "Send message"; const callAction = draft.isEmpty && draft.attachments.length === 0 ? voiceCall : undefined; + // The compact composer's controls are 44px, the smallest thing a finger can hit. const actionClassName = cn( "rounded-full p-0", - compact ? "size-8 self-end" : "size-7", + compact ? "size-(--target) shrink-0 self-end" : "size-7", ); const primaryAction = canStop ? ( ) : callAction ? ( ) : ( ); @@ -1484,10 +1499,10 @@ export function Composer({ data-slot="composer" className={cn( /* - * `py-3` IS LOAD-BEARING ONCE THE TEXT WRAPS. On one line `min-h-14` and the controls - * row's own `items-center` fake the vertical padding, so it read as correct for as long - * as nobody typed a paragraph. Past that the box grows to fit its content exactly and - * the glyphs sit against the border. + * THE PADDING IS LOAD-BEARING ONCE THE TEXT WRAPS. On one line the controls row's own + * `items-center` fakes the vertical padding, so it read as correct for as long as + * nobody typed a paragraph. Past that the box grows to fit its content exactly and the + * glyphs sit against the border. * * It goes on the form rather than on the editor because the editor scrolls internally * at COMPACT_MAX_HEIGHT_PX: padding inside that box would scroll away with the text, so @@ -1499,7 +1514,12 @@ export function Composer({ * started it 42px in from the frame's left edge with nothing under it. The strip is * its own row across the top now, and the three controls keep their row below it. */ - "flex min-h-14 flex-col rounded-2xl border border-border bg-card px-3 py-3 focus-within:border-ring focus-within:ring-3 focus-within:ring-ring/50", + /* + * A pill on a soft shadow, 62px tall around its 44px controls. + * The border is there but clear, so a skin can draw it in a colour of its own, and it + * takes the ring colour, with the ring around it, while the caret is inside. + */ + "flex min-h-[62px] flex-col rounded-[27px] border border-transparent bg-card p-2 shadow-(--shadow-float) focus-within:border-ring focus-within:ring-3 focus-within:ring-ring/50", className, )} onSubmit={handleFormSubmit} @@ -1518,11 +1538,11 @@ export function Composer({ * the middle of that block rather than sitting on the line the person is typing. Empty, * the row is exactly a button tall, so centred and bottom are the same pixel. */} -
+
{attachmentsEnabled ? ( ) : ( )} {/* The slot is ours: `PromptArea` passes no attribute through to its own box. */} -
+
@@ -1621,7 +1650,7 @@ export function Composer({ onChange={handleChange} onImagePaste={canAttach ? stagePastedImage : undefined} onSubmit={submitDraft} - placeholder="Ask anything" + placeholder={placeholder} ref={promptAreaRef} triggers={triggers} value={value} diff --git a/app/src/components/channels/composer/dictation-controls.tsx b/app/src/components/channels/composer/dictation-controls.tsx index 2cd2a7c6af..35976001a4 100644 --- a/app/src/components/channels/composer/dictation-controls.tsx +++ b/app/src/components/channels/composer/dictation-controls.tsx @@ -18,9 +18,12 @@ type Dictation = ReturnType; export function DictationButton({ dictation, disabled, + className, }: { dictation: Dictation; disabled: boolean; + /** Merged over the button's own: the compact composer makes it a 44px control. */ + className?: string; }) { if (!dictation.available) return null; return ( @@ -32,7 +35,7 @@ export function DictationButton({ ? "Dictate a message" : "Dictation needs a supported browser and a secure (HTTPS) connection" } - className="size-8 shrink-0 self-end rounded-full" + className={cn("size-8 shrink-0 self-end rounded-full", className)} disabled={disabled || !dictation.supported || dictation.busy} onClick={() => void dictation.session.start()} size="icon" diff --git a/app/src/components/channels/conversation-presence.tsx b/app/src/components/channels/conversation-presence.tsx new file mode 100644 index 0000000000..fcd4b1afc6 --- /dev/null +++ b/app/src/components/channels/conversation-presence.tsx @@ -0,0 +1,496 @@ +import type { Message } from "@ag-ui/core"; +import { useInfiniteQuery, useQuery } from "@tanstack/react-query"; +import { + createContext, + type ReactNode, + useCallback, + useContext, + useEffect, + useMemo, + useRef, + useState, + useSyncExternalStore, +} from "react"; +import type { BotOrbState } from "@/components/agents/orb/bot-orb"; +import { ChannelAvatar } from "@/components/channels/avatar"; +import { approvalInboxOptions } from "@/lib/approvals"; +import { channelListQueryOptions } from "@/lib/channels/queries"; +import { + type BrowserStep, + browserSteps, + type StepApproval, + stepFor, + turnWaitsOnPerson, +} from "@/lib/channels/step"; +import { cn } from "@/lib/utils"; + +/** + * What one conversation is doing, for its header: the Bot's orb works while the Bot has the turn, + * listens while the person types and lands an answer, and the line under the conversation's name + * says which step the Bot is on, or that it needs the person. + * + * The conversation route provides it and the conversation writes into it (`useReportConversation`), + * so everything on the route reads one conversation and never another's. Readers: + * + * - `useConversationOrb()`: the Bot's orb state. `ConversationOrb` draws it. + * - `useConversationStatus()`: resting, working (with the step, or null for work the server is doing + * that this tab did not start), or needs you. `ConversationStatusLine` draws it. + * - `useConversationSteps()`: the browser steps the Bot took, for the computer pane's step log. + * + * The conversation's reports live in a store outside React, and each reader subscribes to the + * fields it uses, so a keystroke reaches the orb and nothing else, and a new step reaches the line + * and not the orb. What the server says (the conversation list's busy flag, an approval waiting in + * this thread) is read here from the queries the app already keeps. + */ + +/** How long the orb holds "done": its smile springs up and settles away. */ +export const LANDING_MS = 2600; + +/** What the conversation last reported. */ +type Reported = { + /** + * The conversation reporting, null when none is mounted. Readers count a report only from the + * conversation on screen: on the way to another one, the last one's report is still here for a + * moment, until it unmounts and takes it away. + */ + channelId: string | null; + /** The Bot has the turn: asked and not answered yet, browser actions in the middle included. */ + working: boolean; + /** What `stepFor` says about the turn, null when there is none. */ + step: string | null; + /** The composer has something typed in it. */ + drafting: boolean; + /** Counts the turns that ended in an answer, rather than a failure or a Stop. */ + answers: number; + /** The browser steps in the conversation, oldest first. */ + steps: readonly ConversationStep[]; +}; + +/** A browser step, and when this tab first saw it, which is where it sits among the commands. */ +export type ConversationStep = BrowserStep & { at: number }; + +const NO_STEPS: readonly ConversationStep[] = []; + +const AT_REST: Reported = { + channelId: null, + working: false, + step: null, + drafting: false, + answers: 0, + steps: NO_STEPS, +}; + +/** + * When this tab first saw each browser step, by its call. + * + * MODULE SCOPE, SO A STEP KEEPS ITS PLACE. The command log beside these steps lives for the session, + * and a conversation opened again would otherwise stamp every step it already showed with the moment + * it came back, after commands that ran later. A step from before this tab opened is stamped when + * the conversation first showed it. Bounded, oldest out. + */ +const FIRST_SEEN = new Map(); +const MAX_FIRST_SEEN = 1000; + +function firstSeen(callId: string, now: number): number { + const seen = FIRST_SEEN.get(callId); + if (seen !== undefined) return seen; + FIRST_SEEN.set(callId, now); + while (FIRST_SEEN.size > MAX_FIRST_SEEN) { + const oldest = FIRST_SEEN.keys().next().value; + if (oldest === undefined) break; + FIRST_SEEN.delete(oldest); + } + return now; +} + +/** The same steps, unchanged: a streamed token rewrites the array and usually nothing in it. */ +function sameSteps( + current: readonly ConversationStep[], + next: readonly BrowserStep[], +): boolean { + return ( + current.length === next.length && + current.every((step, index) => { + const other = next[index]; + return ( + other !== undefined && + step.item.id === other.item.id && + step.item.result === other.item.result && + step.item.toolCall.function.arguments === + other.item.toolCall.function.arguments && + step.latest === other.latest + ); + }) + ); +} + +type Store = { + read: () => Reported; + write: (next: (current: Reported) => Reported) => void; + subscribe: (listener: () => void) => () => void; +}; + +function createStore(): Store { + let current = AT_REST; + const listeners = new Set<() => void>(); + return { + read: () => current, + write(next) { + const value = next(current); + if (value === current) return; + current = value; + for (const listener of listeners) listener(); + }, + subscribe(listener) { + listeners.add(listener); + return () => listeners.delete(listener); + }, + }; +} + +/** What the server says about this conversation. */ +type ServerFacts = { + /** The server has a turn running here that this tab did not start. */ + working: boolean; + /** An approval in this conversation is waiting on the person. */ + waiting: boolean; + /** This conversation's approvals, so a declined step stops being named before its result. */ + approvals: readonly StepApproval[]; +}; + +const NOTHING_ON_SERVER: ServerFacts = { + working: false, + waiting: false, + approvals: [], +}; + +/** The store, and the conversation on screen. */ +type Presence = { store: Store; channelId: string }; + +const StoreContext = createContext(null); +const ServerContext = createContext(NOTHING_ON_SERVER); + +const unsubscribed = () => () => {}; + +/** + * One field of what the conversation on screen reported, at rest if it has not. Re-renders only + * when that field changes. + */ +function useReported( + presence: Presence | null, + key: K, +): Reported[K] { + return useSyncExternalStore(presence?.store.subscribe ?? unsubscribed, () => { + const reported = presence?.store.read(); + return reported && reported.channelId === presence?.channelId + ? reported[key] + : AT_REST[key]; + }); +} + +/** + * `threadId` is the conversation's thread, for the approvals waiting in it; undefined while the + * conversation loads. The route stays mounted from one conversation to the next, and so does this: + * the conversation leaving takes its reports with it, and `channelId` moves what the server says. + */ +export function ConversationPresenceProvider({ + channelId, + threadId, + children, +}: { + channelId: string; + threadId: string | undefined; + children: ReactNode; +}) { + const [store] = useState(createStore); + const presence = useMemo(() => ({ store, channelId }), [store, channelId]); + const channels = useInfiniteQuery(channelListQueryOptions()); + // The conversation's one poll of the inbox. Its rows and cards read the cache this keeps fresh. + const inbox = useQuery(approvalInboxOptions()); + const row = channels.data?.find((channel) => channel.id === channelId); + // The sidebar's rule (`ChannelRow`), so the header and the conversation's row agree. + const busy = (row?.busy ?? false) || (row?.work?.length ?? 0) > 0; + const working = useUnownedBusy( + channelId, + busy, + useReported(presence, "working"), + ); + const approvals = useMemo( + () => + threadId === undefined + ? [] + : (inbox.data?.requests ?? []).filter( + (request) => request.action.threadId === threadId, + ), + [inbox.data, threadId], + ); + const waiting = approvals.some((request) => request.status === "pending"); + const facts = useMemo( + () => ({ working, waiting, approvals }), + [working, waiting, approvals], + ); + return ( + + {children} + + ); +} + +/** + * The busy flag for work this tab did not start: false while the conversation's own turn runs, and + * for the rest of that busy spell. The server's flag comes down a moment after the tab's own turn + * ends, and without this every answer would end on "Working" as the flag caught up. A run the server + * starts on its own afterwards, such as the one after Allow once, raises a new spell, and shows. + * A spell belongs to one conversation, so opening another starts again. + */ +function useUnownedBusy( + channelId: string, + busy: boolean, + ownTurn: boolean, +): boolean { + const spell = useRef({ channelId, owned: false }); + if (spell.current.channelId !== channelId) + spell.current = { channelId, owned: false }; + if (!busy) spell.current.owned = false; + else if (ownTurn) spell.current.owned = true; + return busy && !spell.current.owned; +} + +/** + * The Bot's orb state for what the conversation is doing. The Bot working outranks the person + * typing, since a correction typed mid-answer waits for the turn to end; typing cuts a landing + * short, and so does an approval waiting here, since the line then says "Needs you". Resting + * outside a conversation route. + */ +export function useConversationOrb(): BotOrbState { + const presence = useContext(StoreContext); + const server = useContext(ServerContext); + const present = useReported(presence, "channelId") !== null; + const working = useReported(presence, "working"); + const drafting = useReported(presence, "drafting"); + // Read whatever conversation it came from: it only counts, and a count from before is landed. + const answers = useSyncExternalStore( + presence?.store.subscribe ?? unsubscribed, + () => presence?.store.read().answers ?? 0, + ); + // The answer whose landing has played out. Behind `answers` while the orb holds "done". + const [landed, setLanded] = useState(answers); + useEffect(() => { + if (answers === landed) return; + // With the conversation gone the landing is over, so coming straight back does not replay it. + if (!present) { + setLanded(answers); + return; + } + const timer = setTimeout(() => setLanded(answers), LANDING_MS); + return () => clearTimeout(timer); + }, [answers, present, landed]); + if (working || server.working) return "thinking"; + if (drafting) return "listening"; + if (server.waiting) return "idle"; + return present && answers !== landed ? "done" : "idle"; +} + +/** How the conversation stands, for the line under its name. */ +export type ConversationStatus = + | { kind: "resting" } + /** `step` is what the Bot is on, or null for a turn the server is running that this tab did not start. */ + | { kind: "working"; step: string | null } + | { kind: "needs-you" }; + +/** + * How the conversation stands. An approval waiting comes first, since the Bot cannot go on with it + * until the person decides. Keystrokes don't reach it. + */ +export function useConversationStatus(): ConversationStatus { + const presence = useContext(StoreContext); + const server = useContext(ServerContext); + const working = useReported(presence, "working"); + const step = useReported(presence, "step"); + if (server.waiting) return { kind: "needs-you" }; + if (working) return { kind: "working", step }; + if (server.working) return { kind: "working", step: null }; + return { kind: "resting" }; +} + +/** + * Whether an approval in this conversation is waiting on the person. A call the server did not ask + * about again, because the same request was already waiting, waits with it while this is true. + * False outside a conversation route. + */ +export function useConversationWaiting(): boolean { + return useContext(ServerContext).waiting; +} + +/** + * The browser steps the conversation's Bot took, oldest first, each with when this tab first saw it. + * None outside a conversation route. + */ +export function useConversationSteps(): readonly ConversationStep[] { + return useReported(useContext(StoreContext), "steps"); +} + +/** + * Report a conversation's turn, draft and browser steps, for as long as it is mounted. Outside a + * conversation route there is nothing to report to, and it returns undefined. + * + * `working` is the turn, not the run, for the reason `ConversationView` gives: a browser action ends + * one run and starts another, and the orb would land in the gap. `failed` is the turn having ended + * without an answer. The step is read off `messages` on every render, as the transcript reads them, + * because the agent grows the same array in place. Hand the composer `onTextChange`, and call + * `onStop` when somebody presses Stop, so a stopped answer does not land either. Nor does a turn + * that ended on a call held for the person, which has no answer yet. + */ +export function useReportConversation({ + working, + failed, + messages, +}: { + working: boolean; + failed: boolean; + messages: readonly Message[]; +}): { onTextChange: (text: string) => void; onStop: () => void } | undefined { + const presence = useContext(StoreContext); + const { approvals } = useContext(ServerContext); + const step = presence ? stepFor(messages, working, approvals) : null; + // Read on every render for the same reason as the step. + const steps = presence ? browserSteps(messages) : NO_STEPS; + // Asked only as the turn ends, so it is not read again on every token of the answer. + const waitsOnPerson = + presence && !working ? turnWaitsOnPerson(messages) : false; + const wasWorking = useRef(false); + const stopped = useRef(false); + /* + * Every report is this conversation's, by the channel on screen when it mounted. The route keys + * the conversation on its channel, so that is the channel it stays. + */ + const store = presence?.store; + const mine = presence?.channelId ?? null; + const report = useCallback( + (change: (current: Reported) => Partial | null) => + store?.write((current) => { + // What another conversation reported is not this one's, apart from the count of answers. + const own = + current.channelId === mine + ? current + : { ...AT_REST, answers: current.answers, channelId: mine }; + const changed = change(own); + return changed ? { ...own, ...changed } : own; + }), + [store, mine], + ); + + // First, so leaving takes everything with it: an answer nobody is there to see does not land. + useEffect(() => { + if (!store) return; + report(() => null); + return () => + store.write((current) => + current.channelId === mine + ? { ...AT_REST, answers: current.answers } + : current, + ); + }, [store, mine, report]); + + useEffect(() => { + if (working && !wasWorking.current) stopped.current = false; + const answered = + wasWorking.current && + !working && + !failed && + !stopped.current && + !waitsOnPerson; + wasWorking.current = working; + report((current) => + current.working === working && !answered + ? null + : { working, answers: current.answers + (answered ? 1 : 0) }, + ); + }, [report, working, failed, waitsOnPerson]); + + useEffect(() => { + report((current) => (current.step === step ? null : { step })); + }, [report, step]); + + useEffect(() => { + report((current) => { + if (sameSteps(current.steps, steps)) return null; + const now = Date.now(); + return { + steps: steps.map((browser) => ({ + ...browser, + at: firstSeen(browser.item.id, now), + })), + }; + }); + }, [report, steps]); + + const onTextChange = useCallback( + (text: string) => { + const drafting = text.trim() !== ""; + report((current) => + current.drafting === drafting ? null : { drafting }, + ); + }, + [report], + ); + const onStop = useCallback(() => { + stopped.current = true; + }, []); + + return store ? { onTextChange, onStop } : undefined; +} + +/** + * The Bot's orb, in the state its conversation is in. Its own component, so a keystroke redraws + * the orb and nothing around it. + */ +export function ConversationOrb({ + participantIds, + size, +}: { + participantIds: string[]; + size: number; +}) { + return ( + + ); +} + +/** + * The line under the conversation's name: "Working · Searching the web" while the Bot works, + * "Working" alone for a run the server resumed, "Needs you" while an approval waits here, and + * nothing at rest. Empty, it takes no height, so the name sits level with the orb. + * + * Always mounted, so the live region is there before the first step arrives: a region added with + * its text already in it is not announced. The step shimmers as a running tool line does, and stops + * for somebody who asked for less motion. + */ +export function ConversationStatusLine({ className }: { className?: string }) { + const status = useConversationStatus(); + return ( +

+ {status.kind === "needs-you" ? ( + Needs you + ) : status.kind === "working" ? ( + <> + Working + {status.step ? ( + <> + {" · "} + {status.step} + + ) : null} + + ) : null} +

+ ); +} diff --git a/app/src/components/channels/conversation-view.tsx b/app/src/components/channels/conversation-view.tsx index f0ff124adf..2543d379e3 100644 --- a/app/src/components/channels/conversation-view.tsx +++ b/app/src/components/channels/conversation-view.tsx @@ -7,6 +7,7 @@ import { useRef, useState, } from "react"; +import { AiNotice } from "@/components/channels/ai-notice"; import { ChatTranscript } from "@/components/channels/chat-transcript"; import { type AgentOption, @@ -18,6 +19,7 @@ import { type QueuedMessage, reduceQueue, } from "@/components/channels/composer"; +import { useReportConversation } from "@/components/channels/conversation-presence"; import { attachmentUrl } from "@/lib/channels/attachments"; import { newId } from "../../lib/new-id"; @@ -39,6 +41,8 @@ export function ConversationView({ onSubmit, onStop, voiceCall, + placeholder, + held, }: { messages: readonly Message[]; busy?: boolean; @@ -97,6 +101,13 @@ export function ConversationView({ /** Stop the Bot mid-answer; forwarded to turn the send button into a stop button. */ onStop?: () => void; voiceCall?: { active: boolean; supported: boolean; onStart(): void }; + /** The composer's empty-state text, e.g. "Message Ana". The composer's own when absent. */ + placeholder?: string; + /** + * Messages the server took and holds for their turn, drawn as Queued at the end of the transcript. + * A task's replies wait there. Unlike the queue this screen keeps, nothing here can take one back. + */ + held?: readonly { messageId: string; text: string }[]; }) { /* * THE QUEUE LIVES HERE BECAUSE BOTH HALVES OF IT DO. @@ -154,6 +165,16 @@ export function ConversationView({ */ const [running, setRunning] = useState(false); const inFlight = pending || running; + /* + * The conversation's header follows it: the orb thinks for the turn and listens to the draft, and + * the line under the name says the step. Not while history is restoring: joining the thread is a + * run of its own, and it is not an answer. Outside a conversation route nothing is reported. + */ + const presence = useReportConversation({ + working: (busy || inFlight) && !restoring, + failed: Boolean(stopped), + messages, + }); /** * Every change to the queue goes through here, so the ref and the state can never disagree. @@ -433,13 +454,14 @@ export function ConversationView({ apply({ id, type: "remove" }); }} queued={queued} + {...(held?.length ? { held } : {})} restoring={restoring} {...(stopped ? { stopped } : {})} working={working} />
{notice} @@ -460,8 +482,17 @@ export function ConversationView({ } : undefined } - onStop={onStop} + onStop={ + onStop && presence + ? () => { + presence.onStop(); + onStop(); + } + : onStop + } onSubmit={(draft) => submit(draft, false)} + {...(presence ? { onTextChange: presence.onTextChange } : {})} + {...(placeholder ? { placeholder } : {})} /* * `inFlight` rather than the `pending` this was given. A drained turn is started from the * effect above rather than from the composer, so the composer's own send tracking knows @@ -502,6 +533,7 @@ export function ConversationView({ */ stoppable={stoppable ?? pending} /> +
); diff --git a/app/src/components/channels/group-chat.tsx b/app/src/components/channels/group-chat.tsx index 2515ce93d2..df032f2e03 100644 --- a/app/src/components/channels/group-chat.tsx +++ b/app/src/components/channels/group-chat.tsx @@ -16,6 +16,7 @@ import { } from "@/lib/groups"; import { markdownComponents } from "@/lib/markdown"; import { newId } from "@/lib/new-id"; +import { AiNotice } from "./ai-notice"; import { ChannelAvatar } from "./avatar"; import { Composer, toAgentOptions } from "./composer"; import { TeamBotConsentCard } from "./team-bot-consent"; @@ -142,6 +143,7 @@ export function GroupChat({ channelId }: { channelId: string }) { }); }} /> +
); diff --git a/app/src/components/channels/split-bubbles.ts b/app/src/components/channels/split-bubbles.ts new file mode 100644 index 0000000000..af3ab4c216 --- /dev/null +++ b/app/src/components/channels/split-bubbles.ts @@ -0,0 +1,66 @@ +/** + * A Bot's reply as the separate thoughts a chat draws as separate bubbles: one bubble per paragraph, + * 8px apart, as a person texting would send them, instead of one tall bubble holding everything. + * + * Paragraphs are split on blank lines, with three exceptions that keep markdown whole: + * - nothing is split inside a fenced code block, including one still streaming in; + * - a list stays with the line that introduces it ("Three things I could start on:"), and its items + * stay together even when the model put blank lines between them; + * - a table stays in one piece. + */ +export function splitBubbles(text: string): string[] { + const blocks: string[] = []; + let current: string[] = []; + let fenced = false; + const flush = () => { + const block = current.join("\n").trim(); + if (block) blocks.push(block); + current = []; + }; + for (const line of text.split("\n")) { + if (FENCE.test(line)) fenced = !fenced; + if (!fenced && line.trim() === "") { + flush(); + continue; + } + current.push(line); + } + flush(); + + const bubbles: string[] = []; + for (const block of blocks) { + const previous = bubbles.at(-1); + if (previous !== undefined && belongsWith(previous, block)) { + bubbles[bubbles.length - 1] = `${previous}\n\n${block}`; + } else { + bubbles.push(block); + } + } + return bubbles.length > 0 ? bubbles : [text]; +} + +/** + * Whether a bubble holds a fenced code block or a table, which wants the whole column on a wide + * screen: a line of code or a row of cells cut at a prose width scrolls sideways inside its bubble. + */ +export function holdsBlock(bubble: string): boolean { + return bubble + .split("\n") + .some((line) => FENCE.test(line) || TABLE_ROW.test(line)); +} + +const FENCE = /^\s{0,3}(```|~~~)/; +const LIST_ITEM = /^\s{0,3}([-*+]|\d+[.)])\s/; +const TABLE_ROW = /^\s*\|/; + +/** Whether a block continues the bubble before it rather than starting one of its own. */ +function belongsWith(previous: string, block: string): boolean { + const lastLine = previous.split("\n").at(-1) ?? ""; + if (LIST_ITEM.test(block)) + return LIST_ITEM.test(lastLine) || lastLine.trimEnd().endsWith(":"); + if (TABLE_ROW.test(block)) return TABLE_ROW.test(lastLine); + // An indented block under a list item is that item's continuation. + if (/^\s{2,}\S/.test(block)) + return previous.split("\n").some((line) => LIST_ITEM.test(line)); + return false; +} diff --git a/app/src/components/channels/tool-activity.tsx b/app/src/components/channels/tool-activity.tsx index 013b3a9986..571f544dc7 100644 --- a/app/src/components/channels/tool-activity.tsx +++ b/app/src/components/channels/tool-activity.tsx @@ -1,4 +1,5 @@ import { CopilotKitContext } from "@copilotkit/react-core/v2/context"; +import { useQuery } from "@tanstack/react-query"; import { readConnectPrompt, SHOW_APP_ACCESS_TOOL, @@ -13,12 +14,14 @@ import { } from "@tabler/icons-react"; import { useCallback, useContext, useMemo, useSyncExternalStore } from "react"; import { Streamdown } from "streamdown"; -import { PendingApprovals } from "@/components/approvals/inline-approval"; +import { ToolCallApprovals } from "@/components/approvals/inline-approval"; +import { approvalInboxCacheOptions } from "@/lib/approvals"; import { markdownComponents } from "@/lib/markdown"; import { connectorName, humanise, ownToolLabel, + personToolLabel, readToolName, } from "@/lib/plugins/tool-name"; import { @@ -26,10 +29,16 @@ import { forDisplay, REFUSAL_MARKER, toolResultFailed, + toolResultHeldBack, } from "@/lib/plugins/tool-result"; import { readTeamBotConsent } from "@/lib/team-bots"; -import { browserStepDetails, isBrowserStep } from "./browser-activity"; +import { + browserOutcome, + browserStepDetails, + isBrowserStep, +} from "./browser-activity"; import type { VisibleChatItem } from "./chat-messages"; +import { useConversationWaiting } from "./conversation-presence"; import { ToolLine } from "./tool-line"; type ToolItem = Extract; @@ -218,18 +227,34 @@ type StepKind = "search" | "fetch" | "browser" | "connector" | "tool"; export type StepDetails = { id: string; kind: StepKind; + /** What it is doing while it runs, and what it did once it has. */ label: string; + /** What it would do, for a step that did not happen or has not yet: "Send your email". */ + plain: string; /** What it was done to: a query, a page, a connector. Muted beside the label. */ detail?: string; href?: string; pending: boolean; failed: boolean; refused: boolean; + /** The person said no on its card. Their decision, not an issue. */ + declined: boolean; + /** + * Not done, because the same request was already waiting on its card and was not asked again. It + * waits with that one, and is not an issue. + */ + alreadyWaiting: boolean; + /** A browser step cut off before it finished, by Stop or by its computer stopping. Not an issue. */ + stopped: boolean; + /** A browser step a computer still starting turned away, so nothing was done yet. Not an issue. */ + notYet: boolean; reason?: string; /** Pages a browser step opened, or a fetch read; zero for anything else. */ pages: number; /** The connector a connector step reached, said the way a person would. */ connector?: string; + /** What it did to the person's own things, once done: "Searched your email". */ + person?: string; inputs: [string, string][]; output?: string; }; @@ -286,15 +311,26 @@ export function stepDetails(step: ToolItem): StepDetails { ); if (isBrowserStep(step)) { const browser = browserStepDetails(step); + // Read the way the computer pane's step log reads it, so the two never disagree. + const outcome = browserOutcome(step); + // A step that didn't happen is named for what it would have done, never as done. + const happened = outcome === "done" || outcome === "pending"; + const plain = browser.plain ?? "Use the browser"; + const label = happened ? (browser.label ?? "Used the browser") : plain; return { id: step.id, kind: "browser", - label: browser.label ?? "Used the browser", + label, + plain, ...(browser.title ? { detail: browser.title } : {}), ...(browser.href ? { href: browser.href } : {}), - pending: browser.pending, - failed: browser.failed, - refused: false, + pending: outcome === "pending", + failed: outcome === "failed", + refused: outcome === "blocked", + declined: outcome === "declined", + alreadyWaiting: outcome === "alreadyWaiting", + stopped: outcome === "stopped", + notYet: outcome === "notYet", ...(browser.reason ? { reason: browser.reason } : {}), pages: browser.visited ? 1 : 0, inputs, @@ -303,33 +339,50 @@ export function stepDetails(step: ToolItem): StepDetails { const name = step.toolCall.function.name; const answer = step.result === undefined ? undefined : asText(step.result); - const refused = answer?.startsWith(REFUSAL_MARKER) ?? false; + // The person holding a call back is not a result either, and its words are for the model. + const heldBack = toolResultHeldBack(step.result); + const declined = heldBack === "declined"; + const alreadyWaiting = heldBack === "alreadyWaiting"; + const refused = !heldBack && (answer?.startsWith(REFUSAL_MARKER) ?? false); const body = refused ? answer?.slice(REFUSAL_MARKER.length).trim() : answer; // This deployment's own tools answer the model, not the person; see `ServerToolLine`. const own = ownToolLabel(name) !== null; + const failed = !refused && !heldBack && toolResultFailed(step.result); const outcome = { id: step.id, pending: step.result === undefined, - failed: !refused && toolResultFailed(step.result), + failed, refused, + declined, + alreadyWaiting, + stopped: false, + notYet: false, inputs, - ...(body && !own ? { output: body } : {}), + ...(body && !own && !heldBack ? { output: body } : {}), }; + // Running, a step says what is happening now; done, what happened. One that did not happen is + // named plainly ("Send your email"), never as though it had. + const state = + failed || refused || heldBack !== null ? "plain" : outcome.pending; // Not a connector: one of this deployment's own tools, or one an external agent runs itself. // Named in words, never as an identifier. - if (!name.startsWith("mcp__")) + if (!name.startsWith("mcp__")) { + const plain = humanise(name); return { ...outcome, kind: "tool", - label: ownToolLabel(name, outcome.pending) ?? humanise(name), + label: (state === "plain" ? null : ownToolLabel(name, state)) ?? plain, + plain, pages: 0, }; + } const [, server = "", ...rest] = name.split("__"); const role = researchRoleOfTool(server, rest.join("__")); - const { label, detail } = readToolName(name); + const { label, detail } = readToolName(name, state); + const plain = readToolName(name, "plain").label; const urls = role === "fetch" ? urlsOf(args) : []; - const base = { ...outcome, label }; + const base = { ...outcome, label, plain }; if (role === "search") { const query = firstString( args.query, @@ -361,15 +414,30 @@ export function stepDetails(step: ToolItem): StepDetails { pages: Math.max(urls.length, 1), }; } + const person = personToolLabel(name); return { ...base, kind: "connector", ...(detail ? { detail } : {}), pages: 0, connector: connectorName(server), + ...(person ? { person } : {}), }; } +/** It ran and it worked: the only kind of step a summary may say was done. */ +function isDone(step: StepDetails): boolean { + return ( + !step.pending && + !step.failed && + !step.refused && + !step.declined && + !step.alreadyWaiting && + !step.stopped && + !step.notYet + ); +} + function joinPhrases(phrases: string[]): string { if (phrases.length <= 1) return phrases[0] ?? ""; return `${phrases.slice(0, -1).join(", ")} and ${phrases.at(-1)}`; @@ -384,10 +452,15 @@ function plural(count: number, one: string, many: string): string { * * Counted by kind rather than listed, because the list is what the line opens to. Research is * never attributed to a provider; a connector is named, because which system was reached is the - * person's own fact. + * person's own fact, and what it did to the person's own mail or calendar is named in their words. + * + * ONLY WHAT HAPPENED. A step that was refused, failed or declined, or never finished, is not called + * done: the first two are counted as issues beside the line, and all of them say how they stand + * when it is opened. With nothing done, the line names the first step plainly. */ export function summarise(steps: readonly StepDetails[]): string { - const of = (kind: StepKind) => steps.filter((step) => step.kind === kind); + const done = steps.filter(isDone); + const of = (kind: StepKind) => done.filter((step) => step.kind === kind); const phrases: string[] = []; if (of("search").length) phrases.push("searched the web"); const read = of("fetch").reduce((sum, step) => sum + step.pages, 0); @@ -395,9 +468,23 @@ export function summarise(steps: readonly StepDetails[]): string { const browser = of("browser"); const browsed = browser.reduce((sum, step) => sum + step.pages, 0); if (browsed) phrases.push(`browsed ${plural(browsed, "page", "pages")}`); - else if (browser.length) phrases.push("used the browser"); + else if (browser.length) { + // Named as the computer's step log names them, up to two; past that, one phrase for all. + const actions = [...new Set(browser.map((step) => step.label))]; + if (actions.length <= 2) + phrases.push(...actions.map((label) => label.toLowerCase())); + else phrases.push("used the browser"); + } + for (const person of new Set( + of("connector").flatMap((step) => (step.person ? [step.person] : [])), + )) + phrases.push(person.charAt(0).toLowerCase() + person.slice(1)); const connectors = [ - ...new Set(of("connector").map((step) => step.connector ?? "")), + ...new Set( + of("connector") + .filter((step) => !step.person) + .map((step) => step.connector ?? ""), + ), ].filter(Boolean); if (connectors.length === 1) phrases.push(`used ${connectors[0]}`); else if (connectors.length > 1) @@ -406,6 +493,7 @@ export function summarise(steps: readonly StepDetails[]): string { ); const tools = of("tool").length; if (tools) phrases.push(`used ${plural(tools, "tool", "tools")}`); + if (phrases.length === 0) return steps[0]?.plain ?? ""; const sentence = joinPhrases(phrases); return sentence.charAt(0).toUpperCase() + sentence.slice(1); } @@ -453,29 +541,85 @@ function IconStack({ kinds }: { kinds: StepKind[] }) { ); } -function StepStatus({ step, active }: { step: StepDetails; active: boolean }) { +/** + * How a step stands, beside its words, for any step that did not simply work. The computer pane's + * step log says the same words (`step-log.tsx`). + */ +export const STEP_STATUS = { + waiting: "Waiting for you", + running: "In progress", + stopped: "Not completed", + notYet: "Not done yet", + declined: "Declined", + blocked: "Blocked", + failed: "Didn't work", +} as const; + +function StepStatus({ + step, + active, + waiting, +}: { + step: StepDetails; + active: boolean; + waiting: boolean; +}) { if (step.pending) return ( - {active ? "In progress" : "Not completed"} + {waiting + ? STEP_STATUS.waiting + : active + ? STEP_STATUS.running + : STEP_STATUS.stopped} + + ); + // Waits with the request it repeated, and once that is answered it was simply not done. + if (step.alreadyWaiting) + return ( + + {waiting ? STEP_STATUS.waiting : STEP_STATUS.stopped} + + ); + if (step.stopped || step.notYet || step.declined) + return ( + + {step.stopped + ? STEP_STATUS.stopped + : step.notYet + ? STEP_STATUS.notYet + : STEP_STATUS.declined} ); if (step.refused) - return Blocked; - if (step.failed) return ( - - Didn't work + + {STEP_STATUS.blocked} ); + if (step.failed) + return ( + {STEP_STATUS.failed} + ); return null; } -function StepRow({ step, active }: { step: StepDetails; active: boolean }) { +function StepRow({ + step, + active, + waiting, +}: { + step: StepDetails; + active: boolean; + /** Held on its approval card until the person answers, or waiting with the request it repeated. */ + waiting: boolean; +}) { + // Only a step that is running now says it is happening; one stopped or held says what it would do. + const label = step.pending && (waiting || !active) ? step.plain : step.label; const line = ( - {step.label} + {label} {step.href ? ( {step.detail} ) : null} - + ); - const reason = - step.failed && step.kind === "browser" ? ( -

- {step.reason ?? "Couldn't finish this step."} -

- ) : null; + // Why a browser step did not work, or the boundary's own words for why it was refused. + const why = + step.kind !== "browser" + ? undefined + : step.failed + ? (step.reason ?? "Couldn't finish this step.") + : step.refused + ? step.reason + : undefined; + const reason = why ? ( +

{why}

+ ) : null; // A browser step says everything it has in its line. Anything else opens to what it was given // and what came back, which no other place in the transcript shows. if (step.kind === "browser" || (!step.inputs.length && !step.output)) @@ -509,10 +659,10 @@ function StepRow({ step, active }: { step: StepDetails; active: boolean }) { return (
  • - + ▸ @@ -542,6 +692,27 @@ function StepRow({ step, active }: { step: StepDetails; active: boolean }) { ); } +/** + * The tool calls the server is holding on an approval card, read from the same inbox the cards are. + * + * A held step has no result yet, the same as one still running or one a stopped turn left behind, + * and only the inbox can tell them apart. Every row asks, so it reads the cache the screen polls. + */ +export function useHeldForPerson(): ReadonlySet { + const inbox = useQuery(approvalInboxCacheOptions()); + return useMemo( + () => + new Set( + (inbox.data?.requests ?? []).flatMap((request) => + request.status === "pending" && request.action.toolCallId + ? [request.action.toolCallId] + : [], + ), + ), + [inbox.data], + ); +} + export function ToolActivity({ group, active, @@ -551,10 +722,22 @@ export function ToolActivity({ active: boolean; }) { const steps = group.steps.map(stepDetails); + const held = useHeldForPerson(); + const waitingHere = useConversationWaiting(); + // On its own card, or with the identical request it repeated while that one still waits. + const waitsForPerson = (step: StepDetails) => + held.has(step.id) || (step.alreadyWaiting && waitingHere); const running = active && steps.some((step) => step.pending); const current = steps.find((step) => step.pending); const issues = steps.filter((step) => step.failed || step.refused).length; const kinds = [...new Set(steps.map((step) => step.kind))]; + /* + * NOTHING DONE IS NOT DONE. With no step that worked, the line is named for the first one plainly + * and marked for how the group stands, so it never carries the done mark. + */ + const nothingDone = !running && !steps.some(isDone); + // Nothing done yet because a step waits on its card, which is drawn beside this line. + const waiting = nothingDone && steps.some(waitsForPerson); // The folded line draws its own steps, so a waiting approval gets its card beside it, not inside. return ( <> @@ -563,14 +746,33 @@ export function ToolActivity({ label={running && current ? `${current.label}…` : summarise(steps)} detail={issues ? plural(issues, "issue", "issues") : undefined} running={running} + failed={nothingDone && steps.some((step) => step.failed)} + refused={ + nothingDone && + !steps.some((step) => step.failed) && + steps.some((step) => step.refused) + } + declined={nothingDone && !issues && steps.some((step) => step.declined)} + notYet={ + nothingDone && + !issues && + !waiting && + !steps.some((step) => step.declined) + } + waiting={waiting && !issues && !steps.some((step) => step.declined)} >
      {steps.map((step) => ( - + ))}
    - step.toolCall.id)} /> diff --git a/app/src/components/channels/tool-line.tsx b/app/src/components/channels/tool-line.tsx index 528d1d794e..dbc533f35f 100644 --- a/app/src/components/channels/tool-line.tsx +++ b/app/src/components/channels/tool-line.tsx @@ -1,4 +1,6 @@ +import { IconCircleCheck, IconClock, IconX } from "@tabler/icons-react"; import type { ReactNode } from "react"; +import { cn } from "@/lib/utils"; /** * One line for one thing a Bot did. @@ -17,10 +19,12 @@ export function ToolLine({ running, refused, failed, + declined, notYet, + waiting, children, }: { - /** Drawn before the label, small and muted. A group of steps puts its kinds here. */ + /** Drawn after the label, small and muted. A group of steps puts its kinds here. */ icon?: ReactNode; /** What was done, in a couple of words: "Searched Slack", "Filled in", "Read the page". */ label: string; @@ -31,63 +35,121 @@ export function ToolLine({ refused?: boolean; /** It was permitted and did not work. A different request might. */ failed?: boolean; + /** The person said no on its card, so it was not done. Their decision, not a failure. */ + declined?: boolean; /** * Nothing was done yet, for an ordinary reason: a computer still starting turned the call away * before it reached anything. Muted like a success, because nothing went wrong, and labelled so it * does not read as one. */ notYet?: boolean; + /** + * Held on its approval card until the person answers. Not running, and not done: muted, with the + * card under the line saying what it would do. + */ + waiting?: boolean; /** Shown when the line is expanded. Without it the line is not expandable. */ children?: ReactNode; }) { const tone = refused - ? "text-destructive" + ? "text-status-problem" : failed - ? "text-amber-600 dark:text-amber-500" + ? "text-status-needs" : "text-muted-foreground"; - + /* + * A hairline hook, then a mark saying how it went, then what was done. Only a line that worked + * gets the check, in the done colour; a group's own kinds come after the label, where they no + * longer compete with the outcome. + */ + const Mark = + refused || failed || declined + ? IconX + : running || notYet || waiting + ? IconClock + : IconCircleCheck; + const markTone = + refused || failed + ? tone + : running || notYet || waiting || declined + ? "text-muted-foreground" + : "text-status-done"; const text = ( - - {icon ? {icon} : null} - - {refused - ? `Blocked: ${label}` - : failed - ? `${label}, didn't work` - : notYet - ? `${label}, not done yet` - : label} + + + + + + + {/* Cut with an ellipsis rather than run past its row: a summary can be long. */} + + {declined + ? `You declined: ${label}` + : refused + ? `Blocked: ${label}` + : failed + ? `${label}, didn't work` + : notYet + ? `${label}, not done yet` + : waiting + ? `${label}, waiting for you` + : label} + + {detail ? {detail} : null} - {detail ? {detail} : null} + {icon ? ( + + {icon} + + ) : null} ); - if (!children) return
    {text}
    ; + if (!children) return
    {text}
    ; return ( -
    +
    {/* * Native
    owns expansion because SDK tool renderers can be re-invoked independently - * of this component's React state. + * of this component's React state. The summary is the line, a finger's height tall, and the + * chevron after the label says it opens. */} - + + {text} ▸ - {text} {/* + * Indented to the label, past the hook and the mark, so the detail reads as belonging to it. * Headings are cut down to the size of the surrounding text. A tool result is markdown a * server wrote for a model, so its `#` is a section marker rather than a page title. */} -
    +
    {children}
    diff --git a/app/src/components/computer/activity-log.tsx b/app/src/components/computer/activity-log.tsx index 417dc47b41..ffb75fbd74 100644 --- a/app/src/components/computer/activity-log.tsx +++ b/app/src/components/computer/activity-log.tsx @@ -9,33 +9,35 @@ * This session only. The record is the audit trail, which is on the server and survives a reload; * this is a window, and it is allowed to be empty when a tab is opened on a conversation that * already happened. + * + * The pane draws these among the Bot's browser steps, in one list (`step-log.tsx`). What is here is + * what only a command log can say: what each call is called, how it ended, and what it printed. */ -import { IconFile, IconFolder, IconTerminal2 } from "@tabler/icons-react"; -import { useSyncExternalStore } from "react"; -import { Empty, EmptyHeader, EmptyTitle } from "@/components/ui/empty"; -import { - activityFor, - type ComputerActivity, - subscribeToActivity, -} from "@/lib/computers/activity"; +import type { ComputerActivity } from "@/lib/computers/activity"; import { CommandOutput } from "./command-output"; -const ICONS = { - command: IconTerminal2, - read_file: IconFile, - write_file: IconFile, - list_files: IconFolder, +/** + * What the line says was done, in the words the call's own line in the conversation uses + * (`computer-tools.tsx`). The subject beside it says what it was done to. + */ +export const ACTIVITY_LABELS = { + command: "Ran a command", + read_file: "Read file", + write_file: "Saved file", + list_files: "Listed files", } as const; -/** What the line says was done. The subject beside it says what it was done to. */ -const LABELS = { - command: "Ran", - read_file: "Read", - write_file: "Saved", - list_files: "Listed", -} as const; +/** How a call ended: a boundary refused it, it exited non-zero, or it worked. */ +export function activityStatus( + entry: ComputerActivity, +): "done" | "blocked" | "failed" { + if (entry.refused === true) return "blocked"; + if (typeof entry.exitCode === "number" && entry.exitCode !== 0) + return "failed"; + return "done"; +} -function timeOf(at: number): string { +export function timeOf(at: number): string { return new Date(at).toLocaleTimeString(undefined, { hour: "2-digit", minute: "2-digit", @@ -43,84 +45,22 @@ function timeOf(at: number): string { }); } -function Entry({ entry }: { entry: ComputerActivity }) { - const Icon = ICONS[entry.kind]; - const failed = - entry.refused === true || - (typeof entry.exitCode === "number" && entry.exitCode !== 0); - - return ( -
  • -
    - - - {LABELS[entry.kind]} - - {/* - The command wraps rather than truncating. In the transcript one line per call is what keeps - it readable; here the whole point is to see exactly what ran, and a long command with its - middle cut out is the one thing this pane must not do. - */} - - {entry.subject} - - - {timeOf(entry.at)} - -
    - - {entry.refused === true ? ( -

    - {entry.output || "A boundary refused it."} -

    - ) : ( -
    - -
    - )} -
    • - ); -} - -export function ActivityLog({ computerId }: { computerId: string }) { - const entries = useSyncExternalStore( - subscribeToActivity, - () => activityFor(computerId), - () => activityFor(computerId), - ); - - if (entries.length === 0) { +/** What came back: the refusal's reason, or what the call printed. */ +export function ActivityOutput({ entry }: { entry: ComputerActivity }) { + if (entry.refused === true) return ( - - - - Nothing yet. Commands the Bot runs and files it reads appear here. - - - +

    + {entry.output || "A boundary refused it."} +

    ); - } - - // Newest first: what somebody watching wants is what just happened, without scrolling for it. - const newestFirst = [...entries].reverse(); - return ( -
      - {newestFirst.map((entry) => ( - - ))} -
    +
    + +
    ); } diff --git a/app/src/components/computer/computer-controls.tsx b/app/src/components/computer/computer-controls.tsx index 328ef03d2f..8f262cae2a 100644 --- a/app/src/components/computer/computer-controls.tsx +++ b/app/src/components/computer/computer-controls.tsx @@ -7,29 +7,39 @@ import { useComputerControl } from "@/lib/computers/use-control"; export function ComputerControlButton({ computerId, onTakeControl, + inline = false, }: { computerId: string; onTakeControl?: () => void; + /** + * The button alone, for a row with no room under it such as the computer pane's header. How a + * change is going is then the screen's to say (`ComputerView`'s `wheelInHeader`). + */ + inline?: boolean; }) { const { control, busy, problem, change, someoneElse } = useComputerControl(computerId); // Someone else's wheel is not the person's to hand back, and not theirs to take from under them either. const human = control?.holder === "human" && !someoneElse; + const button = ( + + ); + if (inline) return button; return (
    - + {button} {control?.transitioning ? ( Finishing the current action… @@ -48,15 +58,20 @@ export function ComputerControlButton({ * The header's way into the Computer: an icon that opens and closes it, with a dot when the * Computer wants the person. Taking and handing back control happen inside the Computer, where the * person can see what they are taking over; the Computer opens itself once for each new prompt. + * + * A quieter dot, `unseen`, says the Bot has been using its computer while it stayed shut, on a + * screen with no room to open it beside the conversation. A prompt for the person outranks it. */ export function ComputerChatControls({ computerId, open, onOpenChange, + unseen = false, }: { computerId?: string; open: boolean; onOpenChange: (open: boolean) => void; + unseen?: boolean; }) { const { control, someoneElse } = useComputerControl( computerId ?? "", @@ -93,7 +108,13 @@ export function ComputerChatControls({ {needsYou ? ( + ) : unseen && !open ? ( + ) : null} diff --git a/app/src/components/computer/computer-panel.tsx b/app/src/components/computer/computer-panel.tsx index d9dc715094..bba1c639e3 100644 --- a/app/src/components/computer/computer-panel.tsx +++ b/app/src/components/computer/computer-panel.tsx @@ -1,21 +1,73 @@ -import { ActivityLog } from "./activity-log"; -import { ComputerView } from "./computer-view"; +import type { Ref } from "react"; +import { + ConversationOrb, + useConversationStatus, +} from "@/components/channels/conversation-presence"; +import { ComputerView, type ComputerViewHandle } from "./computer-view"; +import { StepLog } from "./step-log"; /** The current live computer, separate from screenshots kept in historical tool results. */ export function ComputerViewPanel({ agentId, name, + wheelInHeader = false, + screen, }: { agentId: string; name?: string; + /** Take control is in the pane's header, which opens the screen full size through `screen`. */ + wheelInHeader?: boolean; + screen?: Ref; }) { return (
    - +

    Activity

    - +
    ); } + +/** Small enough for the pane's 48px header, so it blinks and watches and otherwise keeps still. */ +const TITLE_ORB_SIZE = 28; + +/** + * The computer pane's title: the Bot's orb, in the state its conversation is in, whose computer this + * is, and a dot while the conversation needs the person. "Computer" until the Bot's name arrives. + * + * The title alone, which the skins dress as the pane's tab. Take control sits beside it, in the + * header's actions. + */ +export function ComputerPaneTitle({ + agentId, + name, +}: { + agentId: string; + name: string | undefined; +}) { + const needsYou = useConversationStatus().kind === "needs-you"; + return ( + <> + + + {name ? `${name}'s computer` : "Computer"} + + {needsYou ? ( + + ) : null} + + ); +} diff --git a/app/src/components/computer/computer-view.tsx b/app/src/components/computer/computer-view.tsx index c50e055579..f2319ea445 100644 --- a/app/src/components/computer/computer-view.tsx +++ b/app/src/components/computer/computer-view.tsx @@ -1,6 +1,14 @@ import type { BotBrowserApiState } from "@openbot/shared/bot-browser"; +import { IconEyeOff, IconLock, IconShieldCheck } from "@tabler/icons-react"; import { useQuery } from "@tanstack/react-query"; -import { type ReactNode, useEffect, useRef, useState } from "react"; +import { + type ReactNode, + type Ref, + useEffect, + useImperativeHandle, + useRef, + useState, +} from "react"; import { createPortal } from "react-dom"; import { botBrowserKeys, @@ -311,6 +319,18 @@ type Props = { toolCallId?: string; /** How long to wait for a first frame before calling it a problem. Shortened by tests. */ startupGraceMs?: number; + /** + * Take control is in the pane's header, so the bar under the screen says who has control and + * what went wrong, without a second button. The header asks for the full-size view through `ref`. + */ + wheelInHeader?: boolean; + ref?: Ref; +}; + +/** What a header holding the wheel can ask of the screen under it. */ +export type ComputerViewHandle = { + /** The readable full-size view, where a person who has just taken control drives. */ + openFullSize: () => void; }; export function ComputerView({ @@ -325,6 +345,8 @@ export function ComputerView({ finished, toolCallId, startupGraceMs = STARTUP_GRACE_MS, + wheelInHeader = false, + ref, }: Props) { const [shot, setShot] = useState(null); /** @@ -338,6 +360,11 @@ export function ComputerView({ const [expanded, setExpanded] = useState(false); /** The full-size view's live screen, which the bar hands the keyboard back to once an address goes. */ const screen = useRef(null); + useImperativeHandle( + ref, + () => ({ openFullSize: () => setExpanded(true) }), + [], + ); const [streamProblem, setStreamProblem] = useState(null); /** A dropped live stream being retried on its own. A badge, never the overlay. */ const [reconnecting, setReconnecting] = useState(false); @@ -375,6 +402,7 @@ export function ComputerView({ const { control, busy: changingControl, + problem: controlProblem, change: changeControl, refresh: refreshControl, someoneElse, @@ -762,8 +790,13 @@ export function ComputerView({ Audit records that a secret was supplied, not the value. */} {control?.secretWanted ? ( + /* + * The password box: what the page wants, a masked field, and the promise that the value + * goes to the page and nowhere else. "The page", not a site's name: the value goes to the + * element the Bot pointed at, and the frame drawn here can lag behind or show another page. + */
    { event.preventDefault(); if (!secret || sendingSecret) return; @@ -777,11 +810,24 @@ export function ComputerView({ await refreshControl?.(); }} > - -
    +
    + + + + +
    + {/* + Ringed in the accent from the start, since this is what the Bot is waiting for, and + ringed again outside while the field has the caret, as every other field is. + */} +
    + +
    +

    + + It goes straight to the page. {name ?? "The Bot"} never sees it. +

    + {secretProblem ? ( +

    + {secretProblem} +

    + ) : null} +
    -

    - This goes straight to the page. It isn't shown in the - conversation, and the Bot never sees it. -

    - {secretProblem ? ( -

    {secretProblem}

    - ) : null} ) : null} @@ -824,10 +881,21 @@ export function ComputerView({ ? "Someone else has control." : "The Bot has control."} - setExpanded(true)} - /> + {wheelInHeader ? null : ( + setExpanded(true)} + /> + )} + {/* The header's button has no room under it, so what went wrong is said here. */} + {wheelInHeader && controlProblem ? ( + + {controlProblem} + + ) : null}
    ) : null} diff --git a/app/src/components/computer/desktop-illustration.tsx b/app/src/components/computer/desktop-illustration.tsx index a0321c9a73..d93188d814 100644 --- a/app/src/components/computer/desktop-illustration.tsx +++ b/app/src/components/computer/desktop-illustration.tsx @@ -10,8 +10,17 @@ import { cn } from "@/lib/utils"; * Built the way Tailark builds its workspace illustrations — theme tokens, thin borders, skeleton * bars where text would be — so it needs no asset, follows the theme, and never shows a stale * product screenshot. Decorative only: hidden from the tree and inert to the pointer. + * + * `still` holds the cursor where it rests, as reduced motion does: onboarding passes it while the + * picture is off screen or its tab is hidden, so a loop nobody can see does not keep running. */ -export function DesktopIllustration({ className }: { className?: string }) { +export function DesktopIllustration({ + className, + still = false, +}: { + className?: string; + still?: boolean; +}) { return ( ); } @@ -70,8 +79,8 @@ const CLICK_EASE = [ * Percentages rather than pixels, so the same journey fits whatever size the illustration is * drawn at. */ -function PointerCursor() { - const reducedMotion = useReducedMotion(); +function PointerCursor({ still }: { still: boolean }) { + const reducedMotion = useReducedMotion() || still; return ( ; + waiting?: boolean; +}; + +/** + * The browser steps and the command log as one list, oldest first. Stable, so steps seen at the + * same moment, such as a conversation's history arriving at once, keep the conversation's order. + */ +export function stepLogEntries( + steps: readonly ConversationStep[], + activity: readonly ComputerActivity[], + context: StepLogContext, +): StepLogEntry[] { + const browser = steps.map((step): StepLogEntry => { + const status = browserState(step, context); + const details = stepDetails(step.item); + return { + kind: "browser", + id: step.item.id, + at: step.at, + status, + // Only a step that ran, or is running, is named as done; one waiting or cut off says what it would do. + label: + status === "done" || status === "running" + ? details.label + : details.plain, + ...(details.detail ? { detail: details.detail } : {}), + ...(details.href ? { href: details.href } : {}), + ...(status === "failed" + ? { reason: details.reason ?? "Couldn't finish this step." } + : {}), + }; + }); + const commands = activity.map( + (entry): StepLogEntry => ({ + kind: "activity", + id: entry.id, + at: entry.at, + status: activityStatus(entry), + activity: entry, + }), + ); + return [...browser, ...commands].sort((a, b) => a.at - b.at); +} + +const MARKS = { + done: { Icon: IconCircleCheck, tone: "text-status-done" }, + waiting: { Icon: IconClock, tone: "text-status-needs" }, + running: { Icon: IconClock, tone: "text-muted-foreground" }, + stopped: { Icon: IconClock, tone: "text-muted-foreground" }, + notYet: { Icon: IconClock, tone: "text-muted-foreground" }, + declined: { Icon: IconX, tone: "text-muted-foreground" }, + blocked: { Icon: IconX, tone: "text-status-problem" }, + failed: { Icon: IconX, tone: "text-status-problem" }, +} as const; + +/** The colour a state's words take: amber for the person's turn, red for a problem, else grey. */ +function toneOf(status: StepState): string { + if (status === "waiting") return "text-status-needs"; + if (status === "blocked" || status === "failed") return "text-status-problem"; + return "text-muted-foreground"; +} + +function StepRow({ + entry, + children, +}: { + entry: StepLogEntry; + /** What the step was done to, after its words. */ + children?: ReactNode; +}) { + const { Icon, tone } = MARKS[entry.status]; + const label = + entry.kind === "browser" + ? entry.label + : ACTIVITY_LABELS[entry.activity.kind]; + return ( +
  • +
    + + + {label} + + {children} + + {entry.status === "done" ? null : ( + + {STEP_STATUS[entry.status]} + + )} + {/* + A command's time is when it ran. A browser step's is only when this tab first saw it, + which for a conversation opened later is not when it happened, so it is not shown. + */} + {entry.kind === "activity" ? ( + + {timeOf(entry.at)} + + ) : null} + +
    + {entry.kind === "browser" && entry.reason ? ( +

    + {entry.reason} +

    + ) : null} + {entry.kind === "activity" ? ( +
    + +
    + ) : null} +
    • + ); +} + +function BrowserSubject({ + entry, +}: { + entry: Extract; +}) { + if (entry.href) + return ( +     + {entry.detail ?? entry.href} + + ); + return entry.detail ? ( + + {entry.detail} + + ) : null; +} + +export function StepLog({ computerId }: { computerId: string }) { + const steps = useConversationSteps(); + const activity = useSyncExternalStore( + subscribeToActivity, + () => activityFor(computerId), + () => activityFor(computerId), + ); + const working = useConversationStatus().kind === "working"; + const held = useHeldForPerson(); + const waiting = useConversationWaiting(); + const entries = stepLogEntries(steps, activity, { working, held, waiting }); + + if (entries.length === 0) { + return ( + + + + Nothing yet. Pages the Bot opens and commands it runs appear here. + + + + ); + } + + return ( +
      + {entries.map((entry) => + entry.kind === "browser" ? ( + + + + ) : ( + + {/* + The command wraps rather than truncating. In the transcript one line per call is what + keeps it readable; here the whole point is to see exactly what ran, and a long command + with its middle cut out is the one thing this pane must not do. + */} + + {entry.activity.subject} + + + ), + )} +
    + ); +} diff --git a/app/src/components/computer/use-auto-watch-screen.ts b/app/src/components/computer/use-auto-watch-screen.ts index b3195ba956..fb16195e2d 100644 --- a/app/src/components/computer/use-auto-watch-screen.ts +++ b/app/src/components/computer/use-auto-watch-screen.ts @@ -1,4 +1,4 @@ -import { useEffect, useRef } from "react"; +import { useEffect, useRef, useState } from "react"; import { onComputerActivity, onComputerActivityWithdrawn, @@ -14,11 +14,16 @@ import { * * The person's pane comes first. It does not open over something they opened (`busy`), it does not * reopen what they are already watching, and a run they closed stays closed for the rest of it. + * + * NOR OVER THE CONVERSATION. Where there is no room for the pane beside it (`docked` false), the + * pane would be a sheet covering what the person is reading, so it stays shut and `unseen` lights + * the header's computer button instead, until they open it or the call that lit it is taken back. */ export function useAutoWatchScreen({ botId, watching, busy, + docked, open, close, }: { @@ -27,9 +32,11 @@ export function useAutoWatchScreen({ watching: boolean; /** The pane is showing something else the person asked for, which auto-open must not take. */ busy: boolean; + /** The pane would open beside the conversation, rather than as a sheet over it. */ + docked: boolean; open: () => void; close: () => void; -}): { dismiss: () => void } { +}): { dismiss: () => void; unseen: boolean } { /** The run the Bot is in, so a dismissal can name the one it applies to. */ const run = useRef(null); const dismissed = useRef(null); @@ -40,24 +47,44 @@ export function useAutoWatchScreen({ * a page some earlier call loaded holds that call's announcement rather than a later one's. */ const openedFor = useRef(null); + /** The announcement that lit the button instead of opening the pane, or null when it is out. */ + const litFor = useRef(null); + const [unseen, setUnseen] = useState(false); // Read inside subscriptions that outlive the render they were made in. - const latest = useRef({ watching, busy, open, close }); - latest.current = { watching, busy, open, close }; + const latest = useRef({ watching, busy, docked, open, close }); + latest.current = { watching, busy, docked, open, close }; useEffect(() => { + // A light for one Bot is not the next conversation's. + litFor.current = null; + setUnseen(false); if (!botId) return; const stopActivity = onComputerActivity((activity) => { if (activity.botId !== botId) return; run.current = activity.epoch; if (dismissed.current === activity.epoch) return; - const { watching: isWatching, busy: isBusy, open: show } = latest.current; + const { + watching: isWatching, + busy: isBusy, + docked: beside, + open: show, + } = latest.current; if (isWatching || isBusy) return; + if (!beside) { + litFor.current = activity.announcement; + setUnseen(true); + return; + } openedFor.current = activity.announcement; show(); }); const stopWithdrawals = onComputerActivityWithdrawn((withdrawal) => { if (withdrawal.botId !== botId) return; + if (litFor.current === withdrawal.announcement) { + litFor.current = null; + setUnseen(false); + } if (openedFor.current !== withdrawal.announcement) return; openedFor.current = null; latest.current.close(); @@ -68,10 +95,18 @@ export function useAutoWatchScreen({ }; }, [botId]); + // Seen: the person opened the screen, by the lit button or any other way. + useEffect(() => { + if (!watching) return; + litFor.current = null; + setUnseen(false); + }, [watching]); + return { dismiss: () => { dismissed.current = run.current; openedFor.current = null; }, + unseen, }; } diff --git a/app/src/components/delivery/reaching-you.tsx b/app/src/components/delivery/reaching-you.tsx index 3cc9c6d0a4..203c2c04d6 100644 --- a/app/src/components/delivery/reaching-you.tsx +++ b/app/src/components/delivery/reaching-you.tsx @@ -43,7 +43,10 @@ import { import { Separator } from "@/components/ui/separator"; import { agentListQueryOptions } from "@/lib/agents/queries"; import { conversationLabel } from "@/lib/channels/label"; -import { channelListQueryOptions } from "@/lib/channels/queries"; +import { + channelListQueryOptions, + conversationsOnly, +} from "@/lib/channels/queries"; import { type ChatLink, confirmSmsMutationOptions, @@ -349,12 +352,11 @@ function ConnectDialog({ }, }, ); - const conversations = - channels.data?.filter( - (channel) => - channel.active && - (fixedBot === undefined || channel.agentIds.includes(fixedBot)), - ) ?? []; + const conversations = conversationsOnly(channels.data).filter( + (channel) => + channel.active && + (fixedBot === undefined || channel.agentIds.includes(fixedBot)), + ); const selected = channels.data?.find((channel) => channel.id === channelId); const chosen = !!agentId && !!channelId; const error = channels.error ?? link.error ?? sms.error ?? confirm.error; diff --git a/app/src/components/layout/detail-panel.tsx b/app/src/components/layout/detail-panel.tsx index 8795daf96d..d1d3fa150e 100644 --- a/app/src/components/layout/detail-panel.tsx +++ b/app/src/components/layout/detail-panel.tsx @@ -1,6 +1,12 @@ import { IconX } from "@tabler/icons-react"; import { motion, useReducedMotion } from "motion/react"; -import { type ReactNode, useLayoutEffect, useRef, useState } from "react"; +import { + type ReactNode, + useEffect, + useLayoutEffect, + useRef, + useState, +} from "react"; import { Button } from "@/components/ui/button"; import { Sheet, SheetContent, SheetTitle } from "@/components/ui/sheet"; import { EASE_OUT } from "@/lib/motion"; @@ -13,6 +19,9 @@ import { EASE_OUT } from "@/lib/motion"; * * The pane is animated by width and the content inside it is given that width outright, so the * content is laid out once and the pane reveals it. + * + * Whether it would be a sheet is the owner's business too, open or closed: something that opens + * the pane by itself can keep from covering the page it sits beside (`onOverlayChange`). */ const ANIMATION_DURATION_SECONDS = 0.3; @@ -32,17 +41,23 @@ export function DetailPanel({ open, onClose, title, + actions, detail, detailWidth = DEFAULT_DETAIL_WIDTH, + onOverlayChange, children, }: { open: boolean; onClose: () => void; /** Rendered at the left of the detail pane's header row, beside the close button. */ title?: ReactNode; + /** Controls for the detail, at the right of its header row before the close button. */ + actions?: ReactNode; detail?: ReactNode; /** Open width of the detail pane, in pixels. */ detailWidth?: number; + /** Told whether the detail would be a sheet over the page, and again whenever that changes. */ + onOverlayChange?: (overlay: boolean) => void; children: ReactNode; }) { // Reduced motion keeps the fade, which explains the change, and drops the movement. @@ -61,6 +76,12 @@ export function DetailPanel({ observer.observe(element); return () => observer.disconnect(); }, [detailWidth]); + // Read when the answer changes, so an owner passing a new function each render is not told again. + const reportOverlay = useRef(onOverlayChange); + reportOverlay.current = onOverlayChange; + useEffect(() => { + reportOverlay.current?.(overlay); + }, [overlay]); return (
    @@ -76,9 +97,13 @@ export function DetailPanel({ className="gap-0" style={{ width: `min(${detailWidth}px, 100vw)`, maxWidth: "100vw" }} > - - {title ?? "Details"} - + {/* The sheet's own close button sits at the top right, past the actions. */} +
    + + {title ?? "Details"} + + {actions} +
    {open ? detail : null}
    @@ -111,7 +136,8 @@ export function DetailPanel({ > {title}
    -
    +
    + {actions} + ) : undefined + } + icon={IconPlug} + title="No apps to connect yet" + > + {isAdmin + ? "Add apps for everyone in Plugins, then connect your own here or in Connections." + : "When an admin adds apps, you can connect them in Connections."} + + ) : ( + +
      + {rows?.map((app, at) => ( +
    • + {at > 0 ? : null} + { + setProblem(null); + connect.mutate(app.key); + }} + /> +
      • + ))} +
    +     + )} +
    + ); +} + +/** One app: its mark, its name, and Connected, Connect, or why neither. */ +function AppRow({ + app, + botName, + busy, + connecting, + checking, + onConnect, +}: { + app: AppAccess; + botName: string; + busy: boolean; + connecting: boolean; + checking: boolean; + onConnect: () => void; +}) { + const note = noteFor(app, botName); + return ( + + + + + + {app.displayName} + {note ? {note} : null} + + + {checking ? ( + Checking… + ) : app.state === "ready" ? ( + + + Connected + + ) : connectsHere(app) ? ( + + ) : null} + + + ); +} diff --git a/app/src/components/onboarding/demo-stage.tsx b/app/src/components/onboarding/demo-stage.tsx new file mode 100644 index 0000000000..20635caef1 --- /dev/null +++ b/app/src/components/onboarding/demo-stage.tsx @@ -0,0 +1,107 @@ +import { useReducedMotion } from "motion/react"; +import { type ReactNode, useEffect, useRef, useState } from "react"; +import { IllustrationFrame } from "./illustration-frame"; + +/** How long one beat of a demo holds before the next. */ +export const DEMO_BEAT_MS = 2400; + +/** What a demo draws each beat with. */ +export type DemoBeat = { + /** Which beat, from 0 to `beats - 1`. Reduced motion holds `rest`. */ + beat: number; + /** Moving now: on screen, in a shown tab, and motion allowed. Anything animated stops without it. */ + playing: boolean; + /** The reader asked for less motion. */ + reduced: boolean; +}; + +/** + * Whether an element is on screen in a shown tab, kept current. + * + * On screen until an IntersectionObserver says otherwise, and always where there is none to ask, + * so a browser without one still plays rather than freezing on its first beat. + */ +function useOnScreen() { + const ref = useRef(null); + const [visible, setVisible] = useState(true); + useEffect(() => { + const element = ref.current; + let intersecting = true; + const update = () => setVisible(intersecting && !document.hidden); + const observer = + typeof IntersectionObserver === "undefined" + ? null + : new IntersectionObserver(([entry]) => { + intersecting = entry?.isIntersecting ?? false; + update(); + }); + if (element) observer?.observe(element); + document.addEventListener("visibilitychange", update); + update(); + return () => { + observer?.disconnect(); + document.removeEventListener("visibilitychange", update); + }; + }, []); + return [ref, visible] as const; +} + +/** + * The picture a step shows, and the clock it plays on. + * + * A PICTURE, AND SAID TO BE ONE. Everything inside sits in {@link IllustrationFrame}: nothing in it + * can be clicked, typed in or tabbed to, and a screen reader hears `label` once in its place. The + * buttons drawn on the approval card are a drawing of the decision, not the decision. + * + * ONE CLOCK PER DEMO, AND ONLY WHILE SOMEBODY CAN SEE IT. The beat advances every `beatMs` while + * the demo is on screen in a shown tab. Scrolled away, in a background tab, or with the reader's + * reduced-motion setting on, it stops, and `playing` tells the orbs and loops inside to stop too. + * With reduced motion it holds `rest`, the beat that says the most as a still. + */ +export function DemoStage({ + kind, + label, + beats = 1, + rest = beats - 1, + beatMs = DEMO_BEAT_MS, + className = "", + children, +}: { + /** Which demo, for tests and styles. */ + kind: string; + /** What the picture shows, read in place of everything drawn inside it. */ + label: string; + beats?: number; + rest?: number; + beatMs?: number; + className?: string; + children: (now: DemoBeat) => ReactNode; +}) { + const reduced = useReducedMotion() === true; + const [ref, visible] = useOnScreen(); + const [beat, setBeat] = useState(0); + const playing = !reduced && visible; + useEffect(() => { + if (!playing || beats < 2) return; + const timer = setInterval( + () => setBeat((value) => (value + 1) % beats), + beatMs, + ); + return () => clearInterval(timer); + }, [playing, beats, beatMs]); + const shown = reduced ? rest : beat; + + return ( +
    + + {children({ beat: shown, playing, reduced })} + +
    + ); +} diff --git a/app/src/components/onboarding/demos/approval-demo.tsx b/app/src/components/onboarding/demos/approval-demo.tsx new file mode 100644 index 0000000000..497195f77b --- /dev/null +++ b/app/src/components/onboarding/demos/approval-demo.tsx @@ -0,0 +1,113 @@ +import { IconCheck, IconMail } from "@tabler/icons-react"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { buttonVariants } from "@/components/ui/button"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** The three answers an approval offers, in the words the real card uses. */ +export const APPROVAL_CHOICES = [ + "Decline", + "Allow once", + "Always allow here", +] as const; + +/** + * When a Bot asks: a Bot about to send an email waits for an answer, gets Allow once, and + * sends it. + * + * A DRAWING OF AN APPROVAL CARD, FOR THIS STEP ONLY. The real card is the conversation's, and it + * decides a real request; this one decides nothing, so it is drawn here rather than borrowed. The + * buttons say what the real card says, Allow once first and loudest, Always allow here the quietest. + * + * Three beats: waiting for you, Allow once pressed, sent. With reduced motion it waits, because the + * question is the point. + */ +export function ApprovalDemo() { + return ( + + {({ beat, playing }) => { + const pressed = beat >= 1; + const sent = beat === 2; + return ( +
    +
    + + {/* The team step's Research Analyst, with the same face. */} + + +
    +

    + Research Analyst wants to send an email +

    +

    + To ana@example.com +

    +
    + + {sent ? "Sent" : "Needs you"} + +
    +
    + +

    + Hi Ana, Friday at 7 works for all eight of us. I'll book the + table. +

    +
    +
    + {sent ? ( +

    + + Allowed once +

    + ) : ( + <> + + {APPROVAL_CHOICES[0]} + + + {APPROVAL_CHOICES[1]} + + + {APPROVAL_CHOICES[2]} + + + )} +
    +
    + ); + }} +     + ); +} diff --git a/app/src/components/onboarding/demos/computer-demo.tsx b/app/src/components/onboarding/demos/computer-demo.tsx new file mode 100644 index 0000000000..1ed0808dfa --- /dev/null +++ b/app/src/components/onboarding/demos/computer-demo.tsx @@ -0,0 +1,37 @@ +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { DesktopIllustration } from "@/components/computer/desktop-illustration"; +import { ComputerPlaceholder } from "@/components/computer/placeholder"; +import { DemoStage } from "../demo-stage"; + +/** + * A browser and files of its own: the desktop a Bot works on, its browser and its folder of files, + * with its pointer on an errand, and the orb working in the corner. + * + * The computer panel's own drawing of a Bot's desktop, so the picture matches what it shows + * when a Bot is using it. Two beats only to swap the caption; the pointer keeps its own loop, held + * still while the demo is off screen. + */ +export function ComputerDemo() { + return ( + + {({ beat, playing }) => ( +
    + + +
    + + + {beat === 0 ? "Reading the page" : "Saving it to a file"} + +
    +
    + )} +     + ); +} diff --git a/app/src/components/onboarding/demos/self-host-demo.tsx b/app/src/components/onboarding/demos/self-host-demo.tsx new file mode 100644 index 0000000000..0f33d0e154 --- /dev/null +++ b/app/src/components/onboarding/demos/self-host-demo.tsx @@ -0,0 +1,94 @@ +import { WEB_HARNESSES } from "@openbot/shared/harnesses"; +import { AnimatePresence, motion } from "motion/react"; +import type { ReactNode } from "react"; +import { HarnessMark } from "@/components/agents/harness-mark"; +import { appConfig } from "@/lib/generated/application-config"; +import { LaptopIllustration } from "../laptop-illustration"; +import { OrganizationFlow } from "../organization-flow"; +import { DemoStage } from "../demo-stage"; + +/** Every framework the New Bot wizard offers by name; "an agent you already run" is the sentence. */ +export const FRAMEWORKS = WEB_HARNESSES.filter( + (harness) => harness.id !== "byo-url", +); + +const PRODUCT = appConfig.brand.productName; + +/** The three things somebody running OpenBot themselves is deciding, one picture each. */ +const PANELS: Array<{ id: string; caption: string; picture: () => ReactNode }> = + [ + { + id: "local", + caption: "On this computer", + picture: () => , + }, + { + id: "agents", + caption: "With the agents you already have", + picture: () => ( +
    + {FRAMEWORKS.map((harness) => ( + + + {harness.name} + + ))} +
    + ), + }, + { + id: "organization", + caption: "For your organization", + picture: () => , + }, + ]; + +function Panel({ panel }: { panel: (typeof PANELS)[number] }) { + return ( +
    +

    + {panel.caption} +

    + {panel.picture()} +
    + ); +} + +/** + * Running it yourself: on this computer, with the agents you already have, and for your + * organization. The three local-evaluation slides as one picture that turns through them. + * With reduced motion the three sit one above the other. + */ +export function SelfHostDemo() { + return ( + harness.name).join(", ")}; and your organization signing in with Google, Microsoft or Okta`} + rest={-1} + > + {({ beat, reduced }) => + reduced ? ( + PANELS.map((panel) => ) + ) : ( + + + {PANELS[beat] ? : null} + + + ) + } + + ); +} diff --git a/app/src/components/onboarding/demos/team-demo.tsx b/app/src/components/onboarding/demos/team-demo.tsx new file mode 100644 index 0000000000..b4d1a7c211 --- /dev/null +++ b/app/src/components/onboarding/demos/team-demo.tsx @@ -0,0 +1,130 @@ +import { useQuery } from "@tanstack/react-query"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { type AgentProfile, agentListQueryOptions } from "@/lib/agents/queries"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** What a card needs. Examples carry these and nothing more. */ +type TeamCard = Pick & { + /** What it does, in a few words. */ + role: string; + /** Invented, to show what a team looks like: dimmed and labelled, never read as a Bot here. */ + example?: boolean; +}; + +/** + * Stand-ins for a deployment with fewer than three public Bots to show. Invented names on purpose: + * they illustrate a team without claiming any of these exist here. Their seeds are picked to draw + * three different colours: berry, ocean and coral. + */ +const EXAMPLES: TeamCard[] = [ + { + id: "example-research", + name: "Research Analyst", + avatarSeed: "example-research", + role: "Finds and compares", + }, + { + id: "example-data", + name: "Data Analyst", + avatarSeed: "data-analyst", + role: "Reads the numbers", + }, + { + id: "example-support", + name: "Support Desk", + avatarSeed: "example-support-agent", + role: "Answers customers", + }, +]; + +/** The Bots a team card shows: real public Bots first, examples topping up a sparse deployment. */ +export function teamCards(agents: AgentProfile[] | undefined): TeamCard[] { + const shared = + agents?.filter((agent) => !agent.mine && agent.visibility === "public") ?? + []; + return [ + ...shared.slice(0, 3).map((agent) => ({ + id: agent.id, + name: agent.name, + avatarSeed: agent.avatarSeed, + role: agent.title, + })), + // Slicing past the end is [], so three or more real Bots take no examples at all. + ...EXAMPLES.slice(shared.length).map((example) => ({ + ...example, + example: true, + })), + ]; +} + +/** + * Your team of Bots: three Bots, each its own orb in its own colour, and a place for one of yours. + * One at a time, a Bot picks something up and works on it, so the cards read as a team. + */ +export function TeamDemo() { + const { data: agents } = useQuery(agentListQueryOptions()); + const cards = teamCards(agents); + return ( + `${card.name} (${card.example ? "example" : "available"})`, + ) + .join(", ")}, and one of your own`} + rest={-1} + > + {({ beat, playing }) => ( + <> + {cards.map((card, at) => { + const working = at === beat; + return ( +
    + + + +
    +

    {card.name}

    +

    + {card.example ? "Example" : card.role || "Ready"} +

    +
    + + + Working + +
    + ); + })} +
    + +

    + Your own Bot +

    +
    + + )} +     + ); +} diff --git a/app/src/components/onboarding/demos/welcome-demo.tsx b/app/src/components/onboarding/demos/welcome-demo.tsx new file mode 100644 index 0000000000..eeca941845 --- /dev/null +++ b/app/src/components/onboarding/demos/welcome-demo.tsx @@ -0,0 +1,147 @@ +import { IconArrowUp, IconMicrophone, IconPlus } from "@tabler/icons-react"; +import { useEffect, useState } from "react"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** The request the composer types, a thing somebody would really hand off. */ +export const WELCOME_REQUEST = + "Find somewhere near the office for a team dinner on Friday, for eight."; + +/** How long each character takes to land. Linear: a typewriter that eases reads as hesitating. */ +const CHARACTER_MS = 38; + +/** + * Types `text` in while `active`, one character at a time, starting again each time it turns on. + * Off, or with nothing moving, it is the whole text at once. + */ +function useTyped(text: string, active: boolean): string { + const [shown, setShown] = useState(active ? 0 : text.length); + useEffect(() => { + if (!active) { + setShown(text.length); + return; + } + setShown(0); + const timer = setInterval( + () => + setShown((count) => { + if (count >= text.length) { + clearInterval(timer); + return count; + } + return count + 1; + }), + CHARACTER_MS, + ); + return () => clearInterval(timer); + }, [text, active]); + return text.slice(0, shown); +} + +/** + * Welcome's picture: OpenBot's orb listening while a request is typed, then working on it. + * + * A DRAWING OF THE COMPOSER, NOT THE COMPOSER. The real one reads its text once, when it mounts, + * so it cannot be typed into from outside; and a live editor in a picture would be one more thing + * to tab to. This is its outline: the same card, the same buttons, and the words arriving. + * + * Three beats: typing, handed off and working, done. With reduced motion it is the typed request + * with the orb listening. + */ +export function WelcomeDemo() { + return ( + + {({ beat, playing, reduced }) => ( + + )} + + ); +} + +function WelcomeScene({ + beat, + playing, + reduced, +}: { + beat: number; + playing: boolean; + reduced: boolean; +}) { + const typing = beat === 0; + const typed = useTyped(WELCOME_REQUEST, typing && playing); + const sent = !typing; + const state = typing ? "listening" : beat === 1 ? "thinking" : "done"; + return ( + <> + +
    +
    +

    + {sent && !reduced ? WELCOME_REQUEST : typed} + {typing && playing ? ( + + ) : null} +

    +
    + + + +
    + + + + 0 || sent + ? "bg-primary text-primary-foreground" + : "bg-muted text-muted-foreground", + )} + > + + +
    +
    +
    +

    + + + {beat === 2 ? "Found three places" : "Working on it"} + +

    +
    + + ); +} diff --git a/app/src/components/onboarding/intro-animation.tsx b/app/src/components/onboarding/intro-animation.tsx deleted file mode 100644 index 2ed60fa3b1..0000000000 --- a/app/src/components/onboarding/intro-animation.tsx +++ /dev/null @@ -1,51 +0,0 @@ -import { AnimatePresence, motion, useReducedMotion } from "motion/react"; -import { type ReactNode, useEffect, useState } from "react"; - -export type IntroScene = { id: string; label: string; content: ReactNode }; - -/** A looping introduction instead of more wizard steps. */ -export function IntroAnimation({ scenes }: { scenes: IntroScene[] }) { - const reducedMotion = useReducedMotion(); - const [scene, setScene] = useState(0); - useEffect(() => { - if (reducedMotion || scenes.length < 2) return; - const timer = window.setInterval( - () => setScene((current) => (current + 1) % scenes.length), - 4000, - ); - return () => window.clearInterval(timer); - }, [reducedMotion, scenes.length]); - - // Keep every scene available without auto-updating content when motion is reduced. - if (reducedMotion) { - return ( -
    - {scenes.map((item) => ( -
    {item.content}
    - ))} -
    - ); - } - - return ( -
    -
    - - - {scenes[scene]?.content} - - -
    -
    - ); -} diff --git a/app/src/components/onboarding/notify-step.tsx b/app/src/components/onboarding/notify-step.tsx new file mode 100644 index 0000000000..c7fb86341c --- /dev/null +++ b/app/src/components/onboarding/notify-step.tsx @@ -0,0 +1,172 @@ +import { useState } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; +import { attentionSummary } from "@/components/bot-profile/attention"; +import { appConfig } from "@/lib/generated/application-config"; +import { + requestNotificationPermission, + useNotificationPermission, +} from "@/lib/notifications/permission"; +import { StepIntro } from "./onboarding-frame"; +import type { OnboardingStep } from "./onboarding-wizard"; + +const PRODUCT = appConfig.brand.productName; + +/** How long "Notifications on" shows before the step moves on by itself. */ +export const GRANTED_BEAT_MS = 1_000; + +/** + * The notification the preview shows: the one this web app raises today, word for word. + * + * `AttentionNotifier` (`bot-profile/attention.tsx`) titles it " needs you" and says what is + * waiting with `attentionSummary`, which is called here so the body cannot drift from it. A Bot with + * a question and an approval waiting is the common case it fires for. + */ +export function notificationPreview(botName: string): { + title: string; + body: string; +} { + return { + title: `${botName} needs you`, + body: attentionSummary({ + agentId: "preview", + name: botName, + questions: 1, + approvals: 1, + handoffs: 0, + unread: 0, + paused: false, + notify: "all", + }), + }; +} + +/** + * Onboarding's Get notified step: a preview of the notification, and the button that asks the + * browser for permission. + * + * Its buttons are the frame's, so this returns the whole step. "Turn on notifications" asks the + * browser, and shows a spinner while its prompt is up. A yes turns the button green, "Notifications + * on", for a beat and then moves on by itself. A no, a prompt closed without an answer, or a + * browser with no notifications at all says so on the step, and the button becomes Continue. "Not + * now" moves on and asks nothing. The step is optional, so the button keeps its place when Not now + * goes. + */ +export function useNotifyStep(botName: string): OnboardingStep { + const permission = useNotificationPermission(); + const [asking, setAsking] = useState(false); + const [dismissed, setDismissed] = useState(false); + const granted = permission === "granted"; + // Answered, or nothing to ask: the button only moves on from here. + const done = + permission === "denied" || permission === "unsupported" || dismissed; + + /* + * Why the button now says Continue, said just above it in the frame's notice slot rather than + * beside the preview: on a small screen the preview pushes anything under it out of view, and the + * person would see the button change with no reason. + */ + const said = + permission === "denied" + ? "Notifications are blocked for this site. Allow them in your browser's site settings to turn them on." + : permission === "unsupported" + ? "This browser can't show notifications." + : dismissed && permission === "default" + ? "You can turn them on later from a Bot's page." + : null; + + return { + id: "notify", + title: "Get notified", + intro: () => ( + +

    + Allow notifications so {PRODUCT} can tell you when a Bot needs you. +

    +     + ), + demo: () => , + notice: said ? ( +

    + {said} +

    + ) : undefined, + optional: true, + primaryLabel: granted + ? "Notifications on" + : done + ? "Continue" + : "Turn on notifications", + primaryDone: granted, + primaryPending: asking, + onPrimary: + granted || done + ? undefined + : (next) => { + setAsking(true); + void requestNotificationPermission().then((answer) => { + setAsking(false); + if (answer === "granted") setTimeout(next, GRANTED_BEAT_MS); + // The prompt closed with neither answer: the browser still says "default". + else if (answer === "default") setDismissed(true); + }); + }, + secondary: granted || done ? undefined : { label: "Not now" }, + footnote: `Notifications show while ${PRODUCT} is open in a browser tab.`, + }; +} + +/** + * A ghost phone, so the card reads as something arriving on a screen: taller than it is wide, its + * top fading out rather than drawn, the notification over it and four app tiles under it. + */ +function NotifyPreview({ botName }: { botName: string }) { + const preview = notificationPreview(botName); + return ( +
    +
    +
    +
    + + + + + + + {preview.title} + + + now + + + + {preview.body} + + +
    +
    + {[0, 1, 2, 3].map((tile) => ( + + ))} +
    +
    +
    + ); +} diff --git a/app/src/components/onboarding/onboarding-frame.tsx b/app/src/components/onboarding/onboarding-frame.tsx new file mode 100644 index 0000000000..8383db7578 --- /dev/null +++ b/app/src/components/onboarding/onboarding-frame.tsx @@ -0,0 +1,399 @@ +import { IconCheck, IconChevronLeft } from "@tabler/icons-react"; +import { type ReactNode, useLayoutEffect, useRef } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; +import { buttonVariants } from "@/components/ui/button"; +import { appConfig } from "@/lib/generated/application-config"; +import { cn } from "@/lib/utils"; + +/** + * The frame every onboarding step sits in: one DOM, laid out two ways by CSS. + * + * FROM `lg` UP, TWO PANES. On the left, the steps as a list with the one on screen marked, then + * the step's heading and sentence, then Continue and Back. On the right, filling the rest of the + * screen, the step's picture. Steps already reached can be picked from the list; the ones ahead + * cannot. + * + * ONE SCREEN TALL ON A DESKTOP, AND THE LIST IS WHAT GIVES. Nine steps, a heading, a few lines and + * the buttons are taller than a laptop's window on the longer steps, which used to cut Continue and + * Back off at the bottom. So the frame is the window's height, the step's words and its buttons + * keep their full size, and the step list takes what is left: on a short screen it scrolls within + * itself, with the step on screen kept in view. Its rows are spaced so that every step still fits + * whole in a 1280 by 800 window; the list scrolls only below that, and only a window too short for + * the words and the buttons themselves scrolls the page. + * + * BELOW `lg`, ONE COLUMN, for a phone: a round back chevron at the top left, the steps as dashes + * across the top, the heading and its picture, and the buttons pinned to the bottom, full width, + * with a fade above them where the step scrolls under. + * + * The frame owns where things are, never what a step says. Back, the list and the buttons sit in + * the same place on every step, so moving through onboarding never moves the control somebody is + * about to press. Back is drawn twice, once for each layout, and only one is ever displayed: each + * sits where its layout's reading order puts it, so the keyboard meets it where the eye does. + * + * AN OPTIONAL STEP HAS A SECOND BUTTON. "Not now" goes beside the primary (under it on a phone, the + * same size and shape, outlined), and its place is kept when it goes (`reserveSecondary`), so the + * primary does not move mid-step. A footnote, where a step has one, sits just above the buttons. + */ +export function OnboardingFrame({ + titles, + index, + reached, + onBack, + onPick, + primary, + secondary, + reserveSecondary = false, + footnote, + notice, + intro, + demo, + demoFirst = false, +}: { + /** Every step's name, in order. */ + titles: string[]; + /** Zero-based. */ + index: number; + /** The furthest step reached: every step up to it can be picked from the list. */ + reached: number; + onBack: () => void; + onPick: (index: number) => void; + /** Null when the step's own form carries its button. */ + primary: { + label: string; + onClick: () => void; + disabled?: boolean; + /** Drawn green with a check: what the button asked for has happened. */ + done?: boolean; + /** Waiting on an answer, such as the browser's prompt: a spinner beside the label. */ + pending?: boolean; + } | null; + /** A way past the step, such as Not now or Skip intro. */ + secondary?: { label: string; onClick: () => void }; + /** Keep the second button's place while it has none, so the primary doesn't move. */ + reserveSecondary?: boolean; + /** A line about the step, just above the buttons. */ + footnote?: ReactNode; + /** Said just above the buttons, such as a save that failed. */ + notice?: ReactNode; + /** The heading, its sentence, and anything that belongs with them. */ + intro: ReactNode; + /** The picture, or the step's own rows. */ + demo?: ReactNode; + /** Below `lg`, the picture above the heading. */ + demoFirst?: boolean; +}) { + const total = titles.length; + const shape = + "h-[52px] w-full rounded-full px-5 type-sub font-semibold lg:h-10 lg:w-auto lg:min-w-32 lg:rounded-lg"; + return ( +
    + {/* + * On a desktop the list's row is the one that shrinks (`minmax(10rem,auto)` above, against + * the words' and the buttons' `auto`), down to room for the product's name and a step or two. + */} +
    + {/* + * The phone's chrome. Back is not drawn on the first step, rather than drawn disabled: there + * is nowhere behind it, and a control that does nothing is one more thing a screen reader + * stops on. + */} +
    + {index > 0 ? ( + + ) : null} + +
    + +
    + +
    + {intro} +
    + + {/* + * On a desktop the picture fills its column by stretching, at most a screen tall, rather than + * being set to the screen's height: a set height spanning every row would hand its height to + * the empty last row before the step list got its own. No margin there either, which would + * stop it short of the bottom. + */} + {demo ? ( +
    +
    {demo}
    +
    + ) : null} + + {/* + * On a phone, a step taller than the screen scrolls under the buttons rather than being cut + * at them: the fade above says there is more. On a desktop the buttons follow the sentence. + */} +
    +
    + {notice} + {footnote ? ( +

    + {footnote} +

    + ) : null} +
    + {index > 0 ? ( + + ) : null} + {primary ? ( + + ) : null} + {secondary ? ( + + ) : reserveSecondary ? ( +
    + ) : null} +
    +
    +
    +
    + ); +} + +/** + * The steps as a list, on a desktop: each with a mark, done ones ticked, the one on screen filled. + * A step already reached is a button back to it; one ahead is only its name. + */ +function StepList({ + titles, + index, + reached, + onPick, +}: { + titles: string[]; + index: number; + reached: number; + onPick: (index: number) => void; +}) { + // On a screen too short for every step, the list scrolls: keep the one on screen in it, before + // the first paint, so arriving on a late step never shows the list's top for a frame. + const list = useRef(null); + useLayoutEffect(() => { + list.current?.children[index]?.scrollIntoView?.({ block: "nearest" }); + }, [index]); + return ( + + ); +} + +/** + * Where somebody is, as short dashes with the current one filled, on a phone. + * + * A progress bar to assistive technology, read as "Step 3 of 7": the dashes are a picture of that + * sentence, so they are hidden and the sentence is what is announced. + */ +export function StepDashes({ index, total }: { index: number; total: number }) { + return ( +
    + {Array.from({ length: total }, (_, at) => ( + + ))} +
    + ); +} + +/** + * A step's heading and sentence, the way every step draws them: centred on a phone, flush left in + * the desktop's left pane. `children` is the sentence; `after` is a control that belongs with it. + */ +export function StepIntro({ + title, + children, + after, +}: { + title: ReactNode; + children?: ReactNode; + after?: ReactNode; +}) { + return ( +
    +

    + {title} +

    + {children ? ( +
    + {children} +
    + ) : null} + {after ?
    {after}
    : null} +
    + ); +} diff --git a/app/src/components/onboarding/onboarding-wizard.tsx b/app/src/components/onboarding/onboarding-wizard.tsx new file mode 100644 index 0000000000..0867836c4b --- /dev/null +++ b/app/src/components/onboarding/onboarding-wizard.tsx @@ -0,0 +1,277 @@ +import { AnimatePresence, MotionConfig, motion } from "motion/react"; +import { type ReactNode, useEffect, useRef, useState } from "react"; +import { OnboardingFrame } from "./onboarding-frame"; + +/** + * One step: an id that outlives the list changing around it, its name in the step list, its words + * and its picture, and what its buttons say and do. Ids are this app's own fixed words, which is + * what lets them sit in a selector unescaped. + * + * TWO HALVES, BECAUSE A DESKTOP SHOWS THEM SIDE BY SIDE. `intro` is the step's heading (an `h1`), + * its sentence and any control that belongs with them; `demo` is what it shows. From `lg` up the + * intro sits under the step list on the left and the demo fills the right; below, the demo follows + * the intro, or comes first where `demoFirst` says so (the name step, under its orb). + * + * The buttons are the frame's, so a step describes them rather than drawing them: the primary's + * words where the step has its own action ("Turn on notifications"), what that action does given + * the move to the next step, and, on an optional step, a second button such as "Not now" that + * moves on and writes nothing. A step whose own form carries its button (`ownAction`) gets no + * primary at all. + */ +export type OnboardingStep = { + id: string; + /** The step's name in the step list: short, a few words. */ + title: string; + intro: () => ReactNode; + demo?: () => ReactNode; + /** Below `lg`, the demo above the intro rather than after it. */ + demoFirst?: boolean; + /** The primary's words here. "Continue" when absent. */ + primaryLabel?: string; + /** What the primary does, given the move to the next step. Moving on when absent. */ + onPrimary?: (next: () => void) => void; + /** The primary reads as done, green with a check. */ + primaryDone?: boolean; + /** The primary is waiting on something, such as the browser's permission prompt. */ + primaryPending?: boolean; + primaryDisabled?: boolean; + /** The step's own form has the button that moves on, so the frame draws none. */ + ownAction?: boolean; + /** A way past it, beside the primary. Moves on when it has no action of its own. */ + secondary?: { label: string; onClick?: (next: () => void) => void }; + /** + * An optional step keeps the second button's place even while it has none, so the primary stays + * put for the whole step: Get notified drops Not now once the browser has answered. + */ + optional?: boolean; + /** A line about the step, above the buttons. */ + footnote?: ReactNode; + /** Something the step has to say right above its buttons, such as why the button became Next. */ + notice?: ReactNode; +}; + +/** + * A step crossfades with a short travel toward the way the journey is going: the chrome around it + * stays still, so only the content moves, and never by more than 24 px. + */ +const variants = { + initial: (direction: number) => ({ x: 24 * direction, opacity: 0 }), + active: { x: 0, opacity: 1 }, + exit: (direction: number) => ({ x: -24 * direction, opacity: 0 }), +}; + +/** A pane that crossfades as the step changes, keyed on the step so the old one can leave. */ +function Pane({ + stepId, + direction, + children, + attribute, +}: { + stepId: string | undefined; + direction: number; + children: ReactNode; + attribute: "data-step" | "data-step-demo"; +}) { + return ( + + + {children} + + + ); +} + +/** + * The steps, moved through one at a time inside {@link OnboardingFrame}. + * + * WHERE SOMEBODY IS, BY ID. The list grows after it first draws: the self-hosting step joins once + * the server says this is somebody trying OpenBot alone, and that answer can arrive while they are + * on step two. Held as an index, their place would slide; held as an id, the step they are on stays + * the step they are on, and only the list around it changes. + * + * WHO HOLDS THE ID. The route does, through `stepId` and `onMove`, because the step lives in the + * address (`?step=`) so that a reload, browser Back and a consent's return all land on the right + * one. Without `onMove` the wizard keeps the id itself, from `initialStepId`. + * + * HOW FAR SOMEBODY HAS COME. Every step up to the furthest one they have been on can be picked + * from the desktop's step list; the ones after it cannot, because nobody has seen them yet. + * + * FOCUS FOLLOWS THE STEP. A change of step replaces the heading, the picture and the buttons, so + * the element that had focus either vanished or now means something else. Focus moves to the new + * step's heading, which a screen reader then reads, and the first draw leaves focus alone. Motion + * respects the reader's reduced-motion setting: the movement becomes a fade. + */ +export function OnboardingWizard({ + steps, + stepId: heldStepId, + onMove, + initialStepId, + notice, +}: { + steps: OnboardingStep[]; + /** The step to show, when the caller holds it. */ + stepId?: string; + /** Asked to move to another step, when the caller holds it. */ + onMove?: (to: string, way: "forward" | "back") => void; + initialStepId?: string; + /** Said above the buttons on every step, such as a save that failed. */ + notice?: ReactNode; +}) { + const [own, setOwn] = useState(() => initialStepId ?? steps[0]?.id ?? ""); + const current = heldStepId ?? own; + const container = useRef(null); + + // A step that left the list (the server's answer changed) falls back to the first, never to nothing. + const found = steps.findIndex((step) => step.id === current); + const index = found === -1 ? 0 : found; + const step = steps[index]; + + /* + * The furthest step reached, by id, so a step joining the list (the self-hosting one, once the + * server answers) does not shift it. Every step up to it can be picked from the list. + */ + const visited = useRef(new Set()); + if (step) visited.current.add(step.id); + const reachedAt = steps.reduce( + (furthest, candidate, at) => + visited.current.has(candidate.id) ? Math.max(furthest, at) : furthest, + index, + ); + + // The way the journey is going, for the motion: toward a later step, or back to an earlier one. + const shown = useRef(index); + const way = useRef(1); + if (index !== shown.current) { + way.current = index > shown.current ? 1 : -1; + shown.current = index; + } + const direction = way.current; + + const go = (to: number) => { + const target = steps[to]; + if (!target || to === index) return; + if (onMove) onMove(target.id, to > index ? "forward" : "back"); + else setOwn(target.id); + }; + /* + * On from the step this was handed to, not from whichever step is current when it runs: a step + * that moves on after a beat (Get notified, once the browser said yes) must not skip a step if + * the person pressed Back or Continue during that beat. + */ + const showing = useRef(step?.id); + showing.current = step?.id; + const from = step?.id; + const next = () => { + if (showing.current !== from) return; + go(index + 1); + }; + + /* + * Keyed on the id, not the step object: the list is rebuilt on every render of the route, and a + * fresh object would pull focus back to the heading each time anything on the page changed, such + * as the checklist's answer arriving while somebody tabs through it. The first draw is skipped, + * so arriving on a page leaves focus where the browser put it. + */ + const stepId = step?.id; + const arrivedOn = useRef(stepId); + const moved = useRef(false); + useEffect(() => { + if (!stepId) return; + if (!moved.current && stepId === arrivedOn.current) return; + moved.current = true; + // The new heading starts at the top, on the page and in the desktop's picture pane. + window.scrollTo?.({ top: 0 }); + const frame = container.current; + const demo = frame?.querySelector("[data-onboarding-demo]"); + if (demo) demo.scrollTop = 0; + const heading = frame?.querySelector( + `[data-step="${stepId}"] h1`, + ); + if (!heading) return; + // Focusable by script only, so it is not a stop when tabbing through the page. + heading.tabIndex = -1; + heading.focus({ preventScroll: true }); + }, [stepId]); + + return ( + +
    + + {step.demo()} + + ) : null + } + demoFirst={step?.demoFirst === true} + footnote={step?.footnote} + index={index} + intro={ + + {step?.intro()} + + } + notice={ + step?.notice || notice ? ( + <> + {step?.notice} + {notice} + + ) : undefined + } + onBack={() => go(index - 1)} + onPick={(at) => { + if (at <= reachedAt) go(at); + }} + primary={ + step?.ownAction + ? null + : { + label: step?.primaryLabel ?? "Continue", + disabled: + step?.primaryDisabled === true || + step?.primaryPending === true, + done: step?.primaryDone === true, + pending: step?.primaryPending === true, + onClick: () => { + if (step?.onPrimary) return step.onPrimary(next); + next(); + }, + } + } + reached={reachedAt} + reserveSecondary={step?.optional === true} + secondary={ + step?.secondary + ? { + label: step.secondary.label, + onClick: () => + step.secondary?.onClick + ? step.secondary.onClick(next) + : next(), + } + : undefined + } + titles={steps.map((candidate) => candidate.title)} + /> +
    +     + ); +} diff --git a/app/src/components/onboarding/setup-checklist.tsx b/app/src/components/onboarding/setup-checklist.tsx new file mode 100644 index 0000000000..70624db64f --- /dev/null +++ b/app/src/components/onboarding/setup-checklist.tsx @@ -0,0 +1,269 @@ +import { + IconApps, + IconBrain, + IconDeviceDesktop, + IconMessages, + IconSparkles, +} from "@tabler/icons-react"; +import { useMutation, useQuery } from "@tanstack/react-query"; +import { Fragment } from "react"; +import { PageRows } from "@/components/layout/page-shell"; +import { Button } from "@/components/ui/button"; +import { + Item, + ItemActions, + ItemContent, + ItemDescription, + ItemFooter, + ItemMedia, + ItemTitle, +} from "@/components/ui/item"; +import { Separator } from "@/components/ui/separator"; +import { appConfig } from "@/lib/generated/application-config"; +import { recheckSetupMutationOptions } from "@/lib/setup/mutations"; +import { + type SetupFix, + type SetupRow, + type SetupRowId, + setupChecklistQueryOptions, +} from "@/lib/setup/queries"; +import { cn } from "@/lib/utils"; +import { queryClient } from "@/query-client"; + +const ICONS: Record = { + intelligence: IconBrain, + model: IconSparkles, + computer: IconDeviceDesktop, + apps: IconApps, + chat: IconMessages, +}; + +const PRODUCT = appConfig.brand.productName; + +/** + * What this deployment has, and what still needs setting up: one row each, with a pill saying + * Ready, Not checked or Needs setup. + * + * The rows, their sentences and their fixes are the server's (see `lib/setup/queries.ts`): this + * draws them, and holds the one thing only a browser can do, which is asking it to check again. + * An admin is handed the page to open or the variables to set; a member is handed neither, because + * the server never sends them one. + * + * NOTHING HERE BLOCKS ANYTHING. A required gap is drawn plainly, in red, and the step around it still + * lets the person carry on: they cannot set a server's environment from a browser, and keeping them + * out of the app would not set it either. + */ +export function SetupChecklist({ + openPage, +}: { + /** + * Opens one of the app's own screens. Onboarding has to be finished before any other page opens, + * so it carries the page to its last step and opens it from there. + */ + openPage: (target: { label: string; path: string }) => void; +}) { + const checklist = useQuery(setupChecklistQueryOptions()); + const recheck = useMutation(recheckSetupMutationOptions(queryClient)); + + /* + * Check again asks the server to ask Intelligence, the model provider and the computers again, + * and waits for their answers. A plain refetch is answered from what they last said, and asks + * again only for the read after it, once an answer failed or is a few minutes old. A checklist + * that could not be read at all is read again first. + */ + const checking = checklist.isFetching || recheck.isPending; + const checkAgain = ( + + ); + + return ( +
    + {recheck.error ? ( +

    + Couldn't check again. Try again. +

    + ) : null} + {checklist.isPending ? null : checklist.error ? ( + <> +

    + Couldn't check your setup. Check again, or carry on and look later. +

    + {checkAgain} + + ) : ( + <> +

    + {summaryOf(checklist.data.rows)} +

    + +
      + {checklist.data.rows.map((row, at) => ( +
    • + {at > 0 ? : null} + +
      • + ))} +
    +     + {checkAgain} + + )} +
    + ); +} + +/** One line over the list, so the state of the whole list is said before its rows are read. */ +export function summaryOf(rows: SetupRow[]): string { + const gaps = rows.filter((row) => row.status === "needs_setup"); + const unchecked = rows.filter((row) => row.status === "not_checked").length; + const notChecked = + unchecked === 0 + ? null + : unchecked === 1 + ? "1 thing isn't checked yet." + : `${unchecked} things aren't checked yet.`; + if (gaps.length === 0) return notChecked ?? "Everything is set up."; + const required = gaps.filter((row) => row.required).length; + const things = + gaps.length === 1 ? "1 thing needs" : `${gaps.length} things need`; + const needs = + required > 0 + ? `${things} setup, and ${required === 1 ? "1 is" : `${required} are`} required.` + : `${things} setup. ${PRODUCT} works without ${gaps.length === 1 ? "it" : "them"}.`; + return notChecked ? `${needs} ${notChecked}` : needs; +} + +/** The pill's words and colours: done green, a gap that blocks red, everything else amber. */ +function pillFor(row: SetupRow) { + if (row.status === "ready") + return { label: "Ready", tone: "bg-status-done-soft text-status-done" }; + if (row.status === "not_checked") + return { + label: "Not checked", + tone: "bg-status-needs-soft text-status-needs", + }; + return { + label: "Needs setup", + tone: row.required + ? "bg-status-problem-soft text-status-problem" + : "bg-status-needs-soft text-status-needs", + }; +} + +function SetupItem({ + row, + openPage, +}: { + row: SetupRow; + openPage: (target: { label: string; path: string }) => void; +}) { + const Icon = ICONS[row.id]; + const pill = pillFor(row); + return ( + + + + + + {/* `line-clamp-none` makes the title a block, so the tag keeps its own space. */} + + {row.title} + {row.required ? ( + + Required + + ) : null} + + + {row.detail} + + + + + {pill.label} + + + {!row.ready && row.fix ? ( + + + + ) : null} + + ); +} + +const LINK = + "rounded-sm font-medium text-brand underline-offset-2 outline-none hover:underline focus-visible:underline focus-visible:ring-3 focus-visible:ring-ring/50"; + +/** How to close one gap: a page to open, or the variables to set and a guide. */ +function Fix({ + fix, + openPage, +}: { + fix: SetupFix; + openPage: (target: { label: string; path: string }) => void; +}) { + const page = (target: { label: string; path: string }) => ( + { + event.preventDefault(); + openPage(target); + }} + > + {target.label} + + ); + const guide = (url: string) => ( + + Setup guide + + ); + + if (fix.kind === "page") { + return ( +

    + Open {page(fix)}. {fix.docsUrl ? guide(fix.docsUrl) : null} +

    + ); + } + + return ( +

    + Set{" "} + {fix.names.map((name, at) => ( + + {at > 0 ? (at === fix.names.length - 1 ? " and " : ", ") : null} + + {name} + + + ))} + , then restart {PRODUCT}. + {fix.page ? <> Or add one in {page(fix.page)}. : null}{" "} + {guide(fix.docsUrl)} +

    + ); +} diff --git a/app/src/components/responsibilities/new-responsibility-dialog.tsx b/app/src/components/responsibilities/new-responsibility-dialog.tsx index 534f0e02cf..f167c78e0b 100644 --- a/app/src/components/responsibilities/new-responsibility-dialog.tsx +++ b/app/src/components/responsibilities/new-responsibility-dialog.tsx @@ -28,7 +28,10 @@ import { import { Textarea } from "@/components/ui/textarea"; import { agentListQueryOptions } from "@/lib/agents/queries"; import { conversationLabel } from "@/lib/channels/label"; -import { channelListQueryOptions } from "@/lib/channels/queries"; +import { + channelListQueryOptions, + conversationsOnly, +} from "@/lib/channels/queries"; import { createResponsibilityMutationOptions, type ResponsibilityRecord, @@ -74,12 +77,11 @@ export function NewResponsibilityDialog({ const availableBots = bots.data?.filter((bot) => selectedChannel?.agentIds.includes(bot.id)) ?? []; - const offered = - channels.data?.filter( - (channel) => - channel.active && - (fixedBot === undefined || channel.agentIds.includes(fixedBot)), - ) ?? []; + const offered = conversationsOnly(channels.data).filter( + (channel) => + channel.active && + (fixedBot === undefined || channel.agentIds.includes(fixedBot)), + ); const create = useMutation(createResponsibilityMutationOptions(queryClient)); return ( !next && onClose()} open> diff --git a/app/src/components/settings/profile.tsx b/app/src/components/settings/profile.tsx index 3f4069632d..d26a6d1b55 100644 --- a/app/src/components/settings/profile.tsx +++ b/app/src/components/settings/profile.tsx @@ -1,7 +1,8 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { useId, useState } from "react"; -import AgentOrb from "@/components/agents/orb/agent-orb"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; import { PageSection } from "@/components/layout/page-shell"; +import { StepIntro } from "@/components/onboarding/onboarding-frame"; import { Button } from "@/components/ui/button"; import { Input } from "@/components/ui/input"; import { authKeys, currentUserQueryOptions } from "@/lib/auth/queries"; @@ -43,7 +44,9 @@ export function ProfileNameForm({ if (name.trim() && !save.isPending) save.mutate(name.trim()); }} > - + @@ -95,6 +103,12 @@ export function ProfileSettings() { ); } +/** + * Onboarding's last step, under the orb: the person's name, and Get started, which saves it. + * + * Its heading and form are the step's words; the orb above them is its picture, which the + * onboarding frame draws beside it on a desktop and above it on a phone ({@link OnboardingNameOrb}). + */ export function OnboardingNameStep({ onSaved, }: { @@ -102,19 +116,40 @@ export function OnboardingNameStep({ }) { const { data: user } = useQuery(currentUserQueryOptions()); return ( -
    - -

    - Let's get to it. -

    -

    + + } + title="Let's get to it." + > +

    What should we call you? You can change your name later in Settings.

    - -
    + + ); +} + +/** + * OpenBot's orb, listening while somebody types their name. `finished` plays its finish once: the + * check and its burst of rays, which is how onboarding ends. + */ +export function OnboardingNameOrb({ + finished = false, + size = 64, +}: { + finished?: boolean; + size?: number; +}) { + return ( + + + ); } diff --git a/app/src/components/suggestions/proactive-panel.tsx b/app/src/components/suggestions/proactive-panel.tsx index fa428f7eda..1e5bda5d9c 100644 --- a/app/src/components/suggestions/proactive-panel.tsx +++ b/app/src/components/suggestions/proactive-panel.tsx @@ -7,6 +7,7 @@ import { IconRadar, } from "@tabler/icons-react"; import { useInfiniteQuery, useMutation, useQuery } from "@tanstack/react-query"; +import { useRouter } from "@tanstack/react-router"; import { Fragment, useId, useState } from "react"; import { PageEmpty, @@ -44,7 +45,10 @@ import { import { Separator } from "@/components/ui/separator"; import { agentListQueryOptions } from "@/lib/agents/queries"; import { conversationLabel } from "@/lib/channels/label"; -import { channelListQueryOptions } from "@/lib/channels/queries"; +import { + channelListQueryOptions, + conversationsOnly, +} from "@/lib/channels/queries"; import { changeProactiveSettingMutationOptions, createProactiveSettingMutationOptions, @@ -69,7 +73,10 @@ const intervalItems = intervals.map((interval) => ({ const intervalLabel = (minutes: number) => intervals.find((interval) => interval.minutes === minutes)?.label ?? `Every ${minutes} minutes`; -/** Next steps a Bot proposed from background research: start one as a task, or dismiss it. */ +/** + * Next steps a Bot proposed from background research: start one as a task, or dismiss it. Start + * opens a task of its own, and the task's page opens with it. + */ /** How many suggestions show before the rest fold behind "Show more". */ const SUGGESTIONS_SHOWN = 3; @@ -85,6 +92,8 @@ export function SuggestionsInbox({ ); const bots = useQuery(agentListQueryOptions()); const act = useMutation(resolveSuggestionMutationOptions(queryClient)); + // Drawn on its own, as some tests do, there is no router: the task still starts, and is on Tasks. + const router = useRouter({ warn: false }); const [expanded, setExpanded] = useState(false); const botName = (id: string) => bots.data?.find((bot) => bot.id === id)?.name ?? "Your Bot"; @@ -148,7 +157,18 @@ export function SuggestionsInbox({ size="sm" disabled={act.isPending} onClick={() => - act.mutate({ id: suggestion.id, action: "start" }) + act.mutate( + { id: suggestion.id, action: "start" }, + { + onSuccess: (task) => { + if (task) + void router?.navigate({ + to: "/tasks/$taskId", + params: { taskId: task.id }, + }); + }, + }, + ) } > Start as task @@ -289,7 +309,7 @@ function TurnOnResearchDialog({ const [focus, setFocus] = useState(""); const [intervalMinutes, setIntervalMinutes] = useState(240); const id = useId(); - const eligibleChannels = (channels.data ?? []).filter( + const eligibleChannels = conversationsOnly(channels.data).filter( (channel) => channel.active && channel.agentIds.includes(agentId), ); const add = useMutation(createProactiveSettingMutationOptions(queryClient)); diff --git a/app/src/components/tasks/new-task-dialog.tsx b/app/src/components/tasks/new-task-dialog.tsx new file mode 100644 index 0000000000..becb8d913d --- /dev/null +++ b/app/src/components/tasks/new-task-dialog.tsx @@ -0,0 +1,164 @@ +import { useMutation, useQuery } from "@tanstack/react-query"; +import { useId, useState } from "react"; +import { Button } from "@/components/ui/button"; +import { + Dialog, + DialogBody, + DialogContent, + DialogDescription, + DialogFooter, + DialogHeader, + DialogTitle, +} from "@/components/ui/dialog"; +import { Field, FieldGroup, FieldLabel } from "@/components/ui/field"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select"; +import { Textarea } from "@/components/ui/textarea"; +import { defaultAgentProfile } from "@/lib/agents/default-agent"; +import { agentListQueryOptions } from "@/lib/agents/queries"; +import { startTaskMutationOptions } from "@/lib/tasks/mutations"; +import type { TaskRecord } from "@/lib/tasks/queries"; +import { queryClient } from "@/query-client"; + +/** The longest ask the server takes (`POST /api/tasks`). */ +export const INSTRUCTION_MAX = 4000; + +/** + * The + on Tasks: which Bot, and what it should do. Start opens it as a task in a conversation of + * its own and starts its turn there, as a Bot handing work to a task from Chat would, and the task's + * page opens. The Bot is the person's default one until they choose another. + */ +export function NewTaskDialog({ + onClose, + onStarted, +}: { + onClose: () => void; + onStarted: (task: TaskRecord) => void; +}) { + const formId = useId(); + const bots = useQuery(agentListQueryOptions()); + const [picked, setPicked] = useState(""); + const [instruction, setInstruction] = useState(""); + const start = useMutation(startTaskMutationOptions(queryClient)); + const botId = picked || defaultAgentProfile(bots.data)?.id || ""; + const ask = instruction.trim(); + const send = () => { + if (!botId || !ask || start.isPending) return; + start.mutate({ botId, instruction: ask }, { onSuccess: onStarted }); + }; + return ( + { + if (!next && !start.isPending) onClose(); + }} + open + > + + + New task + + The Bot works on it in a conversation of its own. It shows up in + Tasks, and you can reply to steer it. + + + + {/* The submit button is in the footer and reaches this form by id, so DialogBody stays a + direct child of DialogContent and keeps scrolling. */} +
    { + event.preventDefault(); + send(); + }} + > + + + Bot + + + + + What should it do? + +