A multimodal AI-driven simulation environment. A multi-agent care team converses with the trainee; a real-rhythm physiologic monitor runs against emergency-driven scenarios with down-going trajectories; real-time affect capture and live process / sequence / transition-network analytics turn the session into a research-grade learning-analytics stream — every event row enriched with the contemporary vital signs.
Rohy is engineered to support CRETIC — Optimizing Clinical Reasoning in Time-Critical Scenarios: A data-driven multimodal approach, a Research Council of Finland project that develops an innovative gamified platform combining virtual patients and educational-escape-room elements to simulate clinical emergencies for healthcare students and trainees.
The platform is built to serve that research agenda — but is deliberately general. Every component above works in a non-CRETIC simulation context.
Project led by Sonsoles López-Pernas → · School of Computing, University of Eastern Finland.
Real clinical encounters are not a single back-and-forth with one person. A trainee — alone or working as part of a team — moves between the patient, the patient's family when the patient cannot give a history themselves, the bedside nurse who holds the medication record and the recent vitals, and the on-call consultant brought in for a second opinion. Each interlocutor is a distinct AI agent with its own persona, voice, knowledge scope, and arrival cue. Conversations branch and reconverge in a single session transcript, with the dialogue fidelity an actual clinical interview demands.
Sum-of-Gaussians waveform synthesis renders morphologically correct PQRST-T at any heart rate. Rhythms compose with modifier overlays — ST elevation, ST depression, electrolyte derangements, conduction blocks, ectopic beats, signal noise — so a single engine can drive the full continuum from physiological baseline through pathological extreme.
Cases run on a time-keyframed scenario engine with explicit down-going trajectories. Vitals drift toward decompensation unless the trainee intervenes appropriately. Alarms cross severity thresholds, banners surface, audio escalates; acknowledgement and snooze route the alarm through audit history rather than silencing it.
The trainee can type to the patient or speak. A 3D talking head responds with viseme-driven lipsync; in voice mode, a glassy cinema-style subtitle overlay renders the current line as the model streams it. Sentence-level pre-fetching keeps time-to-first-audio sub-second on local engines.
Laboratory and Radiology are first-class rooms in the simulation, not modal popups. A worklist tracks pending orders and ready reports; clicking a ready row opens the hospital-style report inline and adds it to a dismissible pill stack. Reference ranges are gender-specific where it clinically matters; flags surface high/low/abnormal automatically.
An anatomically accurate body silhouette with invisible polygon hit regions surfaces named exam areas. Each region accepts inspection, palpation, percussion, and auscultation as discrete techniques; auscultation includes audio playback at the relevant location.
A research-grade learning-analytics dashboard runs against the live session. Every trainee action is encoded as a verb-typed event, fed through unsupervised clustering, sequence and process mining, and surfaced as state distributions, directed weighted transition networks, and temporal proportion plots — per cluster and per session. The pipeline is the same one a learning-analytics researcher would build by hand. It runs while the case is still in progress.
Camera-side emotion analytics runs entirely in the browser. Multi-task affect models produce dominant-emotion estimates and valence/arousal regression at a configurable sample rate; only aggregated windows leave the device. The affect stream shares the session, the time base, and the active room with the action stream — so behavioural trajectories and affective trajectories can be analysed jointly without an offline join.
The session is not a black box. Every navigation, examination, order, treatment, message, and alarm response is captured as a verb-typed event; each row is enriched server-side with the active room, the contemporary vital signs — heart rate, blood pressure, SpO₂, respiratory rate, temperature, EtCO₂ — and the corresponding affect window at the moment the action occurred.
The End & Debrief button hands the finished case to a Socratic discussant — a senior clinician-educator persona with its own voice, avatar, and LLM, separate from the patient. The opening turn is sent as a silent LLM call so meta-prompts never appear in the learner-utterance audit trail; the debrief itself probes reasoning, scaffolds answers, and asks before telling.
Local LLMs and local TTS are first-class. Cloud providers are optional. The default install path puts a local voice in the patient's mouth and routes inference through a local model server on the same host. Air-gapped installation is a documented operator path.
Multi-tenant ready, role-hierarchy aware, audit-logged, soft-deleted with right-to-erasure purge, instrumented with structured observability.
Four install paths and two upgrade paths cover the realistic deployment surface — from a single classroom laptop to an air-gapped institutional host. The same operator tool handles in-place upgrades with snapshot and automatic rollback on failure.
Multi-arch image, smoke-tested against the release tag before it ships. Migrations auto-run on first boot.
Full production install — bootstrap script wires up rohy.service, nginx, certbot, and the operator user.
Personal devbox, classroom lab, demo box. No systemd, no nginx — runs directly under your user account.
Self-contained source tarball plus a Docker image tarball, both sha256-verified. Documented operator path for sites without internet.
Snapshot the DB, run the pre-flight, apply, verify with smoke and the 27-check tech-test; automatic rollback on failure.
Bump the image tag. Migrations auto-run on the new container's first boot; rollback is a pull of the previous tag.
Full operator manual: docs/INSTALL.md · docs/DEPLOY.md · docs/UPDATING.md · migrations/MANIFEST.md