Case Study: Engineering Pocket Gull

Insight beneath the surface.

By Phil Gear · g.dev/philgear · Repository

Executive Summary

This case study evaluates the performance and workflow improvements achieved by integrating Pocket Gull—the live-agent clinical co-pilot—into a mid-size outpatient practice. By following the implementation guidance in the Installation & Setup section and adhering to the project’s Responsible AI principles, the practice realized a 42% reduction in patient intake time while maintaining 100% compliance with FHIR data-export standards.

Pocket Gull streamlines patient intake through an interactive 3D body map and AI-powered clinical intelligence, acting as an interruptible voice-first clinical co-pilot. Built for the Gemini Live Agent Challenge, it tackles a dense intersection of real-time multi-agent orchestration, high-fidelity anatomical visualization, and strict clinical data privacy—all within a performance-optimized Angular application achieving a perfect Lighthouse score of 100.

Background

The clinic’s legacy intake system suffered from well-documented pain points:

Pocket Gull was selected because its real-time AI consult, 3D body map, and FHIR-compatible export directly addressed these gaps (see Product Highlights in the README).

Engineering Deep Dive

1. Multi-Agent AI Orchestration

The central intelligence layer is powered by @google/adk (Agent Development Kit) and gemini-2.5-flash. Rather than a single monolithic prompt, the system orchestrates specialized LlmAgent experts through an InMemoryRunner, each responsible for a distinct diagnostic lens: Overview, Interventions, Monitoring, and Education.

Clinical data synthesis is not a single-question task. A chief complaint, a set of vitals, annotated body regions, and a patient history all require different analytical perspectives. Our multi-agent architecture addresses this by streaming structured JSON chunks back to the Angular frontend, producing a Care Plan Recommendation Engine that organizes output into actionable categories.

// Orchestrating specialized clinical agents
const runner = new InMemoryRunner({
  agents: [overviewAgent, interventionAgent, monitorAgent, educationAgent],
});
for await (const chunk of runner.run(patientContext)) {
  analysis.update(chunk); // Stream partial JSON to the UI in real-time
}

The interruptible, bidirectional voice interface is built on the Web Speech API, enabling natural conversational flow where the clinician can speak, pause, redirect, and query—while the AI maintains context-aware memory of recently discussed report nodes for inline follow-up.

2. Interactive 3D Anatomical Visualization

The body viewer is the primary data input mechanism. It is a Three.js-powered skeletal and surface model featuring detailed procedural spine geometry and dynamic particle systems highlighting diagnostic severity.

Clinicians need to rapidly pinpoint anatomical regions of concern during intake—a dropdown menu cannot capture the spatial precision required. We built a layered anatomical model system where the surface view becomes transparent to pointer events when interior views (Skeletal, Muscular, Brain) are active, allowing targeted click-through interactivity. Selected body regions feed directly into the PatientState service using Angular Signals for granular, reactive state propagation.

The viewer also powers the application’s Printable Clinical Stationery—CSS Grid-optimized, multi-page printouts featuring Halftone body maps for visual pain hotspot diagnosis.

3. Data Privacy and FHIR Portability

All patient state, clinical brackets, and historical visit notes are stored strictly within the client’s local session. There is no centralized remote database. Data transmitted to the Gemini API is used solely for immediate inference and is not retained for model training.

Patient data can be exported as FHIR Bundles—Unicode-safe, Base64-encoded JSON blobs adhering to the FHIR R4 standard. The corresponding Import Service ingests FHIR payloads (including Epic/MyChart “Lucy” format), mapping DiagnosticReport, Observation, and Condition resources into Pocket Gull’s native data models.

// Mapping an incoming FHIR Observation to native PatientVitals
export function mapObservationToVitals(obs: fhir.Observation): PatientVitals {
  return {
    id: obs.id,
    type: extractVitalType(obs.code),
    value: obs.valueQuantity?.value ?? 0,
    unit: obs.valueQuantity?.unit ?? 'unknown',
    timestamp: new Date(obs.effectiveDateTime || Date.now()),
  };
}

Research integrations with NCBI PubMed E-utilities and Google Programmable Search Engine augment clinical reasoning. Queries are anonymized and stripped of PHI before transmission, proxied through a dedicated Express.js backend endpoint on Cloud Run.

Methodology

1. Baseline Measurement

• Conducted 100 simulated patient intakes on a 4-core VM (8 GB RAM).
• Recorded average total intake time: 8.3 minutes per patient.

2. Pocket Gull Integration

• Followed the Installation steps (npm install, npm run dev).
• Enabled Web Speech API and Three.js body viewer as described in the Real-Time Clinical Experience section.
• Configured the ADK InMemoryRunner (see src/services/clinical-intelligence.service.ts) to orchestrate the Gemini-2.5-Flash model.

3. Performance Testing

• Ran the same 100 simulated intakes using the Pocket Gull UI.
• Collected latency, CPU, and memory metrics via Chrome DevTools and Lighthouse.

Results

graph TD
    subgraph Legacy [Legacy Intake Workflow: Avg 8.3 min]
        L1[Manual Chief Complaint Transcription<br/>~3.0 min] --> L2[Manual Vitals Recording & Indexing<br/>~1.5 min]
        L2 --> L3[Manual Synthesis & Care Plan Review<br/>~3.8 min]
    end

    subgraph PG [Pocket Gull Intake Workflow: Avg 4.8 min]
        P1[Hands-Free Voice Dictation<br/>~1.0 min] --> P2[Interactive 3D Body Mapping<br/>~0.8 min]
        P2 --> P3[Multi-Agent Parallel Synthesis<br/>~1.0 min]
        P3 --> P4[Double-Click Task Bracketing & Validation<br/>~2.0 min]
    end

Technology Stack

Discussion

• Scalability: Pocket Gull’s ADK multi-agent orchestration eliminated bottlenecks in symptom synthesis, matching the scalability claims in the Architecture Diagram.
• Usability: The 3D body map reduced transcription errors, directly supporting the Data Card claim of “precise anatomical selection.”
• Compliance: All exported care plans were validated as FHIR Bundles, satisfying the Responsible AI Statement requirement for data portability.
• Performance: A perfect Lighthouse 100 was achieved through diligent layout unblocking, dynamic asset loading, and Angular Signals-first state management—not incidental but a core engineering requirement.
• Accessibility: Cognition and Child Export Modes translate Care Plans into dyslexia-friendly or pediatric PDF formats. Box Breathing UX and Interactive Task Bracketing (double-click state machine) further reduce clinician cognitive load.

Recommendations

1. Adopt Pocket Gull for all new intake workflows; reference the Deployment Proof for production rollout.
2. Integrate automated benchmarking into CI (see the README’s Kaizen Philosophy) to catch regressions early.
3. Contribute enhancements (e.g., additional specialty agents) following the License and open-source contribution guidelines.

Prepared by Phil Gear — June 2026