A game video can spend twenty minutes waiting for a match, then reveal a useful skill choice for only a moment. Processing every frame equally wastes effort; keeping one frame per scene loses changes within that scene. We want a review tool that preserves the useful moments and explains what it recognized in each one.

The proposed pipeline reads text and visual evidence, estimates both which screen is visible and whether its contents are ready to inspect, then selects representative frames. The reviewer sees original pixels beside detected text, icon crops, identities and their relationships.

graph LR
    A[Video frames] --> B[OCR + images]
    B --> C[Scene + readiness]
    C --> D[UI changes]
    D --> E[Review]

Figure 1. The intended path from video to reviewable evidence. Frame selection must preserve changes in content, even when the scene label stays the same.

This is B020, the system-design entry for four narrower experiments. Existing extraction and review components provide a starting point. The twelve-category top-level scene/readiness contract is implemented; narrower local-view labels remain a separate annotation and recognition vocabulary. The combined visual fallback and adaptive video processing described here remain planned work. This article does not introduce a new accuracy or throughput result.

1. Four experiments, two recognition tracks

The experiment numbers now follow the recommended reading and dependency order. Publication dates still record when the underlying work was first documented, so they do not necessarily increase with that order. The system has two recognition tracks followed by a separate temporal layer:

graph TD
    O[Architecture guide] --> S[Scene recognition]
    O --> E[Scene-conditioned elements]
    S --> S1[B021: OCR + position]
    S1 --> S2[B022: visual fallback]
    E --> E1[B023: title-to-icon association]
    E1 --> E2[B024: bounded-title OCR evaluation]
    S2 --> T[Temporal tracking + keyframes]
    E2 --> T

Figure 2. Read B021 then B022 to understand scene recognition; read B023 then B024 to understand UI elements inside a known scene. Temporal tracking consumes their observations but is not itself a single-frame classifier.

Component Question Output
B020 · System design How do frame observations, UI elements and temporal tracking fit together? Shared boundaries, reader map and cross-experiment acceptance criteria
B021 · OCR scene recognition Which screen do the visible words and positions support? Text regions, current-scene evidence and separate phase context
B022 · Visual scene fallback Which scene does the stable UI layout support when text is insufficient? Visual scene candidate, supporting regions or abstention
B023 · Scene-conditioned elements Which nearby icon does a printed name describe? Text-to-crop links, catalog identities and review suggestions
B024 · Small-title OCR evaluation Which OCR recipe should B023 use for bounded title crops? Raw readings, scores, validity evidence and an engine choice

Reusable code lives in Perception’s ui_understanding and ui_navigation modules. A scene contains panels; panels contain text and icons. A printed name can link to an icon and a catalog entry. Those containment and association relationships describe one image. A separate directed graph describes possible changes between game states.

OCR is especially useful for menus and selection screens. For example, “Quick Start” and “Custom Game” below “New Game” identify an expanded menu. The detected positions also accommodate the menu items shifting downward. Fixed coordinates are useful priors, but they must not keep pointing at the old positions after expansion.

Text and scene are not strictly one-to-one. Hero names, levels and the bottom progress bar appear on several draft pages. Distinctive phrases and their arrangement matter more than the total number of words. Recognizing a battlefield is also a separate task from locating and identifying the creatures fighting on it.

2. Scenes follow a workflow; interruptions do not

The annotation page’s three columns—before starting a match, battle preparation, and battle started—help the owner find labels. They are display groups. Deployment belongs in the third group because it uses battlefield UI.

The proposed runtime hierarchy has twelve in-game top-level categories. Subcategories describe the page within a phase; allowed overlays describe temporary panels within that context.

Top-level scene Subcategories or local states
Menu Main menu, ranking; expanded submenus as local UI state
Match setup Match configuration
Match waiting Matchmaking, network validation
Loading Game loading
Hero selection Candidate and preview details
Skill selection Main skill, subskill
Artifact selection Artifact choices
Spell selection Spell choices
Creature selection Creature choices and refreshes
Draft finalization Attribute rewards, final confirmation
Battlefield Deployment, active battle
Match result Victory; other outcomes require representative examples 
graph LR
    subgraph BEFORE[Before drafting]
        direction TB
        M[Menu] --> S[Match setup]
        S --> W[Waiting / validation]
        W --> L[Loading]
    end
    subgraph DRAFT[Draft choices]
        direction TB
        H[Hero] --> K[Main / subskill]
        K --> A[Artifacts]
        A --> P[Spells]
        P --> C[Creatures]
    end
    subgraph FINISH[Finalization and battle]
        direction TB
        F[Attribute rewards] --> Q[Final confirmation]
        Q --> D[Deployment]
        D --> B[Battle]
        B --> R[Result]
    end
    BEFORE --> DRAFT --> FINISH

Figure 3. Read the normal flow from left to right, then downward within each group; a new match returns through the menu. This is not a complete executable graph. Quick Start can skip setup; cancellation, exits and recovery need their own verified edges. The owner confirmed that final confirmation cannot return to attribute editing.

Switching away from the game can happen anywhere. Non-game visibility therefore sits outside this progression. On return, the system must recognize the current screen again: the match may have advanced while hidden. A remembered state is context with an age, not proof of what is currently visible.

Overlays need parent constraints. A battle spellbook cannot be attached to the main menu. But an interrupt does not necessarily reveal the underlying scene, so the system must also be able to report “disconnect notice visible, base scene unknown.”

Condition First-version processing policy
Stable, readable game page Classify and extract relevant elements
Outside the game Suspend normal extraction; keep checking for return
Pause menu or disconnect notice Record the interruption; keep recovery monitoring alive
Supported or suspected transition Wait for another frame before committing complete-page details
Recognition failure Report unknown with a reason; do not relabel every failure as a transition
Tooltip Preserve the frame and overlay; defer blocked regions if necessary

An occluded icon is not a disappeared icon. A disconnected game is not an unimportant frame to discard. These distinctions affect extraction and tracking as well as annotation.

3. Recognize the scene and its readiness together

A strict “check quality first, classify the scene second” pipeline misses a dependency: knowing whether content is missing often requires knowing which scene is expected. A clear image of a half-built skill page can have excellent image quality and still be unready for extraction.

We will use one public scene-understanding entry point with separate outputs. Shared OCR and visual evidence inform both the scene hypothesis and readiness. This need not be one neural network; rules, OCR and image classifiers can cooperate behind that interface.

graph TD
    A[Frame + past context] --> B[OCR and layout]
    B --> C[Scene candidates + expected content]
    C --> D{Evidence sufficient?}
    D -->|Yes| F[Joint judgement]
    D -->|Missing or conflicting| E[Image / crop classifier]
    E --> F
    F --> G[Scene + readiness + policy]

Figure 4. The latest module design. Image evidence supplements missing or conflicting text evidence; the decision is not based on a word-count threshold.

Consider the empty skill page at 05:31 in session01. Its hero card is visible, but the skill choices have not appeared. The owner identified this as a page-turn transition. The earlier “Hero Skill Overview” label was removed from the current state nodes.

Two outputs are possible, depending on the actual evidence:

  • If scene-specific anchors still support skill selection, report skill selection / subcategory unknown / content not ready / wait.
  • If only the shared hero card and progress bar remain, report current scene unknown / content not ready / wait. Keep the preceding skill scene in a separate context field.

An OCR miss and an animation can both remove expected text from the recognizer’s output. A local image classifier may help distinguish them. Later frames can confirm that an interval was a transition, but this offline interpretation must remain separate from what the streaming system could know at the time.

The output contract should expose scene and subscene, game visibility, overlays, content readiness, the processing decision, supporting evidence and the age of retained context. Current observations, temporal estimates and transition hypotheses stay separate. A legal path through the graph does not prove that someone clicked a button or committed a choice.

4. Keep changes, not duplicates

The owner’s proposed starting point is roughly 10 frames per second for coarse classification, then up to 5 frames per second for detail in useful intervals. These are trial settings. We have not established processing speed, a best sampling rate or coverage of events shorter than the sampling interval.

During a twenty-minute matchmaking wait, the reviewer should see representative frames and the interval’s boundaries, plus any meaningful success, failure or disconnect notice. A changing timer usually need not produce a new review frame. During ten unchanged seconds of main-skill selection, one clear representative may be enough.

However, same scene does not mean same contents. Refreshing creature candidates, changing a highlighted choice, gaining a level, revealing a tooltip or reaching a confirmation state should preserve new evidence. Frame selection must compare relevant content and regions, not only the top-level label. Ambiguous boundaries can trigger denser sampling.

The target review page shows the original frame on the left and a result from the same timestamp on the right:

  • Game-related OCR text and boxes, including unresolved text that existing filtered views can hide.
  • Icon boxes, inspectable crops, bilingual identity suggestions and lines connecting printed names to their icons.
  • Scene, subscene, readiness, uncertain elements and explicit extraction failures.
  • Separate raw predictions and temporal revisions, with timestamps and the age of reused results.

Left and right arrow keys should move between meaningful keyframes. Repeated intervals, transitions and unknowns can be collapsed, but their time ranges and reasons remain inspectable. The original video is retained. This makes compression a review convenience rather than an invisible deletion of difficult evidence.

5. What we decided and what comes next

The discussion changed both the architecture and the annotation workload. This table records the decisions so that an older roadmap does not quietly become the current plan again.

Decision and source Effect
Owner: the blank skill page is a transition Remove the invented stable overview class; already corrected in the current nodes
Owner: finalization cannot return to editing Remove that edge; the current twelve-category contract implements the correction
Owner: switching away can happen at any time Separate visibility from the in-game progression
Owner: Codex can review ordinary scene examples Remove the requirement that the owner first annotate twenty scene clips
Owner: reduce long videos to useful frames Preserve content changes within a scene, not just scene changes
Joint design: readiness depends on candidate scene Share evidence and inference while retaining separate outputs
External review principle: constraints do not establish tracking accuracy Preserve unknowns, conflicts, evidence age and independent evaluation

The existing code provides OCR, catalog matching, title-to-icon relationships, prelabels, review pages and a twelve-category scene/readiness output. The combined visual fallback and adaptive keyframe review are not yet validated as one system. Temporal state tracking also lacks an independent sequence-level evaluation, so transition constraints cannot be treated as measured tracking accuracy.

The next delivery is one existing video’s non-battlefield recognition results, with enough detail for the owner to inspect text, crops and associations. Codex will prepare and visually check ordinary scene labels. The owner can continue verifying creature-selection identities and crop quality using the existing OCR suggestions. A successful catalog lookup is useful assistance, but does not prove that the adjacent crop is correct or unobscured.

AI-reviewed references, automatic predictions and human confirmations must retain their separate origins. The two walkthroughs already used to develop rules remain development material. An accuracy claim requires recordings and reference decisions independent of that tuning. Battlefield creature detection remains a later visual task; it is not solved by recognizing the battlefield scene.

The durable project record is perception/experiments/b020_scene-understanding-system-design/SYSTEM_DESIGN.md. Experiment folders keep their own configurations, fixtures and measurements; reusable behavior stays in the Perception package. Future changes should update the architecture record and Git history while leaving dated experiment measurements intact.