Recognizing Game Screens When Text Is Not Enough

A loading screen can be obvious to a person while providing almost no useful text to an OCR system. A battlefield has the same problem: numbers may be readable, but they do not necessarily say which screen is open. In our Olden Era review gallery, these frames can remain UNKNOWN even when their interface layout is distinctive.

B022 will classify these screens from visual evidence when OCR cannot decide. It will look for combinations of stable UI regions, such as a loading bar or the panels surrounding a battlefield. It must also be able to say that the evidence is still insufficient.

graph LR
    A[Frame + OCR] --> B{Enough evidence?}
    B -->|Yes| E[Scene + readiness]
    B -->|No| C[B022 visual regions]
    C --> D[Candidate or abstention]
    D --> E

Figure 1. B022 supplies additional image evidence to scene understanding. A successful scene match does not automatically mean the page is ready for detailed processing.

This is a September 6 design record, illustrated with existing video frames. The project already has a visual-template baseline. B022’s expanded classifier and its integration into the OCR route are planned work; this article reports no new accuracy measurement.

1. Two screens that text rules miss

The first example appears after a match is set up. Most of the screen is an illustration; a progress bar near the bottom indicates loading. Reading more words cannot help if the distinctive evidence is the bar itself.

The second example is a battlefield. Two hero panels flank the screen, an action-order strip runs across the top, and controls sit along the bottom. Those regions together are stronger scene evidence than any one portrait or patch of terrain.

In the existing cached results, OCR scene understanding leaves both examples unknown. A separate visual-reference observer already labels the loading frame GAME_LOADING, but rejects the battlefield frame. This is a useful starting point: some visual recognition exists, but the two paths do not yet provide the proposed combined decision. These individual examples are development evidence, not a benchmark.

2. Where the visual fallback fits

These components occupy different roles in the shared Perception system:

Experiment	Question it answers
B020: system design	How do scene evidence, UI elements and temporal tracking fit together?
B021: OCR scene classification	Which scene do the visible words and positions support?
B022: visual scene fallback	Can the arrangement and appearance of UI regions identify a scene that text could not?
B023: scene-conditioned UI elements	Which nearby icon does a printed name describe, and which catalog entry does it identify?

The scene and video-review design separates scene identity from visibility, overlays and readiness. B022 follows that contract. It can supply a top-level BATTLEFIELD candidate while leaving deployment versus active battle unknown. An open panel might still prevent detailed processing.

The fallback should run when text is absent, insufficiently distinctive, or contradictory. OCR service failure is a different event and must keep its own error record. Counting words is not enough: a page may contain many numbers and repeated hero attributes without enough evidence to identify its current subscene.

The proposed output is a candidate label, the regions supporting it, match values, separation from competing candidates, and an explicit rejection reason when needed. The combined scene observer retains both the original OCR result and the visual result. Uncalibrated scores from different methods should not simply be added together.

Evidence situation	Intended behavior
OCR is insufficient; several visual regions support loading	Supply a loading candidate with region evidence
Battlefield layout is clear; subscene is not	Identify the top category and leave the subcategory unknown
OCR and visual evidence disagree	Preserve the conflict; wait or send the case for review
Neither method has enough evidence	Keep UNKNOWN and examine the next frame
Scene is recognizable but its choices are missing	Keep readiness separate and defer detail processing

Temporal context can help interpret a sequence, but it must remain distinguishable from this frame’s observations. The previous frame being a battlefield is not proof that the current frame shows one: the player could have switched out of the game.

3. Start with UI regions

The existing TemplateUIRecognizer compares configured screenshot regions against reference images. Each region is resized to a small RGB image; the largest region error determines whether the layout fits. It also rejects unsupported aspect ratios and candidates that are too similar to one another. Its match score is a similarity measure, not a probability of correctness.

B022 should reproduce that baseline first, then test the following progression.

First, match several stable UI regions with some position tolerance. Establish the game viewport before applying relative coordinates. Search a small neighborhood around each expected position, then compare edges, gradients or normalized image correlation. These methods compare appearance or structure; their usefulness here still needs measurement.

For a battlefield, combine the side-panel frames, top strip and bottom controls. Prefer the surrounding interface over the identity inside a portrait or the map beneath it. Do not require a visible hexagonal grid: its display may change. For loading, combine the progress-bar frame with the surrounding layout, using examples with different illustrations and fill amounts. A single memorized illustration would be a fragile shortcut.

Acceptance needs both sufficient regional evidence and separation from other candidate layouts. A hero portrait alone must not make a draft page look like a battlefield. Thresholds, necessary regions and position tolerances should be selected on development data rather than asserted as universal constants.

Next, test learned image features if the templates remain brittle. A pretrained image encoder turns a region into a numerical feature vector. A simple classifier or comparison with reference vectors can then predict its class. Compare regional features with a whole-frame baseline to find out whether the classifier is relying on interface structure or merely recognizing a background.

Fine-tuning a larger classifier comes later, if diverse labeled recordings reveal errors that simpler methods cannot handle. The goal is measured precision with useful coverage; a more complex model does not establish either by itself.

4. Collect confusing cases too

The initial case list contains the loading frame, the battlefield frame, and a skill-selection page caught during a flip. The third frame matters because its hero header remains visible while the choice panel is empty.

The current OCR output already treats this third frame as a transition with content not ready. It is a control case, not another current OCR failure. Adding a visual fallback must not turn it into a confident ready page simply because the hero header matches a familiar template.

The dataset should include different loading artwork, progress amounts, heroes, maps and battlefield states. It also needs lookalikes: draft pages with hero portraits, menus with fantasy illustrations, large overlays, fades, black frames and desktop cutaways. A negative example for detail processing is not necessarily a negative example for scene recognition.

We will reuse the annotation website. Predictions, assistant suggestions and owner-confirmed references remain separate. Codex can prepare obvious labels and supporting regions; only ambiguous cases need owner review. Existing creature-label verification can continue independently.

Long stretches of nearly identical loading frames should not dominate evaluation. Keep representative frames and the boundaries before and after a change. Split data by complete recordings or matches, not by randomly mixing neighboring frames. The two existing walkthroughs are development material; new independent recordings are needed to evaluate generalization.

Although the main runtime trigger is insufficient OCR evidence, the evaluation should also include OCR-positive controls. Otherwise, we could miss false confidence in the first stage and mistakenly judge the complete system only on its easier failures.

5. Evaluate the complete route

Compare OCR alone, the existing visual baseline, the proposed region matcher, and the combined route on the same video-separated cases. Report how often fallback runs, how many additional scenes it identifies correctly, how often it accepts a wrong scene, and how often it still abstains. Show results by scene and by video rather than only a single aggregate score.

Readiness errors deserve a separate count. Calling a transition a battlefield can be a scene error; correctly calling it a battlefield while incorrectly declaring its obscured contents ready is a different failure. Both affect the later video-processing system.

The eventual review should show the original frame beside its scene candidate, supporting UI regions, original OCR findings and any rejection or readiness reason. A loading interval may then be represented by a few useful moments, while a change in visible choices remains reviewable even if the top-level scene stays the same.

The next deliverable is a measured baseline and a reviewable visual fallback, not a claim that battlefield perception is solved. B022 now has a defined scope, a three-case seed record and an evaluation plan. Implementation and independent-video testing follow; creature detection on the battlefield remains separate work.