In Heroes of Might and Magic: Olden Era, Arena players assemble an army through screens offering Heroes, Skills, Artifacts and Creatures. To read an unfamiliar screen, a program must first locate the artwork. Only then can it ask which object the artwork represents.
This experiment gives a pretrained object detector the whole screenshot, not the known positions of the choices. A detector learns to propose rectangles. A separate image model compares each rectangle’s pixels with the clean icon catalog to suggest a name. These are different jobs: a good box can still receive the wrong name.
The result is encouraging within a small pilot. After fine-tuning, the detector finds all 24 labeled objects in eight holdout screenshots, with one extra box. But part of this holdout shares source recordings with training. It does not establish reliable battlefield creature detection.
flowchart LR
A[Whole game screenshot] --> B[Learned detector proposes boxes]
B --> C[Filter implausible sizes]
C --> D[Crop each remaining box]
D --> E[Separate catalog matcher suggests a name]
E --> F[Human reviews box and identity separately]Figure 1. Detection supplies geometry; catalog matching supplies an identity hypothesis. Neither output becomes ground truth automatically.
1. What is being measured?
A crop is the patch of pixels inside a proposed box. The catalog contains labeled reference artwork, such as a Hero portrait or an Artifact illustration. Our detector is class-agnostic: it predicts “candidate artwork,” not the exact Hero or Creature. The downstream B02 catalog matcher performs that second task.
To check geometry, we compare proposed boxes with human-confirmed boxes. Intersection over Union (IoU) is their overlapping area divided by their combined area. At the operating point, a localization match needs IoU of at least 0.50. Precision measures how many accepted boxes are real matches; recall measures how many labeled objects were found. F1 balances those two quantities.
Average precision (AP) also considers how predictions rank across score cutoffs. AP50 uses IoU 0.50; mAP50–95 averages AP over stricter overlap requirements from 0.50 to 0.95. We use the latter to select the initial detector, so a model with the best fixed-cutoff F1 need not win.
The reviewed pilot has 39 screenshots and 117 objects: 30 Heroes, 30 Skills, 27 Artifacts and 30 Creature-selection artworks. These are UI images—not a labeled collection of moving creatures on the battlefield. All detectors share a size filter: reject boxes smaller than 12 pixels on either side or larger than 18% of the screen area.
2. Start with four generic detectors
The four models start from official Torchvision weights trained on COCO, a photographic object dataset. “Zero-shot” here means no game-specific training, not no prior training. We ignore their original photographic class names and evaluate the rectangles.
| Detector | AP50 | mAP50–95 | Precision | Recall | F1 |
|---|---|---|---|---|---|
| Faster R-CNN ResNet-50 FPN V2 | 18.32% | 7.94% | 14.20% | 41.03% | 21.10% |
| RetinaNet ResNet-50 FPN V2 | 13.29% | 4.55% | 15.11% | 17.95% | 16.41% |
| FCOS ResNet-50 FPN | 22.73% | 11.89% | 9.13% | 55.56% | 15.68% |
| SSDlite320 MobileNetV3 Large | 2.50% | 0.81% | 11.76% | 1.71% | 2.99% |
The precision/recall/F1 columns use the same 0.20 score cutoff. FCOS wins on mAP50–95, our selection metric, and becomes the fine-tuning starting point. This comparison does not isolate why a particular architecture succeeds or fails.
There is already a warning about the second stage: given the human-drawn crops, the B02 matcher names Hero, Skill and Artifact correctly on this pilot, but gets only 53.3% of Creature-selection crops correct. Improving boxes alone cannot fix that identity problem.
3. Teach one detector the game layout
A backbone extracts visual features; a detection head reads those features to predict boxes and foreground scores. We keep FCOS’s backbone fixed and train only its heads—4.74 million parameters—for 12 training rounds, using one foreground class.
| Partition | Screenshots | Objects | Job |
|---|---|---|---|
| Training | 27 | 81 | Update detection heads |
| Validation | 4 | 12 | Select saved weights and acceptance cutoff |
| Holdout | 8 | 24 | Report the selected recipe |
Validation selects round 11 and cutoff 0.50 for fine-tuned FCOS, versus 0.35 for the original FCOS. The comparison below uses the same eight holdout frames for both. It is not the full 39-frame baseline table above.
| Holdout measurement | Original FCOS | Fine-tuned FCOS |
|---|---|---|
| AP50 | 23.08% | 100.00% |
| mAP50–95 | 11.56% | 81.85% |
| Localization precision | 35.29% | 96.00% |
| Localization recall | 25.00% | 100.00% |
| Localization F1 | 29.27% | 97.96% |
| Box-and-identity F1 | 19.51% | 85.71% |
The fine-tuned model supplies 24 matched boxes and one extra box. The catalog matcher correctly names 21 of those 24 matches. The combined metric requires both sufficient overlap and the right identity; it is stricter than localization F1.
This is not a fully video-isolated test. Artifact and Creature-selection training/holdout frames share source recordings. Hero and Skill holdout use different recordings. Moreover, the small original collection informed which model family to fine-tune. Treat the result as a layout-adaptation pilot, not an untouched estimate of performance on new games.
4. New screens expose a different problem
Next, the selected detector scans six recordings at 60 evenly spaced times per video. Four recordings are absent from the Golden Dataset; two are evaluation recordings sampled at new times. Eighteen temporally separated, high-response frames are retained, producing 216 candidates at cutoff 0.50.
This selection favors frames where the detector responds; it is not a random sample of all gameplay. Many contain battle or spellbook overlays rather than selection screens.
Figure 2. Detector hypotheses on a new-time spellbook screen. The boxes are predictions, not hand-verified labels. Empty slots, overlay controls and unfamiliar artwork can confuse the system.
This inspection shows why “detecting an icon” and “understanding battle” must remain separate claims. Some rectangles surround useful UI objects; others surround empty slots or unrelated regions. Even sensible rectangles can receive the wrong catalog identity. The 216 suggestions have no complete independent labels, so they provide failure examples—not a measured accuracy.
5. Where this fits in the project
This is Experiment B03. It differs from sliding-window catalog search: learned detection proposes the boxes instead of exhaustive ordinary code. It also differs from fixed-crop classification, which assumes that the correct UI slot is already known.
The September 3 ingestion pipeline uses class-agnostic detection to create coarse review candidates on battle and preparation frames. That does not validate those boxes as creatures. The larger generated corpus includes 27,743 such candidates; they must not be counted as 27,743 labeled battlefield training examples.
Printed names offer a complementary route on draft screens. The OCR experiment can suggest Creature identities for human review without depending on the current visual matcher. Transferring those corrected selection images to battlefield sprites is a future experiment, not a result of B03.
6. Evidence and next test
The next useful detector test needs reviewed battlefield boxes from separate recordings. It should distinguish UI icons, actual units, shadows/effects, empty cells and occluded units. Report localization and naming separately before reporting their combined success.
A good selection-screen detector may be a useful starting point. It is not evidence that standing, moving, flying and partly hidden units will all be found. Likewise, hex-grid registration can propose where to look, but its current measured errors do not justify treating inferred cells as reliable click targets.
Evidence sources: the experiment’s preserved benchmark-summary.json, data-split.json, fine-tuning-summary.json and blind-review-v1 outputs under detector-pipeline-v1. Code remains in perception/experiments/b03_off-the-shelf-detectors-and-segmentors/. This article publishes the existing pilot; no models were retrained for this writing pass.
Comments