Do Verified Gameplay Crops Improve a Catalog-Trained Game UI Icon Classifier?

Game interfaces often place Hero, Main Skill, Artifact, and Creature candidates in predictable slots. Ordinary code can crop those slots; a visual model must then decide whether each crop is valid, what kind of object it contains, and which specific object it shows.

This experiment asks two narrow questions. First, does adding human-verified gameplay imagery improve a classifier trained on clean catalog artwork? Second, can a separately trained validity head reject unusable crops with fewer false rejects than a threshold on classification confidence?

Figure 1. The first comparison changes only the image source used for classifier training. The second freezes that classifier and trains only a validity head. Held-out data never selects checkpoints or thresholds.

1. The model sees a crop, not the game screen

The input is one 224×224 RGB crop. The model does not read the current scene, OCR text, slot semantics, neighboring video frames, detector output, or game rules. It produces three outputs:

1. VALID or REJECT;
2. one of four types: Hero, Main Skill, Artifact, or Creature;
3. an identity inside the predicted type.

This boundary deliberately excludes localization. There is no sliding window, box regression, or battlefield detection in this experiment. Separating crop geometry from visual classification makes it possible to test whether the pixels inside an already chosen crop are sufficient.

2. Frozen human-reviewed data and a source-video split

Training reads a physically copied, immutable snapshot rather than querying a changing annotation database. A gameplay crop enters the snapshot only when the frame, box, identity or reject reason, and review state have all been confirmed by a person. Model predictions, OCR, confidence scores, and heuristics cannot overwrite that ground truth.

The selected source slots contain 681 Hero crops, 709 Main Skill crops, 55 Artifact crops, and 28 Creature crops. These counts include REJECT examples because an invalid crop still comes from a known candidate slot.

What counts as REJECT?

Training, Validation, and Held-out test have different jobs

• Training updates model weights.
• Validation selects checkpoints and rejection thresholds.
• Held-out test is evaluated only after those choices are fixed.

The split boundary is the source video. Thirteen videos provide training data, one different video provides validation data, and two further videos form the held-out test. No source video crosses partitions.

The held-out set contains 113 Hero-slot crops, 257 Main-Skill-slot crops, and 28 Artifact-slot crops; it contains no Creature crops. Its 28 REJECT examples comprise 10 text overlays, 11 tooltips, 6 scene mismatches, and 1 non-Hero crop.

Among the 40 held-out identities, 24 have at least one verified gameplay positive in training, covering 279 held-out crops. The other 16 have only catalog artwork in training, covering 91 held-out crops. This split lets us check whether gameplay supervision helps only directly covered identities or also changes the shared visual representation.

Same-video temporal repetition control

Static UI screens can remain nearly unchanged for several seconds. Within the same video, source type, identity or reject reason, and candidate slot, consecutive crops are grouped when they are at most 6.5 seconds apart and their dHash distance is at most 4. Each temporal group keeps at most its first, middle, and last crop. This formed 920 groups and removed 273 adjacent repetitions.

The exporter does not perceptually deduplicate genuinely independent recordings across splits. Similar views from separate recordings are legitimate repeated observations, although they reduce the benchmark’s effective visual diversity. Cross-video merging is appropriate only when provenance shows that two files derive from the same underlying recording.

3. One visual backbone, three outputs, two training stages

The model uses an ImageNet-pretrained ConvNeXt-Tiny backbone. Its 768-dimensional pooled representation feeds a binary validity head, a four-way type head, and one identity head per type.

Figure 2. REJECT is not a fifth object type. A rejected crop never competes against Hero, Main Skill, Artifact, or Creature identities.

Training is staged rather than joint:

1. Classifier stage. VALID catalog and gameplay crops update the backbone, type head, and corresponding identity head. REJECT crops are not used for type or identity supervision.
2. Validity stage. After selecting the Gameplay-trained classifier checkpoint, the backbone, type head, and all identity heads are frozen. Verified VALID and REJECT crops then train only the binary validity head. REJECT crops never receive type or identity loss.

This separation prevents a tooltip, text overlay, or wrong scene from being forced toward a particular Hero identity.

4. What stays fixed, and how every choice is made

Catalog-only versus Gameplay-trained

The two classifiers start from the same in-memory ConvNeXt-Tiny initialization. They use the same taxonomy, optimizer, learning-rate schedule, batch size, augmentation code, checkpoint rule, 12 epochs, 356 optimizer updates per epoch, and exactly the same identity sequence.

The identity schedule repeatedly shuffles complete cycles of the 354 identities, so each identity receives equal or near-equal opportunities within the fixed update budget. The only treatment change is the source image drawn after an identity is selected:

• Catalog-only always draws from that identity’s catalog pool.
• Gameplay-trained targets a 50/50 catalog/gameplay mix for identities that have verified gameplay positives; identities without gameplay positives continue to draw catalog artwork.

This identity-first design prevents an identity with hundreds of crops from receiving more training opportunities merely because its image pool is larger.

Checkpoint selection

Each classifier checkpoint is scored on Validation only. The rule compares, in order:

1. joint routed identity accuracy;
2. macro identity joint accuracy;
3. type accuracy.

The comparison is lexicographic: the first unequal metric decides. If every metric is exactly tied, the earliest epoch remains selected. Held-out results never participate. Under this rule, Catalog-only selected epoch 5 and Gameplay-trained selected epoch 1.

The validity stage uses a separate Validation-only checkpoint rule: balanced accuracy first, then VALID recall, then AUROC; an exact tie again keeps the earliest epoch. It selected epoch 1.

The confidence baseline and learned-validity score

The confidence baseline multiplies two routed probabilities:

classification confidence = max P(type | crop) × max P(identity | crop, predicted type)

The first term is the largest softmax probability from the type head. The model then routes the crop to that predicted type’s identity head; the second term is the largest softmax probability inside that head. A crop is accepted when the product is at least the selected threshold.

The learned-validity gate instead uses the validity head’s sigmoid probability of VALID. For both scores, higher means “more likely VALID.” Each threshold is selected on Validation by maximum balanced accuracy; ties prefer higher VALID recall and then the higher threshold.

AUROC and AUPRC use VALID as the positive class and evaluate score ranking. End-to-end accuracy is stricter: a REJECT crop must be rejected, while a VALID crop must be accepted with both its type and identity correct.

5. The classification gain is entirely in Artifact

On 370 held-out VALID crops, Catalog-only classifies 353 correctly and Gameplay-trained classifies all 370. The paired comparison contains 17 improvements and 0 regressions.

Hero and Main Skill were already perfect under the Catalog-only baseline. All 17 corrected predictions are Artifact crops from one held-out recording.

Catalog-only routes 10 of those Artifact crops to the Creature head and 7 to the Main Skill head. Real Artifact crops include card backgrounds, borders, scaling, and UI effects that are absent from transparent catalog artwork. Gameplay supervision corrects both the type and identity for every case in this held-out set.

Figure 3. The classification improvement is concentrated in Artifact. The rejection improvement comes from accepting more valid crops at the Validation-selected operating point.

Three representative corrected Artifact crops are shown below:

Misericorde	Truthseeker	Third Eye

The gain is not restricted to identities that directly receive gameplay examples:

The 16 catalog-only identities also improve from 81/91 to 91/91. The benefit therefore extends beyond identities that directly receive gameplay examples. This is consistent with transfer through the shared representation and type classifier, although this experiment does not isolate that mechanism.

6. Learned validity improves the operating threshold, not the ranking

Both scores perfectly separate VALID from REJECT on this held-out set: AUROC and AUPRC are 1.0 for both. Learned validity therefore does not improve ranking on this benchmark.

The difference appears after applying thresholds selected on Validation. Both methods retain 100% REJECT recall (28/28), but learned validity raises VALID recall from 79.5% (294/370) to 95.7% (354/370), reducing false rejects from 76 to 16.

This is a threshold-transfer result, not evidence of better score ranking or measured probability calibration. The learned score’s Validation-selected threshold transfers to the two held-out recordings with many fewer false rejects.

The 16 remaining false rejects comprise 14 Artifact crops, 1 Hero crop, and 1 Main Skill crop:

Artifact: Wooden Ring	Hero: Unfrozen Hero 16	Main Skill: Thaumaturgy

The pattern matches the training imbalance: training contains only 27 VALID Artifact crops, while every REJECT example comes from a Hero slot. Learned validity performs much better than confidence at the selected operating point, but its slot and type coverage remains narrow.

7. What this experiment does—and does not—show

The evidence supports these narrow statements:

• verified gameplay supervision improves classification on this frozen one-seed benchmark;
• the measured gain is entirely in Artifact, while Hero and Main Skill remain unchanged at 100%;
• the benefit extends to held-out identities with no gameplay positives in training;
• learned validity preserves 28/28 REJECT recall while reducing VALID false rejects from 76 to 16 at Validation-selected thresholds.

It does not establish that gameplay supervision will always help, that the observed transfer mechanism is proven, or that rejection works across all slot types. The main limitations are:

• one fixed training seed;
• two held-out source videos with temporal correlation among nearby crops;
• no Creature crops in held-out;
• no reviewed transition or empty crops;
• every REJECT crop, including all 28 held-out REJECT examples, comes from a Hero candidate slot;
• uneven reject-reason and type coverage;
• fixed-crop classification only, not localization or battlefield detection.

The most useful conclusion is not the headline 100%. It is the failure pattern: clean catalog artwork already handles Hero and Main Skill, while real UI context strongly affects Artifact routing; a dedicated validity head moves the acceptance threshold in the right direction, but valid Artifact crops remain its dominant false-reject case.

This is a public record of one fixed-seed development experiment. The experimental model has not been integrated into production inference.