Why Five Game Icons Were Missed—and What Improved Them

The experiment in one minute

The input is a full game screenshot; the output is each requested icon's location and catalog ID. The catalog contains 354 semantic IDs represented by 390 image files; some IDs have several visual variants. A fixed multi-scale sliding window generates every candidate box—the neural network never proposes boxes. H1 asks “who is in this crop?” and H2 asks “is this crop well aligned and complete?” The program fuses both answers, then applies NMS, a threshold, and a top-three limit.

What does this follow-up add to the previous experiment?

The previous article trained the pipeline from generic ImageNet features through H2 and ended with 13 TP / 3 FP / 5 FN on 18 real-screen targets. This follow-up does not ask again whether icon labels help. It traces each of those five misses backward through the pipeline: did the fixed grid fail to cover the target, did H1 choose the wrong ID, did H2 give a good box too little credit, did NMS delete it, or was the final cutoff too strict?

How does the system actually work?

S_crop is not itself an IoU prediction. H2 separately predicts IoU_hat (the hat means estimated), p_full (is the full icon inside?), and p_object (is an object present?), then takes their geometric mean. H2-rank only reranks the original boxes. H2-offset adjusts output coordinates; it does not recrop or run a second forward pass.

First question: can the grid produce a good box?

For every ground-truth object, we ignore all model scores and take the grid box with maximum IoU. Both tasks have 100% oracle recall at IoU 0.70, so finer stride or denser sizes cannot explain the five old misses.

Where do the five misses occur?

Of the old H2 detector's five hero false negatives, four already had a correct-ID proposal at IoU≥0.50 that survived NMS and remained in the top three; the one global threshold rejected them. The fifth was an identity error.

What improvements did we actually try?

A1: separate thresholds by task

Hero and skill scores have different distributions, and the task is known before inference. A1 keeps the old model and α=2 but lets validation choose a hero threshold and a skill threshold. This is the cheapest change.

A2: separate fusion strength by task

The final score is S_identity × S_crop^α. A larger α lets crop quality punish a candidate more strongly. A2 selects hero α, skill α, and their thresholds separately on real-screen validation.

A3: NMS sweep

We compare class-agnostic NMS with same-ID NMS, which suppresses overlaps only when both boxes predict the same identity, across several IoU thresholds. Validation keeps the original class-agnostic rule at 0.10; NMS is not the bottleneck.

A4: type-conditioned output layers

The old 1.81M-parameter trunk remains shared. Only the final predicted-IoU, completeness, objectness, and box-offset layers branch into hero, skill, creature, and artifact outputs—about two thousand additional parameters. Training still uses synthetic icon crops, not gameplay pixels.

A5: preserve the 7×7 spatial feature map

Old H2 average-pools ConvNeXt's 768×7×7 map to 768×3×3. A5 keeps the full layout: a 1×1 convolution reduces 768 channels to 64, a 3×3 convolution produces 32×7×7, and a 96-D MLP feeds the four outputs. It has only 222,119 trainable parameters while preserving finer relative position.

All results

TP requires the correct global ID and greedy one-to-one IoU≥0.50. “Negative accept” is the fraction of six negative task-screens that emit at least one detection.

How should we interpret the numbers?

• A1 buys recall: development-test F1 rises from 0.765 to 0.842, but negative-screen acceptance also rises from 0.500 to 0.667.
• A2 is more balanced: F1=0.824, with negative acceptance down to 0.333.
• A4 wins validation: its validation F1 is 0.833, so it is the strict validation-selected winner, but that lead does not transfer to the development test.
• A5 is the strongest next candidate: it recovers all four threshold misses and leaves only the H1 identity miss.

What should a stable recognizer do next?

Freeze A5 and its validation recipe, collect new recording groups, and hide their labels until one sealed evaluation. If the duplicate-hero identity error persists, start a separate experiment that is allowed to train on real hero-selection crops. Postpone multi-scale features and iterative recropping: grid oracle recall is already 100%, and offsets have not yet shown a stable IoU gain.

Glossary

Proposal / candidate

A box produced by the deterministic sliding-window grid.

IoU

Intersection over Union: overlap divided by combined area.

NMS

Non-maximum suppression removes lower-scoring overlapping duplicates.

Oracle proposal recall

The fraction of objects for which the fixed grid contains a sufficiently overlapping box, ignoring all model scores.

一分钟理解这次实验

输入是一张完整游戏截图，输出是目标图标的位置和 catalog ID。Catalog 有 354 个语义 ID，由 390 张参考图片表示；一些 ID 有多个视觉版本，所以图片数比 ID 数多。固定的多尺度 sliding window 先产生所有候选框，神经网络不会自己生成框。H1 回答“这个 crop 是谁”，H2 回答“这个框裁得好吗”。最后程序融合两个分数，再经过 NMS、阈值和 top-3 得到检测结果。

这篇 follow-up 接着上一篇解决什么？

上一篇从通用 ImageNet feature 一路训练到 H2，最后在 18 个真实目标上得到 13 TP / 3 FP / 5 FN。这篇不重新讨论“图标标签有没有用”，而是把那 5 个漏检逐个沿流水线往回追：是固定 grid 没有覆盖目标、H1 认错 ID、H2 把好框打成低分、NMS 误删，还是最终接受线过严？

系统到底怎样工作？

S_crop 本身不是 IoU prediction。H2 分别预测 IoU_hat（hat 表示估计值）、p_full（完整图标是否在框里）和 p_object（框内是否真的有目标），再取三者几何平均。H2-rank 只用这个分数重排原始框；H2-offset 只调整最终坐标，不会重新 crop，也不会第二次 forward。

先问：是不是 grid 根本找不到好框？

对每个 ground truth，我们忽略所有模型分数，只取 grid 中 IoU 最大的框。英雄和技能在 IoU 0.70 都是 100% oracle recall，因此更小 stride 或更密 size 不能解释旧版的 5 个漏检。

5 个漏检到底在哪里发生？

旧 H2 的 5 个英雄 FN 中，4 个已有正确 ID、IoU≥0.50 的候选框，而且通过了 NMS 与 top-3；它们只是低于统一阈值。最后 1 个是身份错误。

这次实际尝试了哪些改进？

A1：按任务分阈值

英雄与技能的分数分布不同，而 screen task 在推理前已经知道。A1 保留旧模型和 α=2，只让 hero 与 skill 使用各自从 real-screen validation 选出的 threshold。这是最便宜的改进。

A2：按任务分开融合强度

最终分数是 S_identity × S_crop^α。α 越大，crop-quality 对排名的惩罚越强。A2 分别选择 hero α、skill α 与各自阈值，避免用同一校准规则处理两种几何外观。

A3：NMS sweep

我们比较 class-agnostic NMS 与只压制同 ID 重复框的 same-ID NMS，并搜索多个 IoU 阈值。Validation 仍选择原来的 class-agnostic、IoU=0.10；所以 NMS 不是当前瓶颈。

A4：分类型输出头

旧 H2 的 1.81M 参数 trunk 继续共享，只把 predicted IoU、completeness、objectness 与 box offset 的最后输出层分成 hero / skill / creature / artifact 四组，新增参数约两千。它在 synthetic crop 上训练，不读取 gameplay pixels。

A5：保留 7×7 空间特征

旧 H2 把 ConvNeXt 的 768×7×7 feature map 平均池化成 768×3×3。A5 不做这一步：先用 1×1 conv 将 768 channel 压到 64，再用 3×3 conv 得到 32×7×7，最后接 96-D MLP 与四个输出。它只有 222,119 个可训练参数，却保留更细的相对位置。

全部结果

TP 必须同时满足正确 global ID 与 greedy one-to-one IoU≥0.50。“负屏接受率”表示 6 个 negative task-screen 中至少输出一个 detection 的比例。

怎样解读这些数字？

• A1 追求 recall：development-test F1 从 0.765 提到 0.842，但负屏接受率也从 0.500 升到 0.667。
• A2 更均衡：F1=0.824，负屏接受率降到 0.333。
• A4 的 validation 最好：validation F1=0.833，所以严格按 validation 选 winner 时它胜出；但 development test 没有同步领先。
• A5 最值得继续：它把 4 个 threshold miss 全部救回，只剩那个 H1 identity miss。

下一步怎样得到稳定识别器？

先冻结 A5 及其 validation recipe，再收集新的 recording groups，任何人都不先看标签；只跑一次 sealed test。若身份错误仍集中在重复英雄，再开一个使用真实 hero-selection crop 的独立实验。Multi-scale 和 iterative recrop 暂缓：grid oracle 已经是 100%，offset 也尚未证明稳定提高 IoU。

术语表

Proposal / candidate

固定 sliding-window grid 产生的候选框。

IoU

预测框与真实框的交集面积，除以两者并集面积。

NMS

Non-maximum suppression，删除重叠且分数较低的重复框。

Oracle proposal recall

忽略模型分数，只问固定 grid 是否包含足够重叠候选框时的召回率。