Full-Image Icon Search with Sliding Windows and Visual Embeddings

I wanted to understand a deceptively simple perception problem: can a generic visual embedding find hero portraits or skill icons directly from a complete game screenshot, without being told where those icons are?

This post documents the resulting experiment, including the parts that failed. It is best read as a qualitative case study, not a benchmark. The implementation is real and reproducible, but the evidence comes from only seven temporally correlated frames and has no independent ground-truth annotations.

TL;DR: ConvNeXt-Tiny retrieval can attach a plausible identity to a crop that contains substantial icon content, but it does not localize icons reliably in a full screenshot. The forced-top-three search returns shifted, partial, duplicate, and non-target regions, while the 384 × 384 encoder input is 9.36× slower than the earlier 128 × 128 run. The next experiment should separate proposal quality, identity retrieval on ground-truth crops, and end-to-end detection.

Article map

• Task and evidence boundary
• Candidate-window geometry
• ConvNeXt descriptor construction
• Exact retrieval and region selection
• Qualitative results
• Factorized evaluation
• Reproducibility snapshot
• Next experiments

Figure 1. The full pipeline. A plausible nearest identity can still come from a partial or misaligned crop.

The task

At inference time, the input is one complete 1920 × 1078 screenshot. The search is configured for one of two tasks:

Task	Visible targets	Identity space	Reference images
Hero selection	3 portraits	77 heroes	77 portraits
Main-skill selection	3 icons	20 skills	60 level-specific icons

For every output region, the system returns a bounding box and the three nearest labeled identities.

The experiment displays three hero frames and four skill frames. These are not seven independent test cases. They came from one recording: the hero frames contain essentially the same three identities, and three skill frames are near-duplicates. Repeated behavior across adjacent frames tells us about temporal stability, not generalization across matches, identities, layouts, or recording conditions.

Experiment contract

Study property	What was actually used
Source material	One recording; seven displayed frames
Independent sessions	One
Ground-truth boxes or labels	None
Output rule	Exactly three regions per screenshot, with no reject option
Quantitative comparison	Runtime only; 128 × 128 versus 384 × 384 encoder input
Defensible interpretation	Qualitative mechanism and failure analysis, not accuracy or generalization

What the search knows

The experiment deliberately excludes a positional prior. It does not know the icon row, column, slot, or region of interest. It does not use OCR, screen titles, border rules, color heuristics, or an objectness model.

However, “no positional prior” does not mean “no prior.” The search still knows:

• whether it is searching for heroes or main skills;
• a task-specific list of square candidate sizes;
• a fixed stride;
• that exactly three regions must be returned;
• a class-agnostic NMS separation threshold;
• the task-specific labeled reference vocabulary.

That distinction matters. Calling the system “prior-free” or saying box size is its “only spatial prior” would understate how much structure the experiment already assumes.

Candidate geometry

Hero candidates use square source boxes of 128, 144, 160, 176, and 192 pixels with a 20-pixel stride. Main-skill candidates use 72, 84, 96, 108, 120, and 132 pixels with a 16-pixel stride.

Coordinates are integer, half-open boxes [x1, y1, x2, y2), so width is x2 - x1. Regular stride positions are enumerated first. If the stride misses the far edge, one final boundary-aligned position is appended. The configured set is exhaustive, although the last step on an axis may not be uniform.

Task	Box	Horizontal × vertical positions	Windows
Hero	128 × 128	91 × 49	4,459
Hero	144 × 144	90 × 48	4,320
Hero	160 × 160	89 × 47	4,183
Hero	176 × 176	89 × 47	4,183
Hero	192 × 192	88 × 46	4,048
Main skill	72 × 72	117 × 64	7,488
Main skill	84 × 84	116 × 64	7,424
Main skill	96 × 96	115 × 63	7,245
Main skill	108 × 108	115 × 62	7,130
Main skill	120 × 120	114 × 61	6,954
Main skill	132 × 132	113 × 61	6,893

Figure 2. Candidate sizes at their source-frame proportions. The displayed positions are illustrative; every size scans the complete frame.

Before scoring anything, an annotated benchmark should measure the best IoU attainable by this grid for every true icon. That oracle proposal recall would tell us whether square shape, scale range, or stride already imposes a low localization ceiling.

Turning a crop into a visual vector

The encoder is torchvision.models.convnext_tiny with ConvNeXt_Tiny_Weights.IMAGENET1K_V1.

The implementation uses:

• Python 3.12.13;
• PyTorch 2.13.0+cu130;
• torchvision 0.28.0+cu130;
• CUDA 13.0 and cuDNN 9.2;
• an NVIDIA GeForce RTX 5080;
• model.eval() and torch.inference_mode();
• FP16 CUDA autocast for convolution and matrix multiplication;
• float32 similarity reduction.

Query pixels are converted to three-channel RGB float values in [0, 1], then normalized with ImageNet mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225).

Candidate boxes are resampled directly from the float RGB frame with torchvision.ops.roi_align:

output_size = (384, 384)
spatial_scale = 1.0
sampling_ratio = 2
aligned = true

The descriptor path is:

source candidate crop
  → RoI Align to 384 × 384 RGB
  → ConvNeXt feature stages: 96 → 192 → 384 → 768 channels
  → final feature map: 768 × 12 × 12
  → adaptive global average pooling
  → classifier[0] LayerNorm
  → flatten to 768 dimensions
  → L2 normalization

The 1,000 ImageNet logits are not used. The resulting vector is the normalized 768-dimensional pre-logit descriptor.

Reference artwork follows a slightly different resampling path: PIL bicubic resize after alpha compositing. This query/reference operator difference is a possible source of domain mismatch and was not ablated.

What did 384 × 384 actually establish?

An earlier configuration used 128 × 128 encoder inputs and produced a 768 × 4 × 4 final feature map. The displayed configuration uses 384 × 384 and produces 768 × 12 × 12 before pooling.

Configuration	Batch size	Seven-frame runtime
128 × 128	256	28.33 s
384 × 384	64	265.25 s

The 384 configuration was 9.36× slower in these two complete runs. That is the only defensible quantitative conclusion.

This is not a clean resolution ablation because batch size also changed, the canonical 224 × 224 condition is missing, timing was not repeated, and no paired identity or localization annotations exist. Larger absolute cosine scores would not prove better quality anyway; changing input preprocessing can shift the score distribution.

Building the reference vectors

Hero identities have one portrait each. Every main skill has Basic, Advanced, and Expert artwork, giving 60 visual references for 20 skill identities.

Some source artwork contains transparency, while this pretrained ConvNeXt expects three-channel RGB input. Each reference image is:

1. cropped to pixels with alpha greater than 8;
2. padded by 8% of its longest side;
3. composited onto three fixed RGB backgrounds;
4. bicubic-resized and ImageNet-normalized;
5. encoded into three individually L2-normalized vectors;
6. averaged and L2-normalized again.

r(entity, visual variant)
    = normalize(mean(r_dark_blue, r_dark_gray, r_light_gray))

Figure 3. The implemented three-background heuristic. This experiment does not establish that the heuristic improves invariance or accuracy.

Three fixed backgrounds may reduce dependence on one arbitrary fill color, but they may also introduce their own bias. Useful controls would include no compositing augmentation, separate background prototypes, random in-game backgrounds, alpha-aware features, and references cropped from the actual renderer.

Exact cosine retrieval

Let Q contain N normalized candidate descriptors and R contain M normalized reference descriptors. Each row has 768 dimensions.

S = Q Rᵀ
cosine(qᵢ, rⱼ) = qᵢᵀ rⱼ

The dense matrix multiplication compares every candidate with every reference. With only 60 or 77 references, this exhaustive search is simpler than an approximate index such as HNSW or IVF. “Exact” here means no nearest-neighbor approximation; it does not mean exact real-number arithmetic because GPU inference uses FP16 autocast.

The selection algorithm

The reduction from millions of similarities to three displayed regions is where much of the behavior originates:

for every window i:
    per_identity_score[i, c]
        = max similarity over visual references belonging to identity c

    top3_identity_candidates[i]
        = three highest distinct identities

    detection_score[i]
        = top3_identity_candidates[i][0].score

stable-sort every window by detection_score descending

for each window in that order:
    keep it if IoU(window, every kept box) <= 0.10
    stop after exactly three kept regions

Skill levels are collapsed before region selection. NMS is class-agnostic and operates on each window’s top identity score. Identities are distinct within one window’s candidate list, but they are not deduplicated across output regions. Equal scores retain original enumeration order.

The NMS suppression threshold is 0.10. Standard NMS suppresses a lower-scoring proposal only when it overlaps an already retained box by more than that threshold. It does not guarantee one proposal per semantic object.

For example, two displayed skill boxes were approximately:

[672, 464, 756, 548)
[624, 368, 744, 488)

Their IoU is about 0.088. Both survive even though both contain evidence from the same Primal Magic icon. This is NMS behaving as configured, not an implementation error.

The forced-top-three problem

The system has no background class, objectness score, threshold, or reject option. It always returns three regions—even on a screen containing no valid target.

That creates an extreme-value problem. A hero frame tests 21,193 windows against 77 references. A skill frame tests 43,134 windows against 60 references. The largest similarity among millions of comparisons can look impressive simply because so many distractors were tried.

Task	Frames shown	Windows per frame	Total windows	Exact comparisons
Hero	3	21,193	63,579	4,895,583
Main skill	4	43,134	172,536	10,352,160
Total	7	—	236,115	15,247,743

Negative screenshots, unrelated reference libraries, and a calibrated variable-count reject rule are necessary before a maximum cosine can be interpreted as detection evidence. Hero and skill maxima are also not directly comparable because their numbers of windows and reference variants differ.

Representative qualitative outputs

Every panel below comes from the 384 × 384 configuration. I selected three representative panels for readability rather than repeating several adjacent near-duplicate frames. They illustrate behavior; they do not estimate accuracy.

Panel	Localization	Identity retrieval	What the panel demonstrates
Hero selection (Figure 4)	Mixed: substantial portrait content, but several boxes are shifted	Some top-ranked identities are visually plausible	Retrieval can look useful even when the crop is not a clean object box
Repeated skill frame (Figure 5)	Poor: multiple windows survive around one semantic target	The same visual evidence drives several plausible candidates	Low-IoU NMS does not enforce one region per object
Distinct skill frame (Figure 6)	Poor: partial and non-target regions remain	A nearest reference is always returned	Forced top-three output is not calibrated detection

Figure 4. A hero-selection example. Some selected regions contain substantial portrait content, but identity and localization are still scored by the same objective.

Figure 5. A repeated skill-frame failure mode: partial or shifted windows can survive NMS around one semantic target.

Figure 6. A more distinct skill frame. The scorer still forces three regions and has no way to reject non-target content.

What the examples show

The visible outputs support a modest set of observations:

• some selected crops contain substantial target content and rank a visually corresponding reference first;
• many boxes are shifted, partial, or attached to nearby UI decoration;
• multiple windows can survive around one semantic object;
• the displayed 384 × 384 run still contains obvious localization errors.

They do not establish:

• retrieval reliability across 77 heroes or 20 skills;
• a quality benefit from 384 over 128 or 224;
• a benefit from three-background averaging;
• a benefit from three skill-level references;
• that global pooling or missing objectness is the sole cause of failure.

Identity and localization are coupled

The same cosine score chooses the location and assigns its identity. Identity retrieval is therefore judged only on crops selected by the identity model itself. This makes it impossible to tell whether a failure came from candidate geometry, representation quality, reference mismatch, score aggregation, or NMS.

A fair evaluation should separate three capabilities.

Identity-only

Classify independently annotated, ground-truth crops. Report top-1 and top-3 accuracy, mean reciprocal rank, macro accuracy, and correct-class margin.

Proposal-only

Ignore identity and measure the maximum candidate IoU for every true object. Report proposal recall at IoU 0.25, 0.50, and 0.75, normalized center error, and scale error.

End-to-end

Count a detection as correct only when the identity is correct and IoU exceeds a declared threshold. Report class-aware AP, recall, false positives per image, and the proportion of screenshots where all three targets are found correctly.

The benchmark also needs an operational box definition. “Complete icon” might mean source artwork, the in-game mask, glow, circular frame, or surrounding slot. I would annotate both an inner artwork box and an outer rendered-icon/frame box.

Reproducibility snapshot

This table collects the run-defining choices that are otherwise distributed across the method sections. It is a configuration record, not a public reproduction package: the seven source frames and implementation are not distributed with this post.

Component	Displayed-run setting
Input frame	1920 × 1078 RGB
Encoder	torchvision ConvNeXt-Tiny, IMAGENET1K_V1 weights
Descriptor	768-dimensional pre-logit vector after global average pooling, LayerNorm, and L2 normalization
Encoder input	384 × 384 RoI-aligned crop
Hero proposals	128–192 px square boxes, 20 px stride, 21,193 windows per frame
Skill proposals	72–132 px square boxes, 16 px stride, 43,134 windows per frame
Reference bank	77 hero portraits or 60 level-specific images representing 20 main skills
Similarity search	Exact dense cosine matrix multiplication
Region reduction	Stable score sort, class-agnostic NMS at IoU 0.10, then exactly three outputs
Numeric path	FP16 CUDA autocast for the encoder and matrix multiplication; float32 similarity reduction
Hardware	NVIDIA GeForce RTX 5080
Software	Python 3.12.13, PyTorch 2.13.0+cu130, torchvision 0.28.0+cu130, CUDA 13.0, cuDNN 9.2

For a faithful rerun, preserve the proposal enumeration order, boundary-aligned final windows, half-open box convention, preprocessing constants, reference compositing policy, and stable tie behavior. Changing any of these can alter the selected regions even when the model weights remain fixed.

The next three experiments

1. Annotated factorization benchmark

Collect roughly 200–300 de-duplicated screenshots from at least 20 independent sessions, plus at least 50 visually similar negative screens. Split by session, not by frame. Annotate identity, skill level, inner and outer boxes, tooltip or occlusion state, and source session.

Freeze the current grid and post-processing on a development split, then measure oracle proposal coverage, oracle-crop identity retrieval, selected proposals, and end-to-end detection separately.

2. Resolution × crop-quality study

Generate controlled translations, scale errors, truncations, and IoU bins from annotated targets. Compare 128, canonical 224, and 384 with the same preprocessing, crop operator, and proposal set.

Report accuracy versus crop IoU, correct-class margin, negative separation, repeated synchronized latency, throughput, and peak VRAM. This would test whether higher encoder resolution actually improves robustness rather than merely shifting cosine values.

3. Proposal/objectness stage with strong baselines

Compare the current scorer against:

• alpha-aware multiscale NCC or edge template matching;
• oracle-grid proposals;
• a fixed ROI or slot-layout baseline;
• screenshot-derived references;
• a lightweight icon-objectness detector;
• detector proposals followed by the unchanged ConvNeXt retriever.

If detector proposals improve results while the retriever stays fixed, that would support the hypothesis that proposal generation is the main bottleneck. If a masked template matcher wins, generic ImageNet embeddings may simply be the wrong representation for this rendering domain.

Conclusion

On this small set of temporally correlated UI frames, nearest-neighbor ConvNeXt features sometimes rank a visually corresponding reference first when the selected crop contains substantial target content. The exhaustive forced-top-three scorer also produces obvious partial crops, duplicates, and non-target regions.

Because there is no independent ground-truth set or paired quality metric, the effects of input resolution, reference augmentation, grid density, and NMS cannot yet be quantified. The 384 × 384 configuration was 9.36× slower than the 128 × 128 configuration in this implementation.

The most useful next step is not another visual tweak. It is an annotated evaluation that separately measures proposal coverage, identity retrieval on oracle crops, and end-to-end detection.

That decomposition would turn this educational failure analysis into an actual perception benchmark.