Studio

Vision Lab

Teach the bot what to look for and what to read, and save it as a detector.

A detector is a saved, tuned instruction for the bot's eyes. A Template detector finds a picture on the screen: a button, an icon, a monster. A Text (OCR) detector reads text or a number from an area. Steps and library entries use a detector by its name, so improving one detector improves every task that uses it.

NeedsA running emulator to capture from, or saved 540 × 960 screenshots

Open it from Studio ▸ Vision Lab. It also opens in a window from a Task Builder step (Vision Lab or Tune / create in Vision Lab, see the Flow step reference) and from the detector buttons of Monsters, Champions, Generals and Recruit Generals entries.

Vision Lab in Template mode: the Source card with its tray of sources, and the Output card with the live result.

Set it up: a Template detector

On your emulator, open the game at the screen that shows the thing to find.
In Studio ▸ Vision Lab, click the + tile at the bottom of the Source card. Pick your emulator under Port and click Capture once.
Drag a tight box around the button or icon on the Source. The Output card boxes the match in green, and the Result line shows its score.
Add two or three more captures where it should, and should not, be found. Click See results to check for a ✓ or ✗ on every source.
Click Template to adjust the Threshold, filters or Search region until every source is right.
Click Save detector, give it a name such as event_close_x and a Category folder, and click Save.

Header and toolbar

Expand (icon beside the title)

Opens the whole lab in a large window (Expand Vision Lab); click it again to put it back (Collapse Vision Lab).

New (circular arrows)

Clears the sources, template and every setting to start a fresh detector. With work open it asks Clear the current sources, template and settings and start fresh? (Start fresh).

Detector chip

What the lab holds: new · unsaved; a detector's name when the settings match what is saved; or name · edited once you change something.

Template / Text (OCR)

The mode: Template finds a picture, Text (OCR) reads text. Default: Template.

Template

Template mode: opens the Template window with the crop and match settings (tooltip Cropped template, filters & match settings).

OCR settings

Text (OCR) mode: opens the reading settings.

Load detector

Opens a saved detector of the current mode.

Save detector

Saves the current settings as a detector. In Template mode it needs a template first: Crop a template on the source first.

Source card

Hint line

drag to crop · wheel/pinch zoom · right-drag pan

Zoom out / zoom % / Zoom in / Fit to view

Zoom the source; Fit to view resets it. The mouse wheel and a trackpad pinch zoom around the pointer.

Crop box

Drag on the image to draw the template box, drag inside it to move it, and drag an edge or corner to resize it. In Text (OCR) mode the box becomes the reading area.

Panning

When zoomed in: right-drag, Ctrl+drag, or the small arrow pad in the corner (Pan the view (or hold Ctrl and drag)).

Arrows and counter

With several sources, arrows on the image edges step through them and a chip shows the position, such as 2 / 5. The Output card follows along.

Source tray

A tile per source along the bottom: click a tile to show it, × to remove it. The + tile (Add a new source image) opens Add source.

Minimize / Show sources

The arrow at the right of the tray hides or shows the tiles. While hidden, the bar offers View sources · N.

Collapse / Expand

The chevron in a card's title folds it to a thin strip to make room. Cards open again by themselves when the window is wide enough.

Add source dialog

Upload image(s)

Pick one or more saved screenshots; each becomes a source.

Port

The emulator to capture from. Pick a running one from the list, or type its port (for example 5555) or a full address such as 127.0.0.1:5555. The last one used is filled in. Anything else shows Invalid port.

Capture once

Takes one screenshot from that emulator.

Burst · shots · ms

How many screenshots (2–100, default 5) and the gap between them (50–10,000 ms, default 500 ms).

Capture burst

Takes that many screenshots over time, for a moving or animated screen. The Result line counts them in, then reads Burst done — N frames added.

Cancel

Closes without adding.

The Add source dialog: Upload image(s), Port, Capture once, and Burst · shots · ms.

Output card

Result line

The live verdict on the source being shown. With Find all matches on (the default), it counts the matches: Result: 2 matches · best 0.93 @ (270, 812), or Result: below threshold · 0 matches. With it off: Result: match · conf 0.93 @ (270, 812) or Result: below threshold · conf 0.61 …, adding N more like this — enable Find all matches when more copies are visible.

Expand output

Shows the output large, in Output — expanded.

Output view

What the bot sees after your filters, each match boxed in green and the search region dashed in amber. Before a crop: Output preview. Crop a template to see the match.

See results / Hide results

Template mode: every source with ✓ or ✗, the number of matches when more than one, and the best score, under a summary such as 4/5 matched.

Read text now

Text (OCR) mode: reads every source with the current settings and lists the results on the Text output card; the line under the output shows the text of the source being shown, as Read: “…”.

Template window

A window you can keep open while you work; the Output updates as you change settings. Close hides it.

Cropped — drag to move

The template. Drag it to nudge the crop on the source. Before a crop: crop on the source.

Load from file

Uses a saved picture as the template instead of a crop. A PNG's transparent pixels can be ignored when matching.

Save to file

Saves the template as a picture file; a PNG keeps its transparency. Available once there is a template.

Filtered (what matches)

The template after your filters.

Match method

How a match is scored: TM_CCOEFF_NORMED (default), TM_CCORR_NORMED or TM_SQDIFF_NORMED. The default suits most detectors. With a transparency mask in use a note explains that the method is swapped and its scores read differently.

Threshold

How close a match must be, 0.50–1.00, shown as Threshold · 0.85. Default 0.85. Raise it if wrong things match; lower it if the right thing is missed.

Colour mode

Grayscale (default) or Color. Use Color when two things differ only by color.

Preprocessing filters

Switches applied to both the screen and the template: Invert (flip light and dark), Auto contrast (lift faint detail), Sharpen, Denoise (blur) with a strength (3–15, default 3), Edge detect with low and high levels (defaults 50 and 150; matches on outlines only) and Binarize (threshold). All off by default.

Binarize level

The cut-off for Binarize (threshold), 1–255, default 160.

Search region (blank = whole screen)

x, y, w and h of the only area to search: faster, with fewer false hits. Pick region on source lets you draw it in Select focus region and confirm with Use region; Clear searches the whole screen again.

Scale range (min / max / step)

The sizes to try the template at. Default 1.0 / 1.0 / 0.05, which is one size. Widen it when the same thing appears at different sizes.

Find best scale

Tries the range, puts the best size in the best box and switches Lock to best on. Editing the range switches the lock off again.

Lock to best

On: the preview and the saved detector use only the best size. Off: the detector tries the whole range when it runs.

Find all matches

Preview only: boxes every copy above the threshold, up to the number beside it (1–50, default 10). Steps that use the detector still take the single best match. Default: on for a new detector; a loaded detector brings back the setting it was saved with.

Use transparency mask

Available when the template came from a PNG with transparency: ignore its transparent pixels. Default: off.

Text (OCR) mode

Switch the toolbar to Text (OCR). A Text output card joins Source and Output, and OCR settings takes the place of Template. Draw a box on the Source (or type a Focus region), then click Read text now. Reading uses the text recognition built into Windows, so there is nothing to install. The OCR settings window:

Focus region

x, y, w and h of the area to read, or Pick region on source. Blank reads the whole screen.

Filter pipeline

Steps applied top to bottom before reading, each with a switch and, where shown, a value: Grayscale (on), Invert, Auto contrast, Sharpen, Denoise, Threshold (on, level 160), Otsu threshold (picks the level itself), Adaptive threshold (block size 31, for uneven backgrounds), Upscale (on, ×2), Morph close (2), Morph open (2), Dilate (2), Erode (2) and Pad border (10). Hover a step for what it does.

Text engine

A note on the built-in reader, then two boxes: Allowed characters — e.g. 0123456789,.KM keeps only those characters, and Banned characters — e.g. |~^ drops those. Blank boxes keep everything.

Parse

How the text becomes a value; see the table below. Default: Text.

Text

Keep the text as read.

Number

Keep the digits only.

Number (K/M/B)

Take the first number and expand game suffixes: 1.2M becomes 1200000.

Fraction

The number before the slash: 11/160 becomes 11. Blank if there is no slash or that part is not all digits.

Timer (minutes)

Read an HH:MM:SS countdown as whole minutes.

Text output card

One row per source with the text read (you can select and copy it), the source being shown highlighted. Before a read: No text read yet.

Load detector dialog

Titled Load template detector or Load OCR detector: it lists only detectors of the current mode.

Save detector dialog

Name

Placeholder e.g. shield_button. Filled with the loaded detector's name, so saving again updates it. An empty name shows Give the detector a name.

Category folder

Template mode only: pick a folder or type a new one. The line underneath shows where the picture is stored, and whether that folder exists or will be created.

Cancel / Save

Save stores the detector. A name that belongs to a different detector asks Overwrite? first.

Opened from a library entry or a step

Use for this monster / Use for this skin / Use for this general

From a Monsters, Champions, Generals or Recruit Generals entry: saves the settings as a detector (asking for a name when it is new or edited) and links it to that entry. In this window Text (OCR), Expand and New are hidden.

Use this detector

From a Task Builder step: returns the detector on screen to the step, saving it first when it is new or edited. Cancel changes nothing.

How it runs

A step that uses a detector looks for it with the saved threshold, color mode, filters, search region and size. Your source images are only for tuning.
Search regions and focus regions are positions on a 540 × 960 screen, so capture your sources from an emulator set to that size.
A Find template step can override a detector's threshold or scale for that one step.
Renaming a detector updates the library entries linked to it; deleting one leaves steps that used it with nothing to look for.

For game numbers, a reliable recipe is: Invert when the text is light on dark, Otsu threshold on with Threshold off, Upscale, digits only under Allowed characters, and Parse Number. Tune it against a Capture burst rather than a single screenshot.

Examples

A close button that moves a little between events. Capture a burst across three events, crop the X, set a Search region over the top-right corner, raise Threshold until only the X matches on every source, and save it as event_close_x in buttons.
Read the gold amount. Switch to Text (OCR), draw the box over the amount, Allowed characters 0123456789.KMB, Parse Number (K/M/B), Read text now on several captures, and save it as gold_amount.
A monster seen at several zoom levels. Scale range 0.6 / 1.2 / 0.05, Find best scale to check it matches, then switch Lock to best off so the detector tries the range when it runs.

Troubleshooting

The Output shows no match, though the thing is clearly there.

Lower the Threshold a little, check the Search region is not excluding it, and make sure the source is a 540 × 960 capture. If the thing appears at a different size, widen the Scale range and use Find best scale.

Other things match too.

Raise the Threshold, crop a more distinctive part, add a Search region, or set Colour mode to Color when the look-alike differs in color.

Capture once fails.

Check the emulator is running and pick it from the Port list rather than typing it. See Emulator setup.

Save detector says to crop a template first.

Draw a box on the Source, or use Load from file in the Template window, then save.

The detector finds things in the lab but not when a task runs.

Find all matches is preview only; a step takes the single best match. With Lock to best off, the detector tries its whole scale range, and a Search region captured from a different screen size will be in the wrong place.

Read text now gives an empty or garbled result.

Draw the focus region tighter, try Upscale at 3, switch between Threshold and Otsu threshold, and add Allowed characters. Watch the Output view as you change each filter.

The chip says edited.

A setting changed since the detector was loaded or saved. Save detector with the same name to update it, or Load detector to go back to the saved version.