Engineering notes · Python / OpenCV
How it works
Explore how Classic MapleStory Bot turns screen images into strategy decisions through 10 focused code excerpts and visual algorithm diagrams.
Pixels → observations → decisions → input
The client reads visible screen images. Perception estimates positions and status, a planner selects an action, and the runtime schedules local keyboard output. These layers are separate: recognizing a monster does not itself press a key, and issuing a key does not prove that the game accepted it.
- 01Capture
- 02Perception
- 03Decision
- 04Local input
01Capture the game client#
The Windows branch of grab_client first locates the client area, then captures its rectangle with ImageGrab and converts it to RGB. Invalid window or image dimensions raise an error instead of feeding an empty image into recognition.
window = find_client_origin(title)
if window is None:
raise RuntimeError(active_translate("error_game_window_not_found", title=title))
left, top, width, height = window
if width <= 1 or height <= 1:
raise RuntimeError(active_translate("error_game_capture_unavailable"))
image = ImageGrab.grab((left, top, left + width, top + height)).convert("RGB")
if image.width <= 1 or image.height <= 1:
raise RuntimeError(active_translate("error_empty_game_capture_retry"))
return imagegrab_clientL354–363This excerpt is one platform branch. macOS uses a separate native capture path and client-area cropping. Recognition code later works with OpenCV BGR arrays: keep channel order explicit when passing images between Pillow and OpenCV.
02Restrict the search with an ROI#
The minimap detector begins in a bounded upper-left region. It converts that region to grayscale, extracts edges with Canny and obtains candidate contours. Marker centers provide another constraint on which rectangle can be the map.
Work on a relevant region
Game frame
Minimap crop
if frame_bgr.size == 0 or frame_bgr.ndim != 3:
return previous or generic_minimap_definition()
search = frame_bgr[:min(frame_bgr.shape[0], 420), :min(frame_bgr.shape[1], 430)]
edges = cv2.Canny(cv2.cvtColor(search, cv2.COLOR_BGR2GRAY), 45, 135)
contours, _ = cv2.findContours(edges, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
marker_centers = [
(x + width / 2.0, y + height / 2.0)
for x, y, width, height, _ in _marker_components(search, player_pixels)
]
if not marker_centers:
return previous or generic_minimap_definition()detect_minimap_definitionL95–106This is the opening of the detector, not its entire rectangle-ranking algorithm. Invalid frames or missing markers fall back to the previous definition or a temporary default; that fallback should not be confused with a newly verified detection.
03Locate a character with grayscale correlation#
match_player_feature converts both the frame and the small character template to grayscale, then applies TM_CCORR_NORMED. The optional mask can exclude transparent template pixels from the comparison.
A template slides over the image
Template · T
Score map · S(x, y)
def match_player_feature(image_bgr: np.ndarray, template_bgr: np.ndarray,
mask: np.ndarray | None) -> np.ndarray:
"""Match normal player feature templates with tolerant grayscale correlation."""
screen_gray = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2GRAY)
template_gray = cv2.cvtColor(template_bgr, cv2.COLOR_BGR2GRAY)
return cv2.matchTemplate(
screen_gray,
template_gray,
cv2.TM_CCORR_NORMED,
mask=mask,
)match_player_featureL100–110The caller sanitizes non-finite scores and selects the strongest match before applying its threshold. In this snapshot, the feature threshold is 0.995; it is an implementation setting, not a universal accuracy percentage. Capture a stable character feature in both facing directions.
04Turn template scores into targets#
This is the opaque-template branch of detect_template. It rejects templates larger than the search image, computes a score map and collects locations at or above the configured threshold.
if image_bgr.shape[0] < template.shape[0] or image_bgr.shape[1] < template.shape[1]:
return []
scores = cv2.matchTemplate(image_bgr, template, cv2.TM_CCOEFF_NORMED)
ys, xs = np.where(scores >= threshold)
targets = [Target(int(x + template.shape[1] / 2),
int(y) + top_to_feet if top_to_feet is not None else feature_center_to_feet(int(y), template.shape[0]),
float(scores[y, x]))
for y, x in zip(ys, xs)]
return _non_maximum(targets, 15)detect_templateL217–225A location starts at the template’s top-left corner. The code converts it to a horizontal center and a calibrated foot position, then removes duplicates. Transparent templates use a separate masked correlation branch, so the two branches do not share exactly the same matching method.
05Merge repeated detections#
Several neighboring positions or pose templates may match the same monster. _non_maximum sorts candidates by confidence and retains a candidate only if it is far enough from every already-kept result.
def _non_maximum(points: list[Target], min_distance: int) -> list[Target]:
kept: list[Target] = []
for point in sorted(points, key=lambda item: item.confidence, reverse=True):
if all(abs(point.center_x - other.center_x) > min_distance or abs(point.feet_y - other.feet_y) > min_distance for other in kept):
kept.append(point)
return kept_non_maximumL194–199This implementation uses x/y distance, not bounding-box IoU. A higher-confidence nearby candidate wins. Too large a distance can merge two real neighbors; too small a distance leaves duplicates and can distort target counts.
06Segment the minimap marker in HSV#
The marker finder tries an exact BGR match first. When that produces no valid component, this fallback converts the image to HSV and uses the selected server profile’s color range, with extra saturation tolerance.
Color segmentation, then geometry
RGB → HSV → Binary mask
Component → Local position (x, y)
hsv = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2HSV)
hue_low, saturation_low, value_low = player_pixels.activity_hsv_lower
lower = np.asarray(
(hue_low, max(0, saturation_low - 40), value_low), dtype=np.uint8,
)
upper = np.asarray(player_pixels.activity_hsv_upper, dtype=np.uint8)
tolerant_mask = cv2.inRange(hsv, lower, upper)
return matching_components(tolerant_mask, exact_area=False)_marker_componentsL79–86matching_components then filters connected components by expected width, height and area. The fallback tolerates color-management differences without accepting arbitrary yellow shapes. Hue ranges and component geometry belong to the server profile, not one hard-coded color for every server.
07Convert a marker into map-local coordinates#
The image slice uses the detected minimap bounds. A server-specific foot offset maps each marker component to its standing point; the function returns a position only when exactly one candidate remains.
right = min(frame_bgr.shape[1], minimap.end_x + 1)
bottom = min(frame_bgr.shape[0], minimap.end_y + 1)
if minimap.start_x >= right or minimap.start_y >= bottom:
return None
image = frame_bgr[minimap.start_y:bottom, minimap.start_x:right]
candidates: list[tuple[int, int]] = []
for x, y, _, _, _ in _marker_components(image, player_pixels):
# Each server profile maps the matched component's origin to the
# marker's standing point with its own vertical foot offset.
candidates.append((x, y + player_pixels.foot_offset))
if len(candidates) != 1:
return None
x, foot_y = candidates[0]
source_width = max(1, right - minimap.start_x)
source_height = max(1, bottom - minimap.start_y)
source_x = min(source_width - 1, x)
source_y = min(source_height - 1, foot_y)
return float(source_x), float(source_y)detect_minimap_playerL154–171The resulting x/y values are local to the cropped minimap, not desktop coordinates. Bounds are clipped to the available frame. Zero or multiple candidates return None, allowing the caller to avoid inventing a position from ambiguous pixels.
08Read HP/MP with color and column votes#
Here bar is the cropped RGB status-bar image, pixels is its pixel accessor, and valid_columns begins empty. Red-dominant pixels vote for HP; blue-dominant pixels vote for MP. A column needs enough matching rows before it counts.
Estimate fill length, not text
valid_columns: list[int] = []
for column in range(bar.width):
matches = 0
for row in range(bar.height):
red, green, blue = pixels[column, row]
if kind == "hp":
colored = red >= 130 and red >= green * 1.5 and red >= blue * 1.5
else:
colored = blue >= 110 and blue >= red * 1.25 and blue >= green * 1.15
matches += int(colored)
if matches >= max(2, int(bar.height * 0.2)):
valid_columns.append(column)
if not valid_columns:
return 0.0
percent = 100 * (valid_columns[-1] + 1) / bar.width
return 100.0 if percent >= 95 else round(percent, 1)_bar_percent_from_imageL293–308The percentage uses the rightmost valid column, not the proportion of all colored pixels. No valid column yields zero; this snapshot rounds values of at least 95 to 100. A misplaced region or a stray colored pixel can affect the result, so verify the crop before enabling recovery actions.
09Keep the planner independent of vision#
HorizontalHuntPlanner receives an Observation instead of a screenshot. same_line_targets first filters by foot-height tolerance; decide then picks the nearest horizontal target, attacks within range, approaches outside range, or patrols when none remains.
def decide(self, observation: Observation) -> Decision:
targets = self.same_line_targets(observation)
if targets:
target = min(targets, key=lambda m: abs(m.center_x - observation.player_x))
delta = target.center_x - observation.player_x
if abs(delta) <= self.config.target_distance:
return Decision(HuntState.ATTACK, "attack_burst", "right" if delta >= 0 else "left", target)
return Decision(HuntState.APPROACH, "move_pulse", "right" if delta > 0 else "left", target)
return Decision(HuntState.PATROL, "move_pulse")HorizontalHuntPlanner.decideL52–60This is pure decision logic: the returned Decision describes intent but sends no key itself. Unit tests can supply artificial positions and targets to check distance boundaries and direction choices without opening the game.
10Run expensive matching less often#
The runtime separates the rate of monster matching from faster route tracking. It uses time.monotonic to determine when to refresh detections and reuses the last result between refreshes.
def _monsters_for_frame(self, frame: np.ndarray) -> tuple:
"""Run expensive monster matching at a lower rate than route tracking."""
if self._monster_detection_suppressed():
self.last_detected_monsters = ()
self.last_monster_detection_at = 0.0
return ()
now = time.monotonic()
if (self.last_monster_detection_at <= 0.0
or now - self.last_monster_detection_at >= self.MONSTER_DETECTION_INTERVAL_SECONDS):
self.last_detected_monsters = tuple(self._detect_monsters(frame))
self.last_monster_detection_at = time.monotonic()
return self.last_detected_monstersHorizontalHuntRunner._monsters_for_frameL155–166Suppressed detection clears the cached monsters rather than keeping an old target indefinitely. This reduces repeated work but also makes cache age important: the planner must not treat a cached observation as proof that the target is still visible. Timing, stop handling and key release remain runtime responsibilities.
From a detector to a reliable system#
Test each layer separately: a screenshot fixture for recognition, a synthetic Observation for planning, and diagnostic mode before sending input. A high match score alone is not proof of a correct detection. Re-check stale observations, occlusion and ambiguous markers; the runtime releases held keys when stopping or leaving its main loop. This article explains the implementation, not a guarantee of compatibility or permission to automate a game.