VideoInput¶

This class provides access to screenshotting via vnc.

Attributes: screenshot_timeout: The time to wait for client.screenshot() to return screenshot_retries: The amount of times to retry getting a screenshot screenshot_sleep_interval: The sleep between retries when trying to get a screenshot

Type: LIBRARY
Scope: GLOBAL

Keywords¶

Find Cursor Position¶

Detect the cursor in the provided image or a new screenshot.

Note: This keyword relies on a bundled cursor detection model, which increases the installed package size. The current model is trained for simple navigation tasks where mouse is clearly visible, and works better for the regular arrow. Use it carefully in other contexts until a more robust model is available.

Return¶

tuple[integer, integer] | None

(x, y) absolute pixel coordinates of the cursor, or None.

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
image	None	None	POSITIONAL_OR_NAMED	No	Image to search; grabs a screenshot if not provided.
confidence	float	0.85	POSITIONAL_OR_NAMED	No	Minimum confidence (0-1) for a detection to be accepted.

Example¶

${position}=    Find Cursor Position
${position}=    Find Cursor Position    confidence=0.8

Find Text¶

Find the specified text in the provided image or grab a screenshot to search from. The region can be specified directly in the robot file using RPA.core.geometry.to_region

Return¶

list[dictionary]

The list of matched text regions where the text was found. Each match is a dictionary with "text", "region", and "confidence".

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
text	string		POSITIONAL_OR_NAMED	Yes	text or regex to search for, use the format regex:<regex-string> if the text we want to find is a regex.
region	None	None	POSITIONAL_OR_NAMED	No	region to search for the text.
image	None	None	POSITIONAL_OR_NAMED	No	image to search from.
color	None	None	POSITIONAL_OR_NAMED	No	target color of the text. If set, matched text in the wrong color will be skipped.
color_tolerance	integer	20	POSITIONAL_OR_NAMED	No	Color tolerance threshold in %
similarity	None	None	POSITIONAL_OR_NAMED	No	Minimum similarity percentage (0-100) for a match when using RapidOCR. If set, overrides ${OCR_SIMILARITY_THRESHOLD} for this call only.
confidence	None	None	POSITIONAL_OR_NAMED	No	Minimum confidence percentage (0-100) for a match when using RapidOCR. If set, overrides ${OCR_CONFIDENCE_THRESHOLD} for this call only.

Example¶

${matches}=    Find Text    Continue
${matches}=    Find Text    regex:[0-9]{3}
&{region}=    Create Dictionary
...    left=0    top=0    right=800    bottom=600
${matches}=    Find Text    Continue    region=${region}
${matches}=    Find Text    Continue
...    similarity=90    confidence=70

Get Text Position¶

Get the center position of the best match for the specified text. The region to search can be also specified. The center position is round to the nearest integer.

Run with --debug option (or YARF_LOG_LEVEL=DEBUG) to always log the image with the matched region.

Return¶

tuple[integer, integer]

The x and y coordinates of the center of the best match

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
text	string		POSITIONAL_OR_NAMED	Yes	The text to match on screen
region	None	None	POSITIONAL_OR_NAMED	No	The region to search for the text

Example¶

${x}    ${y}=    Get Text Position    Continue
Move Pointer To Absolute    ${x}    ${y}

Grab Screenshot¶

Grabs the current frame through screencopy.

Return¶

Image

Pillow Image of the frame

Raises¶

TimeoutError: If unable to get a screenshot within the timeout period.

Example¶

${image}=    Grab Screenshot

Log Screenshot¶

Grab an image and add it to the HTML log.

Return¶

None

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
msg	string		POSITIONAL_OR_NAMED	No	Message to log with the image

Example¶

Log Screenshot    Desktop after login

Match¶

Grab screenshots and compare until there's a match with the provided template or timeout.

Return¶

list[Region]

list of matched regions

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
template	string		POSITIONAL_OR_NAMED	Yes	path to an image file to be used as template
timeout	integer	10	POSITIONAL_OR_NAMED	No	timeout in seconds
tolerance	float	0.8	POSITIONAL_OR_NAMED	No	The tolerance for image comparison in the compare_images method
region	None	None	POSITIONAL_OR_NAMED	No	the region to search for the template in

Example¶

${regions}=    Match    ${CURDIR}/button.png
Match    ${CURDIR}/button.png    timeout=30    tolerance=0.9

Match All¶

Grab screenshots and compare with the provided templates until a frame is found which matches all templates simultaneously or timeout.

Return¶

list[dictionary]

List of matched regions and template path matched

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
templates	Sequence		POSITIONAL_OR_NAMED	Yes	sequence of paths to image files to use as templates
timeout	integer	10	POSITIONAL_OR_NAMED	No	timeout in seconds
tolerance	float	0.8	POSITIONAL_OR_NAMED	No	The tolerance for image comparison in the compare_images method

Example¶

${templates}=    Create List    ${CURDIR}/ok.png    ${CURDIR}/cancel.png
${matches}=    Match All    ${templates}    timeout=30

Match Any¶

Grab screenshots and compare with the provided templates until there's at least one match or timeout.

Return¶

list[dictionary]

list of matched regions and template path matched

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
templates	Sequence		POSITIONAL_OR_NAMED	Yes	sequence of paths to image files to use as templates
timeout	integer	10	POSITIONAL_OR_NAMED	No	timeout in seconds
tolerance	float	0.8	POSITIONAL_OR_NAMED	No	The tolerance for image comparison in the compare_images method
region	None	None	POSITIONAL_OR_NAMED	No	the region to search for the template in

Example¶

${templates}=    Create List    ${CURDIR}/ok.png    ${CURDIR}/cancel.png
${matches}=    Match Any    ${templates}    timeout=30

Match Text¶

Wait for specified text to appear on screen and get the position of the best match. The region can be specified directly in the robot file using RPA.core.geometry.to_region.

Return¶

tuple[list[dictionary], Image]

It returns a tuple with:

The list of matched text regions where the text was found, sorted by confidence.
The image (used for debugging).

Each match is a dictionary with "text", "region", and "confidence".

Raises¶

ValueError: If the specified text isn’t found in time

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
text	string		POSITIONAL_OR_NAMED	Yes	text or regex to match, use the format regex:<regex-string> if the text we want to find is a regex.
timeout	integer	10	POSITIONAL_OR_NAMED	No	Time to wait for the text to appear
region	None	None	POSITIONAL_OR_NAMED	No	The region to search for the text
color	None	None	POSITIONAL_OR_NAMED	No	The color of the searched text
color_tolerance	integer	20	POSITIONAL_OR_NAMED	No	The tolerance of the color of the searched text
similarity	None	None	POSITIONAL_OR_NAMED	No	Minimum similarity percentage (0-100) for a match when using RapidOCR. If set, overrides ${OCR_SIMILARITY_THRESHOLD} for this call only.
confidence	None	None	POSITIONAL_OR_NAMED	No	Minimum confidence percentage (0-100) for a match when using RapidOCR. If set, overrides ${OCR_CONFIDENCE_THRESHOLD} for this call only.

Example¶

${matches}    ${image}=    Match Text    Continue
Match Text    Continue    timeout=60
${matches}    ${image}=    Match Text    Continue
...    similarity=90    confidence=70

Read Text¶

Read the text from the provided image or grab a screenshot to read from.

Return¶

string

text read from the image

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
image	None	None	POSITIONAL_OR_NAMED	No	image to read text from

Example¶

${text}=    Read Text
${image}=    Grab Screenshot
${text}=    Read Text    ${image}

Restart Video Input¶

Restart video stream process if needed.

Return¶

None

Example¶

Restart Video Input

Set Ocr Method¶

Set the OCR method to use.

Return¶

None

Raises¶

ValueError: If the specified method is not supported.

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
method	string	rapidocr	POSITIONAL_OR_NAMED	No	OCR method to use. Either “rapidocr” or “tesseract”.

Example¶

Set Ocr Method    tesseract

Start Video Input¶

Start the video stream. No-op for VNC.

Return¶

None

Example¶

Start Video Input

Stop Video Input¶

Stop the video stream. No-op for VNC.

Return¶

None

Example¶

Stop Video Input

Wait Still Screen¶

Monitors the screen for a set 'duration' (e.g., 30s), checking every 'interval' (e.g., 5s). Fails if the screen is not still for still_duration.

Return¶

None

Raises¶

TimeoutError: If the screen does not remain still for the required still_duration within the total duration.

Positional and named arguments¶

Name	Type	Default Value	Kind	Required	Documentation
duration	float	30.0	POSITIONAL_OR_NAMED	No	Total time to monitor the screen (in seconds)
still_duration	float	10.0	POSITIONAL_OR_NAMED	No	Time the screen must remain still (in seconds)
screenshot_interval	float	1.0	POSITIONAL_OR_NAMED	No	Interval between screenshots (in seconds)

Example¶

Wait Still Screen
Wait Still Screen    duration=60    still_duration=5