Screen-read automation on Android

What it does

A screen-aware script can wait for visible state, read UI element text when the app exposes it, read selected text through OCR, match an image, or check a pixel color before deciding what to do next.

How ScriptTap handles it

Accessibility-based UI reading uses Android's exposed UI tree. In UI Find and Delay Until UI Element, the target-square beside Query opens the EditorSelector Select UI Element mode; choosing a visible exposed node fills Query, Target, Match=Exact, and Nth. OCR reads selected visible text from the screen with on-device Latin, Chinese, Devanagari, Japanese, or Korean models that ship with ScriptTap, work offline immediately after installation, and create the selected recognizer lazily when OCR runs. Image and pixel checks use user-controlled screen capture. Scripts can combine these separate signals in one workflow.

Good uses

• Wait until a visible label, prompt, button, or icon is present.
• Branch based on text that appears in a selected area.
• Use a pixel or image condition when an app exposes no reliable UI text.

Layered screen reading

Accessibility handles exposed UI text and nodes, OCR reads visible pixel text, and image or pixel tools recognize visual states. This layered approach lets each script use the clearest user-approved signal available on screen.

Privacy and safety

Do not use screen-read workflows to collect passwords, payment details, or private messages. Keep scripts focused on the visible state needed for the user's own workflow and review the Privacy Policy for current permission details.

Mini-workflow example

Open Select UI Element from the target-square beside Query and choose an exposed button label. For pixel-rendered text, select a tight OCR area. For an icon, use image matching with that icon as the asset.

Related pages

• OCR automation
• Image matching automation
• No-root Android automation