Screen-aware automation
Screen-read automation on Android
Screen-aware scripts can use different kinds of evidence. ScriptTap separates Accessibility-based UI reading, selected-area OCR, image matching, and pixel checks so the user can choose the right signal.
What it does
A screen-aware script can wait for visible state, read UI element text when the app exposes it, read selected text through OCR, match an image, or check a pixel color before deciding what to do next.
How ScriptTap handles it
Accessibility-based UI reading uses Android's exposed UI tree. OCR reads selected visible text from the screen. Image and pixel checks use user-controlled screen capture. These are related tools, not a single view into every app.
Good uses
- Wait until a visible label, prompt, button, or icon is present.
- Branch based on text that appears in a selected area.
- Use a pixel or image condition when an app exposes no reliable UI text.
Limits
Some apps expose little Accessibility data, some text is only visible as pixels, and some visual states change with animation or theme. ScriptTap does not expose private app internals or read protected content that Android does not provide to the user-approved feature path.
Privacy and safety
Do not use screen-read workflows to collect passwords, payment details, or private messages. Keep scripts focused on the visible state needed for the user's own workflow and review the Privacy Policy for current permission details.
Mini-workflow example
First try a UI read for a button label. If the app does not expose that label, switch to OCR for the small area where the label appears. If the label is an icon instead of text, use image matching for that icon.