mirror of
https://github.com/appium/appium.git
synced 2026-05-03 00:41:07 -05:00
docs: Add documentation on input events (#14219)
This commit is contained in:
Mykola Mokhnach
@@ -0,0 +1,72 @@
|
||||
## Low-Level Insights on Android Input Events
|
||||
|
||||
|
||||
### What Are Input Events
|
||||
|
||||
Android OS uses events concept to handle signals received from different input devices.
|
||||
It supports a wide range of different devices, such as touch screen, light pen, mouse, keyboard,
|
||||
but most of them are using [MotionEvent](https://developer.android.com/reference/android/view/MotionEvent) or [KeyEvent](https://developer.android.com/reference/android/view/KeyEvent) APIs, which are derived from the base [InputEvent](https://developer.android.com/reference/android/view/InputEvent) class. These APIs are quite flexible and support a wide range of different settings.
|
||||
We are particularly interested in the part of these APIs, which are responsible for touch and
|
||||
keyboard events generation/emulation.
|
||||
|
||||
|
||||
### How Input Events Are Working
|
||||
|
||||
An event is an object, which is generated in response to a signal from an input device.
|
||||
These objects are then delivered to the corresponding kernel subsystem, which processes them
|
||||
and notifies all listening processes about taps, key presses, swipes, etc.
|
||||
This means that in order to emulate a signal generated by
|
||||
an external device, such as touch screen, it is necessary to just send event objects with
|
||||
the same properties and in the same sequence as they would be generated by a real device.
|
||||
|
||||
|
||||
### Lets Simulate a Single Tap
|
||||
|
||||
Each input device has a set of actions whose property ranges and sequences are already predefined
|
||||
in the operating system. These actions we call "tap", "swipe" or "double tap", etc. The properties
|
||||
of each action could be found either in the Android documentation or in the OS source code.
|
||||
In order to perform events sequence, which is recognized as single tap, it is necessary to generate
|
||||
the following motion events:
|
||||
- `ACTION_POINTER_DOWN`
|
||||
- wait 125ms (525ms or longer wait will synthesize a long tap action instead)
|
||||
- `ACTION_POINTER_UP`. The `downTime` property should be set to the same timestamp as for `ACTION_POINTER_DOWN`
|
||||
|
||||
It is also important, that coordinates and other properties of both the starting and the closing event should be equal except of the `eventTime` one, which is always equal to the current system timestamp in milliseconds (`SystemClock.uptimeMillis()`).
|
||||
The `MotionEvent` object itself could be created via [obtain](https://developer.android.com/reference/android/view/MotionEvent#obtain(long,%20long,%20int,%20float,%20float,%20int)) API, where parameters are the corresponding event properties.
|
||||
|
||||
After events are created they must be passed to the system for execution.
|
||||
Such action is not secure, so it is only possible in instrumented tests via [injectInputEvent](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/app/IUiAutomationConnection.aidl) method of `IUiAutomationConnection` interface.
|
||||
This is a very low-level method and it can only be accessed via reflection in automated tests.
|
||||
Normally, UiAutomator APIs have wrappers over it (like `touchDown`, `touchMove`, etc.), that already simulate the stuff described above.
|
||||
|
||||
|
||||
### How About More Complicated Actions
|
||||
|
||||
In theory it is possible to emulate any input action using a generated events sequence.
|
||||
Although, some actions, like multi-finger swipe,
|
||||
are really complicated and require a lot of events to be generated
|
||||
with correct properties and timings. The OS simply ignores given events if they don't follow
|
||||
internal action requirements. There is also a little assistance from UiAutomator framework,
|
||||
because Google only has wrappers for a limited set of simple actions, like `tap`, `drag` or `swipe`.
|
||||
So, in order to generate two-finger symmetric swipe we need to supply the following events chain:
|
||||
- `ACTION_POINTER_DOWN` (finger1)
|
||||
- `ACTION_POINTER_DOWN` (finger2)
|
||||
- start a loop, that generates `ACTION_POINTER_MOVE` event each `20ms` for both `finger1` and `finger2` until `ACTION_POINTER_UP` is performed. The `downTime` should be set to the same timestamp as for the corresponding `ACTION_POINTER_DOWN`. The coordinates of each move event should be points belonging to the path between the corresponding start and end point coordinates normalized by the current timestamp (x0 + sqrt(sqr(x0) + sqr(x1))) * k, y0 + sqrt(sqr(y0) + sqr(y1))) * k).
|
||||
- `ACTION_POINTER_UP` (finger1) The `downTime` property should be set to the same timestamp as for the corresponding `ACTION_POINTER_DOWN`
|
||||
- `ACTION_POINTER_UP` (finger2) The `downTime` property should be set to the same timestamp as for the corresponding `ACTION_POINTER_DOWN`
|
||||
|
||||
Google uses 5ms as interval duration between move events in UiAutomator code,
|
||||
but according to our observations this value is too little,
|
||||
which causes noticeable delays in actions execution.
|
||||
|
||||
|
||||
### Further Reading
|
||||
|
||||
Unfortunately, there is no so much detailed information on this topic. The only reliable source
|
||||
of the information are Android OS sources themselves. Consider visiting the following resources:
|
||||
|
||||
- https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewConfiguration.java
|
||||
- https://android.googlesource.com/platform/frameworks/uiautomator/+/refs/heads/master
|
||||
- https://github.com/appium/appium-uiautomator2-server/tree/master/app/src/main/java/io/appium/uiautomator2/utils/w3c
|
||||
- https://github.com/appium/appium-uiautomator2-server/tree/master/app/src/test/java/io/appium/uiautomator2/utils/w3c
|
||||
- https://github.com/appium/appium-espresso-driver/tree/master/espresso-server/app/src/androidTest/java/io/appium/espressoserver/lib/helpers/w3c
|
||||
@@ -0,0 +1,58 @@
|
||||
## Low-Level Insights on iOS Input Events
|
||||
|
||||
|
||||
### What Are Input Events
|
||||
|
||||
iOS uses events concept to handle signals received from different input devices. An event is an object, which is generated in response to a signal from an input device.
|
||||
These objects are then delivered to the corresponding kernel subsystem, which processes them
|
||||
and notifies all listening processes about taps, key presses, swipes, etc.
|
||||
This means that in order to emulate a signal generated by
|
||||
an external device, such as touch screen, it is necessary to just send event objects with
|
||||
the same properties and in the same sequence as they would be generated by a real device.
|
||||
|
||||
|
||||
### Lets Simulate a Single Tap
|
||||
|
||||
The events API itself is a part of Apple private API and is not open sourced and neither
|
||||
it is documented. XCTest framework also does not expose any public APIs for input events
|
||||
generation. Although, there is a possibility to perform events generation via XCTest
|
||||
private undocumented APIs. In particular, we are interested in [XCPointerEventPath](https://github.com/appium/WebDriverAgent/blob/master/PrivateHeaders/XCTest/XCPointerEventPath.h) and [XCSynthesizedEventRecord](https://github.com/appium/WebDriverAgent/blob/master/PrivateHeaders/XCTest/XCSynthesizedEventRecord.h) interfaces.
|
||||
These APIs allow to create chains of input events and supply them to the system kernel for execution.
|
||||
|
||||
In order to synthesize a single tap it is necessary to:
|
||||
- Create a new `XCPointerEventPath` instance and init it for touch at the starting point
|
||||
- Add a new `liftUp` event at `0.125s` offset using `liftUpAtOffset:` method
|
||||
- Add the generated event path object to `XCSynthesizedEventRecord` instance using `addPointerEventPath:` method
|
||||
- Execute the events using `synthesizeWithError:` method of `XCSynthesizedEventRecord` instance and control the returned error
|
||||
|
||||
There are several limitations to these APIs:
|
||||
- Each `XCPointerEventPath` instance can only be executed for a single action. If one tries to add, for example, two taps to a single path, then these are effectively ignored
|
||||
- Each `XCPointerEventPath` instance can only be initialized for a particular pointer type: touch, mouse (since Xcode 10.2) or keyboard (since Xcode 10.2)
|
||||
- Events can only be added with increasing offset values to an existing `XCPointerEventPath` instance
|
||||
|
||||
|
||||
### How About More Complicated Actions
|
||||
|
||||
Unfortunately, the API is private and has zero documentation.
|
||||
That is why one can only figure out what it really can while playing with it and trying different
|
||||
input combinations.
|
||||
It is known that providing multiple `XCPointerEventPath` instances with
|
||||
overlapping timeouts will generate a multitouch action with the amount of fingers equal
|
||||
to the amount of the supplied event paths.
|
||||
So, in order to generate two-finger symmetric swipe we need to supply the following events:
|
||||
|
||||
- Create a two `XCPointerEventPath` instances and init them for touch at the starting point
|
||||
- Add a `moveToPoint` event at `0.525s` offset using `moveToPoint:` method to each path
|
||||
- Add a `liftUp` eventa at `0.525s` offset using `liftUpAtOffset:` method to each path
|
||||
- Add the generated event paths to `XCSynthesizedEventRecord` instance using `addPointerEventPath:` method
|
||||
- Execute the events using `synthesizeWithError:` method of `XCSynthesizedEventRecord` instance and control the returned error
|
||||
|
||||
|
||||
### Further Reading
|
||||
|
||||
Unfortunately, there is no information on this topic at all (private API `¯\_(ツ)_/¯`). Consider visiting the following resources:
|
||||
|
||||
- https://github.com/appium/WebDriverAgent/tree/master/PrivateHeaders/XCTest
|
||||
- https://github.com/appium/WebDriverAgent/blob/master/WebDriverAgentTests/IntegrationTests/FBW3CTouchActionsIntegrationTests.m
|
||||
- https://github.com/appium/WebDriverAgent/blob/master/WebDriverAgentTests/IntegrationTests/FBW3CMultiTouchActionsIntegrationTests.m
|
||||
- https://github.com/appium/WebDriverAgent/blob/master/WebDriverAgentLib/Utilities/FBW3CActionsSynthesizer.m
|
||||
@@ -238,6 +238,7 @@ module.exports = {
|
||||
]],
|
||||
['Image Comparison', 'image-comparison.md'],
|
||||
['iOS', ['ios',
|
||||
['Low-Level Insights on iOS Input Events', 'actions.md'],
|
||||
['XCUITest Mobile Gestures', 'ios-xctest-mobile-gestures.md'],
|
||||
['XCUITest Mobile App Management', 'ios-xctest-mobile-apps-management.md'],
|
||||
['iOS Pasteboard Guide', 'ios-xctest-pasteboard.md'],
|
||||
@@ -248,6 +249,7 @@ module.exports = {
|
||||
['Pushing/Pulling files', 'ios-xctest-file-movement.md'],
|
||||
]],
|
||||
['Android', ['android',
|
||||
['Low-Level Insights on Android Input Events', 'actions.md'],
|
||||
['UiSelector Guide', 'uiautomator-uiselector.md'],
|
||||
['Espresso Datamatcher Guide', 'espresso-datamatcher-selector.md'],
|
||||
['Android Code Coverage Guide', 'android-coverage.md'],
|
||||
|
||||
Reference in New Issue
Block a user