**DIFAI ERC Advanced Grant Project**
# The Android touch input pipeline
*Last updated: 2024-08-05*
This article aims to describe the flow of touchscreen events through the Android software stack. The starting point is the source code of the touchscreen drivers in the Android variant of the Linux kernel. Events then flow through the Linux input event subsystem, the Android C++ framework, the Android Java framework, and finally to Android applications.
Android and Linux support multiple types of input events. The scope of this article is limited to touch events only, with the goal being to provide some insight into what goes on between the hardware and application levels.
It is not intended to be an exhaustive examination of every line of code in the event handling pipeline; some less relevant parts of the code are skimmed over or omitted entirely.
If you have any questions/comments/corrections on this article, please contact [Andrew Ramsay](https://www.gla.ac.uk/schools/computing/staff/andrewramsay/).
## Contents
Each of the following sections describes a single section of the touch input pipeline, working through in order from kernel to operating system to the point where the events are handed off to applications.
1. Touchscreen drivers and the Linux kernel
2. Linux input event subsystem
3. Android C++ framework
4. Android Java framework
Most of the information presented below is device-agnostic; some specific references are made to the Pixel Tablet and Pixel 7 Pro in section 1.
## 1. Touchscreen drivers and the Linux kernel
The ultimate source of touch events for most Android devices is the Linux kernel driver for the integrated touch sensing hardware. There are many different sensors available and so the implementation of the driver will vary from device to device, but the basic approach will be the same in most cases.
When loaded during the boot process, the kernel driver will configure the touch sensor by communicating with the device's firmware, typically over an [I2C](https://en.wikipedia.org/wiki/I%C2%B2C) or [SPI](https://en.wikipedia.org/wiki/Serial_Peripheral_Interface) bus. At this initialization stage the driver will also create a system input device that will appear under `/dev/input/` to report the events it will generate.
The driver will also register an [interrupt handler](https://en.wikipedia.org/wiki/Interrupt_handler) function to be called when events are generated. The implementation of this function will be responsible for retrieving the raw event data and repackaging it into discrete events that can be reported to the Linux input subsystem. See the [next section](#2-linux-input-event-subsystem) for details of this subsystem.
A driver may also implement other device-specific functionality related to touch handling. Examples include reducing sensor polling rates to save power when the screen is off, or handling of predefined "wake up" gestures to turn the screen back on.
### 1.1 Interrupt handlers
An interrupt handler is a function inside a kernel driver (usually written in C) that will be executed in response to an "interrupt" (event) being generated by the hardware the driver is responsible for managing.
The details of interrupt handling are not directly relevant to this topic, but to illustrate the concept there is a very simple example of an input driver [here](https://www.kernel.org/doc/html/latest/input/input-programming.html). The `button_interrupt` function is registered as an interrupt handler and when the interrupt is generated by the hardware the handler is triggered, constructing and reporting a button event:
```c
static irqreturn_t button_interrupt(int irq, void *dummy)
{
// create a "button down" event
input_report_key(button_dev, BTN_0, inb(BUTTON_PORT) & 1);
// signals that this is the end of this collection of events,
// allowing the button event to be propagated to event listeners
input_sync(button_dev);
return IRQ_HANDLED;
}
```
### 1.2 What data is available to a touchscreen driver?
As a minimum, a driver will usually receive a simple array of touch points with several fields associated with each point:
* a point ID
* a status indicator (e.g. "finger is down", "no finger detected")
* x/y coordinates
* type of touch ("finger", "palm", etc)
There may also be other fields available depending on the hardware capabilities, including information on the size and orientation of the touch contact patch, or possibly even the pressure being applied if the hardware can provide this.
The driver will also usually have a fixed upper limit on the number of fingers that can be detected, presumably reflecting what is encoded into the firmware.
Although the driver will have access to the raw capacitive data from the sensor, under normal circumstances this would only used for debugging or testing purposes. The work of converting the capacitive data into a discrete set of touch points happens at the firmware level.
Some drivers do create an input device that can be used to retrieve this raw capacitive data, but on unmodified phones these will not be accessible to regular applications due to the permissions applied to the devices. If any application had access to the raw touch data, it would create a multitude of security issues.
For example: a malicious application could monitor touches while another application was in the foreground and retrieve passwords typed using the onscreen keyboard. As a result, if you wish to access this data it will likely be at minimum necessary to root your device in order to override the default permissions.
### 1.3 What does the driver do with the data?
Once the data is retrieved from the device over the I2C/SPI bus, the task of the driver is to report the touch points through the [input subsystem API](https://www.kernel.org/doc/html/latest/input/index.html). This is covered in more detail in the next section.
As part of this process, the driver may need to apply certain simple transformations to the data provided by the firmware at this stage. Examples of this may include:
* Clamping x/y coordinates to fixed boundaries (to prevent stray events from being reported with off-screen coordinates)
* Converting original units into display pixels
* Dealing with differences in supported hardware (`if capability X exists do A else do B`)
The implementation in the driver will usually involve looping over the array of touch points, calling the appropriate input API functions for each point to create a collection of events for the active set of points. The collected events are then reported to the input subsystem.
### 1.4 Modifying kernel drivers
This section will briefly describe modifications that can be applied to the touchscreen drivers for the Pixel Tablet and Pixel 7 Pro to make some of the internal touch data from the driver available to applications. If you're not interested in this, continue from [section 2](#2-the-linux-input-event-subsystem).
**Please note that following these instructions will involve a factory reset of your device, and may leave it in a unbootable state if something goes wrong.**
Proceed at your own risk!
#### Device 1: Pixel Tablet
##### 1. Initial preparation
Follow the instructions at https://source.android.com/docs/setup/build/building-pixel-kernels to unlock the bootloader of your device, flash an appropriate Android build, and check out a copy of the kernel sources.
The "Building Pixel Kernels" page recommends using the `aosp-master-with-phones` build which among other things enables root access through `adb`. However at the time of writing (April 2024) this build hadn't been updated for over 6 months, and the instructions below will also work if you use a production build because [Magisk](https://github.com/topjohnwu/Magisk) will be used to gain superuser access.
Once you reach the point in the guide where you need to compile the kernel, use this command:
> SKIP_MRPROPER=1 FAST_BUILD=1 BUILD_AOSP_KERNEL=1 ./build_tangorpro.sh
Without defining `BUILD_AOSP_KERNEL=1` it appears the build script will not generate the `system_dlkm.img` image that is required later.
If you find the build process fails with an error message mentioning `libsubcmd`, you will need to modify the following files in the kernel repository:
* `aosp/tools/lib/subcmd/Makefile`
* `private/gs-google/tools/lib/subcmd/Makefile`
The required modification is shown in [this diff](https://review.lineageos.org/c/LineageOS/android_kernel_google_gs201/+/365943/2/tools/lib/subcmd/Makefile).
Once the build process finishes successfully, you can proceed to the next section.
##### 2. Patching the kernel
Download this patch file [pixel_tablet.patch](misc/pixel_tablet.patch) and apply it to the source folder for the driver:
```sh
cd /private/google-modules/touch/novatek/nt36xxx
patch -p0 < pixel_tablet.patch
```
Then recompile the kernel and check for any error messages. If the build completes successfully the new build should then be ready to flash.
The modified driver will now create a character device at `/dev/nt36xxx` when the device is booted. Reading from this device with a sufficiently large buffer will return two structures packed sequentially.
The first structure is an array of touch points as reported by the device firmware. The array is always 10 values long. Each entry consists of 6 32-bit unsigned integers. These are:
* x and y positions
* contact patch size
* a strength/pressure value
* a touch ID
* a touch status (distinguishing between finger vs no finger).
The size of this structure is 10 x 24 = 240 bytes.
The second structure is an array of raw capacitive readings from the touchscreen sensor. On this particular device the grid is 50x32, so there are 1600 values. Each of these is a 16-bit signed integer. The total size of this structure is actually 3202 bytes rather than 3200 bytes because 2 bytes of other currently unused information are appended to the raw data.
The full structure size is 240 + 3202 = **3442 bytes**, so this is the minimum buffer size applications should use when reading from the character device.
##### 3. Flashing the kernel
To make flashing the various images to the device simpler, you can use the following shell script which should be copied into the directory where you checked out the kernel code:
```sh
#!/bin/sh
# Reboot the device into fastboot mode.
# Note that there are two distinct modes here:
# fastboot mode: via "adb reboot bootloader". Used to flash regular partitions
# fastbootD mode: via "fastboot reboot fastboot". Used to flash dynamic partitions
adb reboot bootloader
# required to disable verification before flashing custom kernels according
# to the documentation (unsure if disable-verity is needed)
fastboot oem disable-verification
fastboot oem disable-verity
# Flash the regular images: boot, dtbo, vendor_kernel_boot
# These target regular non-dynamic partitions
fastboot flash boot "./out/mixed/dist/boot.img"
fastboot flash dtbo "./out/mixed/dist/dtbo.img"
fastboot flash vendor_kernel_boot "./out/mixed/dist/vendor_kernel_boot.img"
# Reboot into fastbootd mode
fastboot reboot fastboot
# And then flash the two images that target dynamic partitions
fastboot flash vendor_dlkm "./out/mixed/dist/vendor_dlkm.img"
fastboot flash system_dlkm "./out/mixed/dist/system_dlkm.img"
# Wipe user data (probably not required every time)
fastboot -w
# Reboot into system
fastboot reboot
```
This script will reboot the device multiple times as required to flash the various firmware images, then finish by rebooting into the normal system mode.
You may find that the newly flashed device is very slow to boot. It can seem to hang for over 5 minutes on the boot logo screen, but if left long enough it should eventually finish booting successfully. This behaviour may be caused by some missing drivers or other misconfiguration introduced by the newly flashed kernel, but doesn't prevent the device from being used to access the raw touch data.
##### 4. Rooting the device
After patching, compiling, and flashing the modified kernel there is a final step required to make accessing the new character device more convenient: rooting the device to allow superuser access. This allows you to easily adjust permissions on the character device and also disable SELinux enforcement if necessary.
The widely recommended method for doing this on current Android devices is to use [Magisk](https://github.com/topjohnwu/Magisk). The installation instructions can be found [here](https://topjohnwu.github.io/Magisk/install.html). Note that for this specific device, you should patch the `init_boot.img` file! (for a more detailed guide, try [here](https://xdaforums.com/t/guide-unlock-bootloader-root-google-pixel-tablet-with-magisk-adb-fastboot-command-lines.4615841/)).
To make the new character device readable from a standard Android application, run `adb shell`, type `su`, grant permission through the Magisk prompt that will appear, then run `setenforce 0 && chmod a+r /dev/nt36xxx`.
#### Device 2: Pixel 7 Pro
##### 1. Initial preparation
Follow the same steps as for the Pixel Tablet. The only difference is that the command to compile the kernel uses a different script:
> SKIP_MRPROPER=1 FAST_BUILD=1 BUILD_AOSP_KERNEL=1 ./build_cloudripper.sh
##### 2. Patching the kernel
Download [pixel_7_pro.patch](misc/pixel_7_pro.patch) and apply it to the source folder for the driver:
```sh
cd /private/google-modules/touch/synaptics
patch -p0 < pixel_7_pro.patch
```
This driver already created a character device at `/dev/tcm0` when the device is booted, but it seems to be used for sending commands to the firmware. The patch replaces some of the existing code so that reading from it will return touch data instead.
As with the Pixel Tablet, reading from this device with a sufficiently large buffer will return two structures packed sequentially.
The first structure is an array of touch points as reported by the device firmware. The array is always 10 values long. Each entry consists of 5 32-bit unsigned integers followed by 2 32-bit signed integers. These are:
* a point index
* x and y positions
* a strength/pressure value
* a touch status value
* contact patch size (major and minor dimensions)
The size of this structure is 10 x 28 = 280 bytes.
The second structure is an array of raw capacitive readings from the touchscreen sensor. On this particular device the grid is 39x18, so there are 702 values. Each of these is a 16-bit unsigned integer.
The full structure size is 280 + 702 = **982 bytes**, so this is the minimum buffer size applications should use when reading from the character device.
##### 3. Flashing the kernel
Identical to the Pixel Tablet instructions.
##### 4. Rooting the device
Identical to the Pixel Tablet instructions.
### References
* https://source.android.com/docs/setup/build/building-pixel-kernels
* [Magisk](https://github.com/topjohnwu/Magisk)
* [Source code for the Novatek touch sensor used by the Pixel Tablet](https://android.googlesource.com/kernel/google-modules/touch/novatek_touch/+/refs/heads/android-gs-tangorpro-5.10-android14/nt36xxx/)
* [Source code for the Synaptics touch sensor used by the Pixel 7 Pro](https://android.googlesource.com/kernel/google-modules/touch/synaptics_touch/+/refs/heads/android-gs-pantah-5.10-android14)
## 2. The Linux input event subsystem
### 2.1 Handling multi-touch events
Linux input event handling is a complex topic, so here we focus solely on the handling of multi-touch events. This subset of the API is documented [here](https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html).
The most important part of this document is the description of the multi-touch reporting protocol. Note that there are two versions of this protocol named `A` and `B`, but as indicated near the top of the documentation page all current drivers should now be using version `B`.
The input event API requires that individual touches from the same device be reported as separate events by the kernel driver. The multi-touch protocol provides a mechanism to effectively bundle these events together and report them as having occurred at the same time as far as higher levels in the event handing pipeline are concerned.
To illustrate further, this is an annotated version of the example event sequence for protocol version `B`:
```bash
ABS_MT_SLOT 0 # prepare to update the contents of slot 0 (making it the active slot)
ABS_MT_TRACKING_ID 45 # assign a tracking ID of 45 to slot 0
ABS_MT_POSITION_X x[0] # set the x-coordinate value in slot 0 to x[0]
ABS_MT_POSITION_Y y[0] # set the y-coordinate value in slot 0 to y[0]
ABS_MT_SLOT 1 # slot 1 now becomes the active slot
ABS_MT_TRACKING_ID 46 # assign a tracking ID of 46 to slot 1
ABS_MT_POSITION_X x[1] # set the x-coordinate value in slot 1 to x[1]
ABS_MT_POSITION_Y y[1] # set the y-coordinate value in slot 1 to y[1]
SYN_REPORT # generate a "sync" event, informing event listeners about both new touches
```
### 2.2 Consuming events
Each physical device registered with the input subsystem will create at least one [character device](https://linux-kernel-labs.github.io/refs/heads/master/labs/device_drivers.html) in the `/dev/input/` directory. These are special files that provide access to a stream of events from an input device.
For example, a mouse may be represented as `/dev/input/mouse0` and repeatedly reading from this file would produce a stream of mouse motion and button click events.
System services and applications read from these device files to retrieve input events. It's possible to simply dump the raw data stream by running a command like `sudo cat /dev/input/mouse0`, but most consumers will use [libevdev](https://www.freedesktop.org/wiki/Software/libevdev/) or some other [language-specific binding](https://github.com/gvalkov/python-evdev) for this task.
### 2.3 Other features
Beyond the basic x/y coordinates, the API provides scope for reporting other types of touch data if the source provides them. For example, [section 3.5](https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html#event-usage) of the documentation describes how drivers may optionally report extra information about a touch, including pressure and information about the size and/or orientation of the touch contact patch.
It's also worth noting that the API doesn't assume a finger is being used. The description of the `ABS_MT_TOOL_TYPE` parameter states:
> The protocol currently mainly supports MT_TOOL_FINGER, MT_TOOL_PEN, and MT_TOOL_PALM
i.e. there is support for reporting touches from a finger or a stylus/pen, and that palm touches may also be reported. The latter case would apply to devices that try to detect inadvertent palm touches and report them as such instead of generating erroneous finger touch events.
Another important task performed by the API is [finger tracking](https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html#finger-tracking). According to the documentation, all type `B` multi-touch devices need to be capable of supplying the API with unique touch IDs to distinguish and track fingers between successive frames. As noted in the previous section, the implementation of this feature is typically done in the device firmware rather than in the driver.
### References
* [Linux input subsystem documentation](https://www.kernel.org/doc/html/latest/input/index.html)
* [Multi-touch protocol documentation](https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html)
* [libinput: Palm detection](https://wayland.freedesktop.org/libinput/doc/1.11.3/palm_detection.html)
## 3. Android C++ framework input event handling
The Android operating system is split into two sections, one implemented in C++ and one implemented in Java. The C++ framework sits below the Java framework that applications interact with directly. implementing a raft of functionality best written in native code for various reasons.
One of the tasks the C++ framework is responsible for is consuming events generated through the input subsystem, so examining what happens to the events once they are read from input devices is the focus of this section.
The path that events take through the C++ framework is complex to describe in detail. A higher-level summary is given first in the [outline section](#31-c-framework-input-event-handling-an-outline); for a more in-depth examination see [the following section](#32-c-framework-input-event-handling-the-details).
The website [cs.android.com](https://cs.android.com/android/platform/superproject/main/+/main:) is very helpful to follow along with in this section as checking out a complete local copy of the AOSP source code is a lengthy process requiring a lot of free disk space. Links to most of the classes and methods discussed below are provided inline.
### 3.1 C++ framework input event handling: an outline
This section contains a simplified outline of the event handling procedure (more detail can be found [in the next section](#32-c-framework-input-event-handling-the-details)).
---
**Figure** 1: _A simplified diagram of the Android C++ input event handling pipeline. Events are read from a character device under /dev/input by the `EventHub` class, handed off to an `InputDevice` for "cooking" into an internal Android format, then returned to `InputReader`. From here the cooked events are sent to the Java part of Android using IPC calls._
---
**Gathering events**
* a singleton instance of the `InputManager` class ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/InputManager.cpp)) is created within one of the Android system services. This object owns a singleton instance of a class called `InputReader` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/InputReader.cpp)), which in turn owns a singleton instance of `EventHub` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/EventHub.cpp)).
* The `EventHub` class is described in the code as "Grand Central Station for events", the point at which all input events are read in by the operating system.
* The `InputReader` class polls events from its instance of `EventHub`. These are then parcelled out (based on the ID of the device originating the events) to a collection of `InputDevice` objects ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/InputDevice.cpp)), each of which manages the state for a physical input device.
**Processing events**
* `InputDevice` objects contain one or more "mappers", subclasses of `InputMapper` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/InputMapper.h)) which "transform raw input events into cooked event data". Here "raw input events" means the event format used by the Linux input API, while "cooked event data" refers to internal Android event data structures used by subsequent steps.
* For the multi-touch devices we are concerned with, each of these `InputDevices` will have a mapper called `MultiTouchInputMapper` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/MultiTouchInputMapper.h)). Objects of this class manage a set of event "slots" analogous to the slots used by the multi-touch input API, updating their internal state each time new events are generated by the corresponding physical device.
* The "cooking" of raw events happens within this mapper. Among other things, various types of device-dependent calibration or scaling may be applied during the "cooking" process, in addition to a 2D transformation of the x/y coordinates to place them in the "un-rotated display's coordinate space".
* The end result of the "cooking" process is a list of new Android events (releases, presses, moves, etc) which is ultimately returned to the `InputReader`. From here events flow through a short collection of classes used to do some basic filtering (e.g. rejecting apparent palm touches) and metrics collection before being dispatched.
**Dispatching events**
* The `InputDispatcher` class ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp)) is responsible for figuring out where batches of events should be delivered. At this point the targets are "windows" which are associated with `Activities` in applications.
* Events may be dropped for many reasons at this stage: if a valid target no longer exists, if a system policy prevents sending events to the resolved target, or if the target window is currently disabled or not visible.
* Having found a target, the event is handed over to an `InputPublisher` object ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/libs/input/InputTransport.cpp;l=600;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)). Instances of this class own an `InputChannel` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/include/input/InputTransport.h;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=237)), which is a wrapper around a [Unix socket](https://en.wikipedia.org/wiki/Unix_domain_socket).
* Finally, the event is copied into an `InputMessage` data structure ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/include/input/InputTransport.h)) and an [IPC](https://en.wikipedia.org/wiki/Inter-process_communication) call is made by sending this structure through the socket to a receiver owned by the objects in the Android Java framework responsible for the target window.
### 3.2 C++ framework input event handling: the details
This section explores the flow of events through the Android C++ framework in more detail, building on the overview presented in the previous section.
---
**Figure 2**: _a diagrammatic look at the key parts of the C++ event handling pipeline_. See below for more details.
---
#### Gathering events
The simplest place to start tracing event flow through the Android C++ framework is in the `InputManager` class. A singleton instance of this class is created by one of the Android system services, and it in turn creates a singleton instance of `InputReader` This class is responsible for reading events from a singleton instance of the `EventHub` class.
As noted above, the `EventHub` is described in the code comments as ["Grand Central Station for events"](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/include/EventHub.h;l=232;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e), acting as a single source for collected events from all input devices available to the system.
The `InputReader` object owning the `EventHub` spawns a thread to read events from it in a loop, calling `EventHub::getEvents` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/InputReader.cpp;l=136;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) to retrieve events.
Once the latest batch of available `RawEvent` objects ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/include/EventHub.h;l=59;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d?)) have been retrieved, the `InputReader` batches events sourced from the same input device and hands them off to an `InputDevice` instance that corresponds to the physical device for processing ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/InputReader.cpp;l=356;drc=190beaa49a35da1d9dcf66be9cfccfd23b0eb467)). Each `InputDevice` is responsible for managing the state associated with its physical counterpart.
#### InputDevices and Mappers
Within an `InputDevice` object, a batch of events from the corresponding physical device are broken down further and dispatched to one or more `Mapper` objects (a device may produce multiple types of input event which may require multiple mappers to handle them).
There are various different types of mapper, all subclassing the `InputMapper` base class ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/InputMapper.h)). The full set of classes can be viewed [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/). The purpose of a mapper class is to "transform raw input events into cooked event data", i.e. translating the original data from the Linux event subsystem into the format used in the next stage of the Android input pipeline.
##### MultiTouchInputMapper and TouchInputMapper
For multi-touch input events, the relevant mapper type is the `MultiTouchInputMapper` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/MultiTouchInputMapper.h)). This is itself a subclass of `TouchInputMapper` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.h)), which deals with the different types of supported touch devices. The supported device list includes touchscreens, but it also must support other touch-enabled devices like a drawing tablet where there is no direct mapping between the touch-sensitive surface and the display screen. It also includes functionality to deal with scrolling and button/cursor states that may be associated with certain types of devices, but we can ignore these to focus on the handling of multi-touch events alone.
When a `MultiTouchInputMapper` is processing an event, it delegates a lot of the processing to the superclass implementation in `TouchInputMapper`. After some less relevant updating of button/cursor states, the type of the current event is checked. If it is tagged as being a `SYN_REPORT` event (indicating a batch of related events as produced by the multi-touch input API described in the previous section) a call is made to the `syncTouch` virtual method which is actually implemented by `MultiTouchInputMapper` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/MultiTouchInputMapper.cpp;l=64;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)).
This method is more complex. The `MultiTouchInputMapper` owns an object called a `MultiTouchMotionAccumulator` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/accumulator/MultiTouchMotionAccumulator.h)). This mainly consists of a vector of `Slot` objects, each of which stores a set of attributes for a single touch. These are essentially copied from the Linux input subsystem API, including values like x/y position, pressure/distance values, and touch orientation/size information where available.
When `MultiTouchInputMapper::syncTouch` is called it iterates over the set of `Slots` in the accumulator object. The goal here is to check if each `Slot` is currently empty or populated, and if populated it copies the attributes into a new but very similar data structure called `RawPointerData` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.h;l=67;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) that also stores an array of touches. Most of the simple attributes are copied as-is, but the code also attempts to assign "pointer IDs" to each touch. The output from the function is the copied and slightly modified set of touch data structures.
This method also sets an upper bound on the number of touches that the operating system will handle from a single multi-touch event. If the number of populated slots exceeds `MAX_POINTERS`, any further touches will be ignored. This value is defined [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/include/input/Input.h;drc=13412bb94816e57e3a2de1018c65192f5b1a7261;l=165) and is set to 16. Note that the hardware may impose a lower limit on the number of touches, e.g. the Pixel Tablet touchscreen driver has a limit of 10 simultaneous touches.
Execution now jumps back to `TouchInputMapper::sync` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=1451;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)). After an [interesting workaround](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchCursorInputMapperCommon.h;drc=13412bb94816e57e3a2de1018c65192f5b1a7261;l=51) for Bluetooth input devices, the `TouchInputMapper::processRawTouches` method is called.
A couple of notable things happen [in this method](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=1510). A lot of the code deals with stylus input devices, so can be ignored for regular touchscreens. The newly created array of touch events is looped through, with a call being made on each iteration to `TouchInputMapper::cookAndDispatch`.
`TouchInputMapper::cookAndDispatch` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=1562)) is a fairly complex method when considered in detail, but the overall purpose is to take the current array of touch events stored by a `TouchInputMapper` and convert (or "cook") them into a new list of Android system events that can be dispatched to subsequent stages in the pipeline. It is worth examining some parts of this process more closely.
Before "cooking" the raw touch events, a call is made to a method called `consumeRawTouches` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=1838)). The purpose of this method appears to be to deal with touch events involving virtual keys. These are the on-screen buttons that typically appear along the bottom of the screen, e.g. "Home" or "Back".
In certain circumstances this method may consume the raw touch events, generating virtual key touches and causing the rest of the processing done by `cookAndDispatch` to be effectively skipped. Also in this method is a code block which aims to "filter out stray virtual key presses when interacting with the touch screen" by temporarily disabling the virtual keys for a short period after touches on the main area of the screen. The [code comments](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=1945;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d) explain 2 specific scenarios being targeted here: preventing scrolling/dragging motions from overshooting onto a virtual key, and preventing taps near the bottom of a virtual keyboard from inadvertently hitting a virtual key.
##### "Cooking" events
Back in `TouchInputMapper::cookAndDispatch` and assuming the events were not consumed by virtual keys, the next step is to actually "cook" the raw pointer data into the format used by the rest of the Android C++ framework.
This is implemented by the `TouchInputMapper::cookPointerData` method ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=2286)). In essence this is simply looping over each touch point and copying fields from the "raw" data structure into another similar "cooked" data structure, but there are extra steps required to transform certain values during this process.
Some examples:
* **Touch coordinates**: a key step here is to apply a 2D transformation to the raw x/y coordinates. This converts them from device coordinates to the "un-rotated display's coordinate space", the common convention used subsequently in the processing pipeline (see [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=850) for the implementation of this transform).
* **Touch sizing**: touch devices can report both "major" and "minor" values associated with a touch, corresponding to the input event API event attributes of `ABS_MT_TOUCH_MAJOR` and `ABS_MT_TOUCH_MINOR` (see [the documentation](https://www.kernel.org/doc/html/latest/input/multi-touch-protocol.html#event-usage) for details). If present these provide information about the size of the contact patch.
* Similarly the `ABS_MT_WIDTH_MAJOR` and `ABS_MT_WIDTH_MINOR` attributes can be used to provide information about the tool (or finger) generating the touch event. Depending on the calibration mode of the device, the final cooked `size` value for the touch will be some combination of these values, depending on which are reported by the device driver.
* For example, in the case where a device doesn't provide the "tool" attributes, the "size" value is calculated either by averaging the major and minor sizes (if the latter is valid) or simply taking the major size (if the minor size is unavailable).
* There is also a "scale and bias" step applied here which may 1) multiply the original size by some "scale" value, and/or 2) add a "bias" value to the original size.
* **Touch pressure**: if pressure reporting is supported, this is also multiplied by a "pressure scale" with a value taken from a `Calibration` object ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.h;l=247;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)).
* **Touch distance**: very similar to pressure, if supported the raw value will be multiplied by a calibrated "distance scale".
A simple way to view the various calibration scaling factors on a rooted Android device is to run `dumpsys input` from a superuser shell (standard users will not have access). This will display a mass of useful information from the input subsystem including active window states, detected input devices, hardware information, and calibration coefficients. The calibration information is listed under the `Touch Input Mapper` section, and will look something like this:
```plaintext
Translation and Scaling Factors:
XScale: 0.264
YScale: 0.557
XPrecision: 3.793
YPrecision: 1.796
GeometricScale: 0.410
PressureScale: 0.016
SizeScale: 0.004
OrientationScale: 0.000
DistanceScale: 0.000
HaveTilt: false
TiltXCenter: 0.000
TiltXScale: 0.000
TiltYCenter: 0.000
TiltYScale: 0.000
```
##### Generating Android touch events
`cookAndDispatch` now needs to actually generate Android motion events from the transformed raw events. For multi-touch screens, the `TouchInputMapper` device mode will be `DeviceMode::Direct` meaning control moves into the else branch of this code block ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=1609;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d)):
```c
// Dispatch the touches either directly or by translation through a pointer on screen.
if (mDeviceMode == DeviceMode::POINTER) {
// this block is executed for drawing tablets or similar devices
} else {
if (!mCurrentMotionAborted) {
updateTouchSpots();
out += dispatchButtonRelease(when, readTime, policyFlags);
out += dispatchHoverExit(when, readTime, policyFlags);
out += dispatchTouches(when, readTime, policyFlags);
out += dispatchHoverEnterAndMove(when, readTime, policyFlags);
out += dispatchButtonPress(when, readTime, policyFlags);
}
if (mCurrentCookedState.cookedPointerData.pointerCount == 0) {
mCurrentMotionAborted = false;
}
}
```
The set of `dispatch*` function calls builds up a vector of `NotifyMotionArgs` objects representing events of different types. Most of the implementations are relatively simple and follow the same pattern.
Taking `dispatchButtonPress` as an example, it compares the current `mCurrentCookedState` member variable (populated by `cookAndDispatch` above) with the previous `mLastCookedState` to find the set of button IDs that have just been pressed. For each pressed button, it calls `dispatchMotion` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=3785;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) which constructs and returns a `NotifyMotionArgs` with an `action` type of `AMOTION_EVENT_ACTION_BUTTON_RELEASE` and a large number of other values describing the event.
However for touch events, the most important method is `dispatchTouches` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/reader/mapper/TouchInputMapper.cpp;l=2034;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)). This is a more complex method. A simplfied pseudocode representation might look like this:
```python
out = []
if last touch state == current touch state
# same set of pointers (touches) as previously
if number of touches > 0
# create "touch point moved" events
out += dispatchMotion(..., AMOTION_EVENT_ACTION_MOVE, ...)
else: # our set of touch points have changed
calculate sets of touch IDs that have gone up/down/moved
# update properties of touchs that were tracked in the
# previous call and still exist in the current state
updateMovedPointers(...)
for each touch ID that has gone up
# create "touch point up" event
out += dispatchMotion(..., AMOTION_EVENT_ACTION_POINTER_UP, ...)_
# create "touch point moved" events
if any touch points moved
out += dispatchMotion(..., AMOTION_EVENT_ACTION_MOVE, ...)
for each touch ID that has gone down
# create "touch point down" event
out += dispatchMotion(..., AMOTION_EVENT_ACTION_POINTER_DOWN, ...)
```
Note that each `NotifyMotionArgs` instance actually encapsulates a _set_ of touch actions of the same type, so each call to `dispatchMotion` may be generating events for multiple touches.
##### Dispatching events
The eventual result of all of this complexity is a vector of these `NotifyMotionArgs` objects. Despite the name of the methods in the previous section, no "dispatching" of events is happening at this point. Instead, this vector of event structures is returned back up the call stack until it reaches the original `InputDevice` instance, which returns it to the `InputReader` instance.
Before reaching the `InputDispatcher` object responsible for actually dispatching events, events pass through a small pipeline of objects implementing an interface that allows each one to examine and optionally filter out events. This takes place in the `InputManager` instance. There are several miscellaneous classes involved (although some may be disabled at compile time) as listed in [this comment](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/InputManager.cpp;l=119;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e).
Of these only a few are particularly relevant to touch event handling:
1. `UnwantedInteractionBlocker` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/UnwantedInteractionBlocker.cpp)): filters out touches with large contact patch sizes. For example, [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/tests/UnwantedInteractionBlocker_test.cpp;l=570?) is one of the tests showing a touch with a small size will be accepted while a large one is rejected.
2. `InputProcessor` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/InputProcessor.cpp)): annotates events as either `NONE`, `DEEP_PRESS` (long press) or `AMBIGUOUS_GESTURE`. The classifier performing this task seems to be implemented by a separate system service.
The final step in the pipeline is always the `InputDispatcher` class ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp)). This deals with sending events to the Java framework via IPC calls. However before reaching that stage, the dispatcher first has to determine the correct destination for the events.
The C++ framework at this level has no awareness of the widgets implemented by the Java framework. Instead events are dispatched to "windows", which are managed by a `WindowManager` system service. This service is actually part of the Java framework with an interface [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/WindowManager.java) and an implementation [here](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/WindowManagerImpl.java). The component responsible for drawing to the display based on information provided by the `WindowManager` is called `SurfaceFlinger` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/surfaceflinger/SurfaceFlinger.h)). For more details, see [this page](https://source.android.com/docs/core/graphics/surfaceflinger-windowmanager).
Back in the `InputDispatcher` object, it needs to determine which window handle(s) the current batch of events should be sent to. There is a lengthy method called `findTouchWindowTargetsLocked` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=2309;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) which performs this task, returning a vector of `InputTarget` objects ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputTarget.h;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=35)).
Rather than try to go through all this line by line, it's more helpful to focus on how the `InputTargets` are determined, using the the code in the "pointer down" case as an example:
1. A call to `findTouchedWindowAtLocked` is made ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=1289)). This takes x/y coordinates and iterates over the current set of window handles. The first handle for a window which is not a "spy" window (i.e. transparently reading touches intended for a window below it) and which is currently accepting touches is returned.
2. Assuming a "down" event, a call is then made to `findOutsideTargetsLocked`. This is related to the `WATCH_OUTSIDE_TOUCH` [flag](https://developer.android.com/reference/android/view/WindowManager.LayoutParams#FLAG_WATCH_OUTSIDE_TOUCH) that applications may enable, allowing them to receive a special [event](https://developer.android.com/reference/android/view/MotionEvent#ACTION_OUTSIDE) when motion is detected outside their bounds. This method iterates through the current set of windows until it reaches the "touched window", adding a new `InputTarget` for each window it encounters which has this flag set.
3. A check is made to determine if the permission model allows events to be "injected" into the touched window ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=2350;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)).
4. If any "spy" windows are active, retrieve them and add them to a vector containing the touched window as the first entry ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=2377;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)).
5. Abort if no touchable windows were found.
6. Iterate over the found window(s) and update a `TouchState` object ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/TouchState.h)) accordingly. This manages a vector of current `TouchedWindow` objects ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/TouchedWindow.h)), each of which track their own pointer states.
7. After [more security checks](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=2631;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d), the collection of `TouchedWindows` from the `TouchState` is used to create/update `InputTarget` objects for the selected windows.
8. Finally the vector of `InputTarget` objects is returned.
`InputTarget` objects are important because they are used to store an instance of `InputChannel` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/include/input/InputTransport.h;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=237)). Each instance of this class is a wrapper around a [UNIX socket](https://en.wikipedia.org/wiki/Unix_domain_socket) pair ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/libs/input/InputTransport.cpp;l=408;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)) and this is the transport used to perform IPCs to the Java framework.
Now that the framework has the collection of `InputTargets`, the final steps in the C++ framework's event dispatch process are:
1. [Retrieve](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=2043;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d) a `Connection` object ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/Connection.h)) for each target, which "manages the dispatch state" for an `InputChannel`. Most importantly it handles a queue of outgoing events and events that have been sent but not acknowledged as processed.
2. [Populate the outbound event queue](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;l=3298;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1) on each `Connection`.
3. [Start a dispatch cycle](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1;l=3635) to publish the events in the outbound queue.
4. For motion events this leads to `InputDispatcher::publishMotionEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/services/inputflinger/dispatcher/InputDispatcher.cpp;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1;l=3588)) (where a global scaling factor may be applied to the touch coordinates) and then finally to `InputPublisher::publishMotionEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/native/libs/input/InputTransport.cpp;l=648;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)).
The final `return mChannnel->sendMessage(&msg);` line at the end of this method is where the event message is sent over the Unix socket to the receiver in a process of the Android Java framework.
### References
* https://cs.android.com
* https://newandroidbook.com/files/AndroidInput.pdf
* https://source.android.com/docs/core/graphics/surfaceflinger-windowmanager
## 4. Android Java framework
### 4.1 Important classes and their relationships
Moving to the Android Java framework, the obvious place to pick up the flow of events is the point where they are unmarshalled after the IPC call. However before diving into this, it's important to understand the purpose and relationships of some of the Java classes involved:
1. `Activity` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/Activity.java)). This is the class familiar to all Android developers, representing a "a single, focused thing that the user can do". As described in [the documentation](https://developer.android.com/reference/android/app/Activity), instances of this class will automatically create a `Window` object to display the content of the `Activity` (this is not something developers will typically need to care about). A tree of `View` objects can then be associated with the `Activity` by calling its [setContentView](https://developer.android.com/reference/android/app/Activity#setContentView\(int\)) method.
2. `Window` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/Window.java)) is an abstract class "for a top-level window". This is subclassed by the "Android-specific Window" class `PhoneWindow` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/com/android/internal/policy/PhoneWindow.java;l=529;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d)). `Activity` objects create an instance of `PhoneWindow` at [this point](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/Activity.java;l=8852;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e), and register it with the `WindowManager`.
3. `View` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java)), "the basic building block for user interface components". The `ViewGroup` subclass ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;l=139;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) is "the base class for layouts, which are invisible containers". The [documentation](https://developer.android.com/reference/android/view/View) has more details.
4. `ViewRootImpl` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java)). This is an internal class described as being "The top of a view hierarchy, implementing the needed protocol between `View` and the `WindowManager`". The key points to note in this context are that it contains a reference to a `Window` object (actually a `PhoneWindow`) and a `View`, via a member variable that is an instance of `AttachInfo` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;l=31435;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)). The `View` object here will be the view at the top of the tree of `Views` associated with that window.
The key points to take from this are:
1. Each `Activity` has an associated `Window` which receives touch events
2. Each `Activity` has a tree of `View` objects that make up its UI
3. `ViewRootImpl` provides the link between the `Window` and the `View` at the root of the tree
### 4.2 Creating activities and windows
As noted above, a new `PhoneWindow` is [created](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/Activity.java;l=8643;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1) when an application instantiates an `Activity`. The `Activity` calls the `addView` method of `WindowManagerGlobal` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/Activity.java;l=7059;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)). This creates a `ViewRootImpl` instance which is associated with the supplied `View`, and `ViewRootImpl::setView` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=1362;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)) is called.
The "view" object in this context is actually an instance of `DecorView` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/com/android/internal/policy/DecorView.java)). This is a subclass of `ViewGroup` (and so `View`) into which the view hierarchy of the `Activity` is embedded.
The `DecorView` deals with the system UI elements like the status bar and navigation bar. Importantly this is also when the Java version of the `InputChannel` class is instantiated ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=1451;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)), creating a new socket pair matching those on the C++ side.
Further down in the same method, the newly created channel is associated with a `WindowInputEventReceiver` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=1591;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)), a subclass of `InputEventReceiver` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/InputEventReceiver.java)). This class manages an `InputChannel` and receives `InputEvent` objects transferred over the IPC mechanism from the C++ framework.
### 4.3 Input events in the Java framework
---
**Figure 3**: Event flow in the Java framework. Incoming IPC calls result in events flowing through a pipeline of "stages" heavily focused on IME functionality. Events which are not consumed by earlier stages are then handed off to the `View` at the root of the tree owned by the parent `Activity`.
---
This `onInputEvent` [method](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=9948;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e) of `WindowInputEventReceiver` is the best place to begin tracing the incoming events. After a compatibility check that seems to involve [ensuring backwards compatibility of button states](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/InputEventCompatProcessor.java;l=53;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d), events are added to the receiver's queue before being handled in `doProcessInputEvents` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=9630?)), which in turn calls `deliverInputEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=9657)).
At this point the parent `ViewRootImpl` instance has to decide on the initial destination of the event. The Java framework defines an abstract `InputStage` class ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=6596)) with a variety of subclasses. As the name suggests, each of the subclasses represent a potential stage in the processing of an event, and can also be chained together to allow an event to be passed through multiple stages.
Each stage has the opportunity to optionally perform some processing on the event (including marking it as "handled") before it is forwarded to the next stage.
Almost all of the stages in this pipeline are related to the presence or absence of an Input Method Editor (IME). An IME is a UI component that allows a user to enter text or digits. On a typical phone or tablet without a keyboard, the IME will be the familiar virtual keyboard or number pad.
The events coming in from the IPC channel will be of type `InputEvent`. This has a collection of subclasses, of which `MotionEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/MotionEvent.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=190)) is the relevant class for touch events.
Focusing on `MotionEvent` handling, the pipeline of `InputStages` looks like this ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=1566;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)):
1. `NativePreImeInputStage`: only concerned with `KeyEvents`, potential interception of "Back" key events
2. `ViewPreImeInputStage`: allows a `KeyEvent` to potentially be [consumed](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;l=15769;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d) by a `View` instead of being handled by an active IME
3. `ImeInputStage`: delivers events to the IME. The events may not be consumed, e.g. if the IME is not focused or is not visible/active
4. `EarlyPostImeInputStage`: the first stage to handle pointer (touch) events directly. It forwards any event that isn't a `KeyEvent` or `MotionEvent` and does some processing on those 2 types. For the `MotionEvent` case with a touchscreen device, some [limited processing](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=7132;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1) is done. Among other things this applies a y-offset to the event's coordinates based on the scrolling position tracked by the `ViewRootImpl`.
5. `NativePostImeInputStage`: "Delivers post-ime input events to a native activity." (not touch-specific)
6. `ViewPostImeInputStage`: the most important stage in the scope of this article, it "delivers post-IME input events to the view hierarchy". Touch events are processed separately from others ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1;l=7418)). The root `View` member of the `ViewRootImpl` instance then has its `dispatchPointerEvent` method called (this `View` will be the `DecorView` instance rather than the top level `View`/`ViewGroup` contained in the corresponding `Activity`).
7. `SyntheticInputStage`: "Performs synthesis of new input events from unhandled input events" (this shouldn't be relevant for touch events).
Depending on the type of event, it may be inserted into the pipeline at different stages ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;l=9677;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1)). For example, if the event is synthetic (flagged as "unhandled"), it will skip the first 6 stages and be delivered directly to the final stage which deals with synthesized events.
The other possibility is that events may skip the first few IME stages and be inserted at the `EarlyPostImeInputStage`. This appears to be what happens with touch events.
### 4.4 Delivering `MotionEvents` to touch listeners
Touch events will typically end up passing through to `ViewPostImeInputStage` in the pipeline above, from where they are dispatched through the `processPointerEvent` method of the same class. The [implementation](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewRootImpl.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;bpv=1;bpt=1;l=7431) will then call `mView.dispatchPointerEvent(event)`.
The `mView` object in this instance is the one associated with the `ViewRootImpl` instance, i.e. the `DecorView` (the special `View` that manages the status bar and other system areas).
However because neither `DecorView` nor `ViewGroup` implement the `dispatchPointerEvent` method, this will ultimately result in a call to the base class (`View`) [version of the same method](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=16155).
For touch events, this means a call to `dispatchTouchEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=15826)), but because `ViewGroup` _does_ override this method of `View`, control will ultimately end up in `ViewGroup.dispatchTouchEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;l=2628;drc=efb735f4d5a2f04550e33e8aa9485f906018fe4e)).
This is a much more complex method because it handles the task of relaying the original event to a specific `View`.
(As an aside, just before this point it's possible to override the `onInterceptTouchEvent` method by subclassing `ViewGroup` to access/manipulate all touch events for child views (see [the documentation](https://developer.android.com/develop/ui/views/touch-and-input/gestures/viewgroup) for more details)).
The core of `ViewGroup::dispatchTouchEvent` is [this block of code](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;l=2694;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d) dealing with `DOWN` events. The main purpose of this block is to obtain a list of `TouchTarget` objects which each "describe a touched view and the ids of the pointers that it has captured”:
* If the current event is a `DOWN` action, it loops over the children of the `ViewGroup` in z-order from front to back by iterating backwards through the array of children maintained by the `ViewGroup`.
* If a child `View` is unable to receive events, or if the touch coordinates fall outside of the `View`, the loop skips to the next child.
* Next, it checks for an existing `TouchTarget` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=8975)) for the current `View`. If one is found, the object is updated with the pointer ID(s) of the current touch and the loop ends.
* If no target exists, the event is then offered to the `View` for handling through a call to `dispatchTransformedTouchEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=3082)). This method is principally concerned with passing a transformed version of the original touch event to the child `View`.
* The transformation involved is a 2D translation of the event's x/y coordinates into view-local coordinates. This is done by applying an offset based on the vertical and horizontal scroll positions of the parent `View`/`ViewGroup` and the position of the child `View` within the parent view.
* This modified version of the event is then passed to `View::dispatchTouchEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=15826)).
* If the `View` handles the event, a new `TouchTarget` is added to the linked list maintained by `ViewGroup`. Subsequent events from the same touch IDs will then also be delivered to the same `View` (assuming they stay within its bounds) based on these target objects
Returning to `View::dispatchTouchEvent`, after a final security check which prevents touches being delivered to obscured/hidden windows ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=15904)) the `View` will first check if the application has registered an [OnTouchListener](https://developer.android.com/reference/android/view/View.OnTouchListener) on itself.
If a listener exists (and the view is not disabled), the `onTouch` method is invoked with the current `View` and `MotionEvent` as parameters [on this line](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;l=15856;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d).
**This is where an application would be notified of a `MotionEvent` occurring in normal circumstances.**
```java
// (Code from the linked section in View.java above, comments added)
// check if a touch listener has been attached to this View object
// and that the View is not currently disabled. If these tests pass, the
// onTouch method of the listener is invoked
ListenerInfo li = mListenerInfo;
if (li != null && li.mOnTouchListener != null
&& (mViewFlags & ENABLED_MASK) == ENABLED
&& li.mOnTouchListener.onTouch(this, event)) {
// the listener returned true to indicate it has consumed the event
result = true;
}
// if the listener doesn't consume the event (or no listener exists)
// the event is offered to the View's default onTouchEvent method,
// which also returns true if it consumes the event.
if (!result && onTouchEvent(event)) {
result = true;
}
```
The boolean return value of each `OnTouchListener.onTouch` method is used to to indicate if the event was consumed by the listener. If the listener does not exist or does not consume the `MotionEvent`, the fallback option is to instead call `View::onTouchEvent` ([source](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/View.java;l=15860;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d)). This method can be overridden by subclasses of `View`. The default implementation does a significant amount of event handling. Some potentially interesting parts include:
* Handling of [delegate views](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/TouchDelegate.java;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d;l=37)
* Updating the pressed/unpressed state of the `View`
* Generating click events (which will trigger any `OnClickListeners`)
* Checking if a "move" event has triggered a long press (including some allowance for touch "slop" around the bounds of the view)
For the other types of touch events not handled by the above code (i.e. not just `DOWN` events), this [subsequent block](https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/view/ViewGroup.java;l=2788;drc=3d19fbcc09b1b44928639b06cd0b88f735cd988d) in `ViewGroup::dispatchTouchEvcent` takes care of dispatching them to existing targets:
* If the linked list of current `TouchTargets` is empty, the event is delivered to the `ViewGroup` itself where the default method implementations will handle it.
* Otherwise the list of targets is iterated over and the current event is delivered to each in turn through `dispatchTransformedTouchEvent` as above.
### References
* https://cs.android.com
* https://developer.android.com/reference/android/view/package-summary
* https://developer.android.com/reference/android/app/Activity
* https://newandroidbook.com/files/AndroidInput.pdf
* https://utzcoz.github.io/2020/05/06/Analyze-AOSP-input-architecture.html