If one day, AI no longer just answers questions, but can actually pick up a mouse, operate a keyboard, and use a mobile phone like a human being, then mobile phone automation will enter a new stage. Open source projects on GitHub ESP32 iPhone Agent We are trying to realize this idea: with only an ESP32-S3 development board that costs about US$7, a computer, and an AI model that supports visual understanding, the AI can observe the iPhone screen and automatically click, slide, and enter text according to the task to complete a series of operations.
The most interesting thing about this project is that it does not take the traditional iOS automation route, but simulates the way humans use their phones. It projects the iPhone screen to the computer through AirPlay, and the program on the computer obtains the picture and submits it to the visual model for analysis; at the same time, the ESP32-S3 connected to the USB interface of the mobile phone acts as a mouse and keyboard, simulating real input operations through USB HID. AI is responsible for observation and decision-making, the computer is responsible for running the agent program, and the ESP32 is responsible for executing instructions. The three work together to form a complete automated closed loop.
In actual use, the user only needs to enter a natural language task, such as "Open the clock application and set the alarm clock to seven o'clock every morning." The agent will plan the operation steps based on the screen content, gradually click the corresponding buttons, set the time and repeat date, and finally save the settings. The project also demonstrates the ability to automatically open a whiteboard application and draw a smiley face. Every step of the operation will leave screenshots and action records, allowing users to observe the execution process, understand what the AI has done, and whether any misoperations have occurred.
Compared with traditional iPhone automation solutions that rely on XCTest, WebDriverAgent, Xcode and developer signatures, the biggest advantage of this design is that the threshold is lower. The phone does not need to install special applications, nor does it need to be jailbroken or configured with an Apple developer account. In theory, it can be compatible with more applications through screen visual recognition and pointer operations, including interfaces that lack standardized automation interfaces. The project adopts the MIT open source license, and developers can study the code, modify functions, and access different visual language models according to their own needs.
From the perspective of technical architecture, ESP32-S3 itself is not responsible for running large AI models, nor is it responsible for complex task planning. The real smart part still runs on the computer or model server, and the development board is only responsible for executing simple mouse, keyboard and scrolling commands. This division of labor reduces hardware costs and gives developers the flexibility to choose a cloud API or a compatible model deployed on-premises. However, low-cost hardware does not mean that the entire system is cost-free. Computers, network environments, model inference resources, and necessary configuration work are still indispensable.
Of course, this technology currently has some limitations. For example, some iOS security interfaces, Apple Pay, and digital copyright-protected video screens cannot be mirrored normally; screen mirroring may also be interrupted after the phone is locked; the current keyboard input mainly supports ASCII characters, and the coordinate adaptation of horizontal screen applications is not yet perfect. In addition, the visual model may misjudge the button position or perform the wrong action, so the project provides execution logs and emergency stop functions, which still require manual supervision when used.
The value demonstrated by ESP32 iPhone Agent is not just to control the iPhone with a cheap development board, but to provide a more general idea of implementing mobile phone intelligence: AI does not have to rely on each application to open a separate interface, but can complete tasks by observing the screen, understanding the interface, performing actions and verifying the results. In the future, if visual understanding, action planning, and device-side computing capabilities continue to improve, this architecture is expected to be used in scenarios such as mobile phone testing, repetitive operations, personal automation, and barrier-free assistance. It is still some distance away from a completely reliable universal mobile phone assistant, but it has already allowed people to see that AI has moved from "telling you what to do" to "doing it for you" and is becoming an engineering practice that can be built and verified by yourself.
Project address:https://github.com/onehumanbeing/esp32-iphone-agent