PhoneAgent: An AI-Powered iPhone Assistant Using OpenAI

PhoneAgent is an advanced autonomous agent for iOS built on OpenAI’s architecture. It mimics human interaction by navigating the interface—tapping, swiping, and typing—to execute tasks across various applications.

Capabilities

• Send a selfie along with your weekend itinerary to a specific contact.
• Search for and download any application from the App Store.
• Message flight details directly to a contact.
• Access the Control Center to toggle the flashlight or other settings.

Interaction Methods

• Voice and Text: Control the agent through written instructions or voice commands by tapping the microphone icon.
• Always-On Mode: PhoneAgent can run in the background, listening for a specific wake word (“Agent” by default). For example, saying “Agent, open Settings” will trigger the action immediately.
• Real-Time Feedback: The agent provides status updates via system notifications, ensuring you are kept informed as it progresses through tasks.

Technical Architecture

• Action Management: The model is responsible for calculating coordinates for taps, scrolls, and swipes, as well as handling text input and app initialization.
• System Integration: By leveraging Xcode’s native UI testing tools, the agent interacts with the OS and third-party apps without requiring a jailbreak. A TCP server manages the communication bridge between the host application and the UI test process.

How to Run PhoneAgent

1. Clone the repository to your local machine.
2. Open the project file in Xcode.
3. Navigate to PhoneAgentUITests.swift and run the testLoop function.
4. Provide your OpenAI API key when prompted, then begin issuing text or voice commands.

Operational Logic

PhoneAgent operates within the standard iOS sandbox, utilizing the Xcode UI testing framework to inspect the screen hierarchy and control the interface. The GPT-4.1 model acts as the "brain," driving device interaction through the following capabilities:

• Screen Recognition: Reading and interpreting active UI elements.
• Precision Interaction: Tapping specific buttons and navigating menus.
• Data Entry: Typing directly into text fields and forms.
• App Management: Launching and switching between applications.