OpenAI is rolling out a significant update to its Codex AI model, aiming to transform it into a more comprehensive partner for the millions of developers who rely on it weekly. This latest iteration allows Codex to interact with your computer, utilize a wider array of tools, generate images, and learn from past interactions, according to OpenAI News.
The enhanced Codex update now enables the AI to operate applications directly, seeing and typing with its own cursor. This allows for parallel agent work on macOS without interrupting user activity, proving beneficial for iterating on frontend changes or interacting with tools lacking APIs.
Codex Beyond Code
Codex's capabilities are extending natively to the web with an integrated in-app browser. Users can now provide precise instructions by commenting directly on web pages, streamlining frontend and game development workflows. The model can also leverage GPT-4 Turbo Vision to generate and refine images within the same workflow, aiding in product concept visualization and mockups.
Over 90 new plugins have been released, integrating tools like JIRA, CircleCI, and Microsoft Suite to broaden Codex's context-gathering and action-taking abilities.
Full Lifecycle Support
The update introduces deeper support for the software development lifecycle. Developers can now address GitHub review comments, manage multiple terminal tabs, and connect to remote development environments via SSH in alpha. Rich previews for various document types and a summary pane for tracking agent plans further streamline the development process.