OpenAI Codex Gets Smarter, Broader

OpenAI is rolling out a significant update to its Codex AI model, aiming to transform it into a more comprehensive partner for the millions of developers who rely on it weekly. This latest iteration allows Codex to interact with your computer, utilize a wider array of tools, generate images, and learn from past interactions, according to OpenAI News.

The enhanced Codex update now enables the AI to operate applications directly, seeing and typing with its own cursor. This allows for parallel agent work on macOS without interrupting user activity, proving beneficial for iterating on frontend changes or interacting with tools lacking APIs.

Codex Beyond Code

Codex's capabilities are extending natively to the web with an integrated in-app browser. Users can now provide precise instructions by commenting directly on web pages, streamlining frontend and game development workflows. The model can also leverage GPT-4 Turbo Vision to generate and refine images within the same workflow, aiding in product concept visualization and mockups.

Over 90 new plugins have been released, integrating tools like JIRA, CircleCI, and Microsoft Suite to broaden Codex's context-gathering and action-taking abilities.

Full Lifecycle Support

The update introduces deeper support for the software development lifecycle. Developers can now address GitHub review comments, manage multiple terminal tabs, and connect to remote development environments via SSH in alpha. Rich previews for various document types and a summary pane for tracking agent plans further streamline the development process.

Persistent Context and Proactive Assistance

Codex can now carry work forward over time by reusing conversation threads and preserving context. It can schedule future tasks and autonomously resume long-term projects. A new memory feature allows Codex to retain user preferences and learned information, speeding up future tasks and improving quality.

The AI proactively suggests work based on project context, connected plugins, and memory, helping users prioritize tasks by identifying open comments, pulling relevant Slack or Notion context, and presenting actionable lists.

Availability

These updates are rolling out to Codex desktop app users signed in with ChatGPT, with personalization features and expanded computer use capabilities coming soon to Enterprise, Edu, and EU/UK users. Computer interaction is initially available on macOS.