Marlene Mhangami: Playwright for Functionality Testing

Marlene Mhangami, a Senior Developer Advocate at both Microsoft and GitHub, recently delivered a presentation titled "Beyond Code Coverage: Functionality Testing with Playwright." Mhangami, who works within the Core AI group focusing on developer productivity, shared insights into how AI can be integrated into software development workflows, particularly in the realm of testing.

The Role of Functionality Testing

Mhangami began by setting the stage, emphasizing that while code coverage is a common metric, it doesn't guarantee that software functions as expected from a user's perspective. She highlighted the increasing volume of code being created, citing GitHub's Octoverse report which showed over a billion commits in 2020 and projected growth to 14 billion by 2025. This surge in development underscores the need for robust testing strategies.

A key theme of the presentation was the impact of engineering environments on AI-assisted development. Mhangami presented a slide that suggested a strong correlation between a clean engineering environment and increased AI productivity gains. She explained that AI thrives in environments with good test coverage, modularity, and well-written code, allowing it to more effectively assist developers in completing tasks and improving software quality.

AI's Impact on Developer Productivity

The presentation explored the question of whether AI truly makes developers more productive. Mhangami presented a case study that illustrated how unchecked AI adoption, without a focus on clean code, could lead to increased entropy and a decrease in code quality, despite a rise in pull requests. This highlighted the importance of a structured approach to integrating AI tools.

Conversely, she suggested that when AI is used effectively within a clean, well-maintained codebase, it can amplify productivity. This involves leveraging AI for tasks like generating tests, writing code, and even refactoring. Mhangami touched upon the concept of Test-Driven Development (TDD), explaining the typical Red-Green-Refactor loop and noting that AI can assist in each stage, from generating initial failing tests to refactoring code to meet requirements.

Playwright for Functionality Testing

Mhangami then introduced Playwright, an open-source testing framework developed by Microsoft. She described Playwright as a tool that automates end-to-end testing in the browser by simulating user interactions. Playwright supports multiple programming languages, including Python, TypeScript, and C#, and can run tests in both headed and headless modes. The demo showcased how an AI agent could be used to interact with the Playwright CLI to generate and run tests based on feature requests.

The demonstration involved a scenario where an AI agent, provided with a feature request email, was able to identify the necessary tests, write Playwright scripts to execute them, and report on their success. This illustrated the potential for AI to significantly speed up the test writing process and ensure that features function as intended.

Best Practices for AI-assisted testing

Mhangami concluded by offering best practices for developers looking to leverage AI in their testing workflows:

• Add screenshots to Pull Requests (PRs): This provides visual context for code changes and test results.
• Use headless mode for multi-tasking: Headless mode allows for running tests in the background without opening a browser window, enabling more efficient parallel execution.
• Commit before running the 'Healer': This implies a workflow where code is committed before using AI tools to fix or refactor it, ensuring a stable baseline.
• Generate tests one feature at a time: This approach helps maintain focus and manage complexity when working with AI-generated tests.

The presentation underscored the growing synergy between AI and software development, particularly in ensuring the quality and functionality of applications through effective testing strategies with tools like Playwright.