Higharc's Vaidas Razgaitis on ML Production

Vaidas Razgaitis, a Senior Engineer at Higharc, discusses the critical challenge of translating cutting-edge machine learning research into tangible, production-ready features. He emphasizes that this transition is not merely a technical hurdle but a complex interplay of systems and processes.

The Research-to-Reality Challenge

Razgaitis outlines three core problem areas in bringing ML research to production: making research legible, creating a modular codebase that can readily incorporate new research, and treating decomposition as a design problem. He highlights that a research paper's output, while scientifically sound, often lacks the structure or clarity needed for direct implementation by production engineers.

He notes that the gap between researchers and production engineers is significant. Researchers typically focus on novel algorithms and papers, while production engineers need robust, maintainable, and deployable code. This divergence necessitates a deliberate strategy to bridge the two worlds.

Key Strategies for Productionization

Razgaitis proposes a three-pronged approach to tackle this challenge:

• Make Research Legible: This involves ensuring that research outputs are understandable and actionable for engineers. A clear handoff process, potentially including project maps, is essential so that engineers can grasp the scope and requirements of the work.
• A Monorepo for New Research: Razgaitis advocates for a monorepo structure where modular code and templates are maintained. This allows for the efficient integration of new research findings, providing a clean foundation for prototypes and subsequent development.
• Decomposition as a Design Problem: Projects must be decomposed in a way that facilitates iterative development and review. This means planning for future integration, ensuring that initial prototypes can be built without introducing significant risks or dependencies that hinder progress.

He references a blog post by the pragmatic engineer that emphasizes the importance of writing things down, suggesting that clear documentation and specifications are vital for scaling engineering teams and ensuring consistent quality.

Understanding the ML Engineering Workflow

The presentation delves into the structure of Higharc's ML projects, illustrating a layered architecture within a monorepo. This architecture includes an AI Gateway that routes requests, handling JWT authentication and directing traffic to appropriate backend microservices. These microservices, built as isolated Docker containers, communicate via a bridge network, allowing for service resolution by name rather than IP address.

Razgaitis details the responsibilities of each layer: the API layer handles requests and responses, the service layer manages business logic and external services, and the data layer handles persistence and database interactions. He also touches upon the importance of defining clear interfaces and data schemas, often using Pydantic, to ensure seamless communication between these components.

The workflow involves leveraging Python for the core ML logic, utilizing Jupyter notebooks for experimentation and computation, and employing GitHub Actions for automated testing, linting, and deployment. This structured approach aims to streamline the process of taking research from experimentation to production-ready code, ensuring that models are not only functional but also maintainable and scalable.