Project Overview

Join a growing community of professionals advancing the next wave of AI. As an AI Trainer, you’ll play a hands-on role by analyzing and providing feedback on data to improve LLM performance, helping ensure that the next generation of AI technology is accurate and trustworthy.

We are seeking a skilled Operations Manager to work as a project consultant in our AI Labor Marketplace. This is not a full-time employment position — you will be engaged as an expert project consultant on a contract basis.

Location: U.S.-based experts only

Engagement: Part-time, project-based expert evaluation work

Work Type: Remote

Project Summary

This project supports the development of an AI benchmarking framework designed to measure how effectively artificial intelligence models perform real-world, economically valuable knowledge work. The benchmark evaluates AI systems against tasks commonly performed by operations leaders responsible for managing business processes, operational performance, resource allocation, workforce planning, process improvement, and cross-functional execution.

Contributors will create expert benchmark outputs, define evaluation criteria, and assess AI-generated work products across a wide range of operational scenarios. Example tasks may include process analysis, operational planning, performance management, workforce optimization, SOP development, KPI design, root cause analysis, operational risk assessment, change management planning, vendor management, service delivery optimization, and business operations decision-making. The goal is to establish rigorous, expert-driven standards for measuring AI performance on practical operational workflows and management responsibilities.

Consultant Engagement Terms

This is a project-based consultant role. Consultants will be paid on a per-project basis; hourly rates are estimates based on anticipated completion time. Consultants control their own schedule, provide their own tools, and may simultaneously provide services to other vendors/employers (subject to those vendors’ allowances).

Responsibilities

Contributors will:

• Create expert benchmark responses for operations and business process management scenarios
• Develop scoring rubrics that define high-quality operational analysis, planning, execution, and decision-making standards
• Evaluate AI-generated recommendations, analyses, reports, and operational deliverables against established benchmarks
• Assess the accuracy, completeness, practicality, scalability, and business impact of AI-generated outputs
• Review AI-generated work involving process improvement, operational strategy, workforce management, service delivery, performance measurement, and cross-functional coordination
• Identify flawed assumptions, operational risks, process inefficiencies, unsupported recommendations, and material omissions
• Apply professional judgment to determine whether outputs meet real-world operational standards and business requirements
• Provide written rationale supporting evaluation decisions and scoring outcomes
• Participate in calibration exercises and quality review activities when required

Expected Outcomes

• High-quality benchmark responses reflecting operational leadership and business management best practices
• Clear, defensible evaluation rubrics aligned with real-world operational workflows and organizational objectives
• Consistent and reliable assessment of AI-generated operational work products
• Identification of strengths, weaknesses, risks, and performance gaps in model outputs
• Expert evaluations that support rigorous benchmarking of AI systems on economically valuable operational work
• Documentation that enables repeatable and scalable model evaluation

Qualifications

• Minimum 3+ years of professional experience in operations management, business operations, operational excellence, process improvement, or related functions
• Strong understanding of operational workflows, performance management, process optimization, workforce planning, and resource allocation
• Experience managing teams, operational processes, budgets, service delivery functions, or business performance initiatives
• Experience creating, reviewing, or evaluating operational plans, standard operating procedures (SOPs), business processes, performance dashboards, KPI frameworks, or operational reports
• Ability to analyze complex business scenarios and articulate operational reasoning clearly in writing
• Strong attention to detail and ability to identify inefficiencies, execution risks, process gaps, and operational improvement opportunities
• Experience working in corporate operations, manufacturing, logistics, healthcare operations, technology operations, retail operations, professional services, or similar environments preferred
• Relevant certifications are preferred where applicable (e.g., Lean Six Sigma Green Belt/Black Belt, PMP, Certified Manager (CM), APICS/ASCM certifications, Prosci Change Management, or equivalent)
• U.S.-based