Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
asks OpenAI Third-party contractors upload real jobs and assignments from their current or previous workplaces so they can use the data to evaluate the performance of their next generation Artificial intelligence modelsAccording to records from OpenAI and training data company Handshake AI obtained by WIRED.
The project appears to be part of OpenAI’s efforts to create a human baseline for various tasks that can then be compared to AI models. In September, the company launched a new product evaluation A process for measuring the performance of its AI models against human professionals across a variety of industries. OpenAI says this is a key indicator of its progress toward achieving artificial general intelligence, or an AI system that outperforms humans at most economically valuable tasks.
“We’ve hired people from different professions to help collect real-world tasks similar to the ones you do in your full-time jobs, so we can measure how well the AI models perform on those tasks,” says one confidential document from OpenAI. “Take existing chunks of long-term or complex work (hours or days+) that you’ve done in your career and turn each one into a task.”
OpenAI asks contractors to describe tasks they’ve done in their current or past jobs and upload real-life examples of work they’ve done, according to an OpenAI presentation about the project seen by WIRED. Each example should be “a tangible output (not a summary of the file, but the actual file), e.g., Word doc, PDF, Powerpoint, Excel, image, repo,” the presentation notes. OpenAI says people can also share fabricated working examples they’ve created to illustrate how they would realistically respond in specific scenarios.
OpenAI and Handshake AI declined to comment.
Real-world tasks have two components, according to the OpenAI presentation. There is the task request (what a person’s manager or colleague has asked them to do) and the deliverable (the actual work they have produced in response to that request). The company emphasizes several times in the instructions that examples shared by contractors should reflect the person’s “real on-the-job work”In reality finished.”
One example in an OpenAI presentation shows an assignment from a “senior lifestyle director at a luxury concierge firm for high-net-worth individuals.” The goal is to “prepare a short, 2-page PDF draft overview of a 7-day yacht trip to the Bahamas for a family who will be traveling there for the first time.” It includes additional details regarding the family’s interests and what the itinerary should look like. The “experienced human director” then shows what the contractor will upload in this case: an actual Bahamas itinerary created for the client.
OpenAI instructs contractors to delete company intellectual property and personally identifiable information from the work files they upload. Under a section titled “Important Reminders,” OpenAI asks workers to “remove or anonymize any: personal information, proprietary or confidential data, or material non-public information (e.g., internal strategy, unpublished product details).”
One of the files shown by the WIRED document refers to a ChatGPT tool called “Superstar peeling” which provides tips on how to delete confidential information.
AI labs that receive confidential information from contractors on this scale could be subject to trade secret misappropriation claims, Evan Brown, an intellectual property attorney at Neal & McDevitt, tells WIRED. Contractors who provide documents from their previous workplaces to an AI company, even if they have been cleared, may be at risk of violating their former employers’ non-disclosure agreements, or exposing trade secrets.
“The AI Lab places a lot of trust in its contractors to decide what is classified and what is not,” Brown says. “If they let something through, do AI labs take the time to determine what is a trade secret and what is not? It seems to me that the AI lab is putting itself at great risk.”