How it works for AI teams
The samples, asking for a pilot, and then a client's projects: progress and quality, exports and the API, invoices, AI models, and when the data is deleted.
Screenshots are from our demo, with made-up people and companies. Also see how it works for experts.
Contents
- The landing page and the open samples
- Asking for a pilot
- Signing in and creating an account
- Account security: two-factor login and passkeys; Delete my account asks for the account's deletion, done 7 days on unless Keep my account is pressed first (what was earned is still paid)
- Projects, quality and exports
- Starting a project
- Invoices and paying online
- AI models
- Data retention and deletion
- The API and delivery webhooks
The landing page and the open samples
Anyone · canonset.com
The samples are real, verified data a buyer can judge before talking to us: no sign-in, no forms.
1.Landing page
What we deliver, how quality is checked, live numbers (environment tasks, sandbox-verified coding tasks, reward hacks, and open roles or downloadable sample records) and the two ways in: Start a pilot for buyers, For experts for people who want to work.
2.Samples
Four sample sets: verified coding tasks, preference comparisons, rubric grading, and an RL environment. Each opens a viewer and offers the data as a download.
3.Coding tasks with sandbox-verified tests
Each task shows the problem, the starting code, the reference solution and the tests, with the sandbox run showing at least one test failing on the starting code and all passing on the reference solution.
4.Preference comparisons
Two AI answers to the same prompt, the expert's choice and strength of preference, and the reason in a few sentences.
5.Rubric grading
An AI answer graded criterion by criterion. A wrong core claim caps the overall score at 2, however well the rest reads.
6.The expense-approvals environment
A simulated expense-review app for a fictional company, with a written policy and 50 tasks an agent is scored on. Tabs: Overview, Play a task, the 50 tasks, the policy, the agent interface, and how each task was verified.
7.Play a task
Act as the agent: open a report, approve, reject, adjust or flag lines, then submit for grading. Show the solution plays the reference actions.
8.Grading an attempt that did nothing
Submitting without acting fails the checks: graders are built so that doing nothing, approving everything or rejecting everything never passes.
9.Grading the reference solution
The reference passes every check, including the ones that the other reports in the queue are left alone.
Good to know
- Every sample can be downloaded as JSONL; nothing on these pages needs an account.
- The environment runs in the browser: nothing a visitor does is saved.
Asking for a pilot
A buyer · Start a pilot (/pilot)
The only way in for a buyer. Our team answers by email, sets up a call, and creates the client account after the call.
1.The form
How a pilot works on the left; the request on the right: name, work email, company, role, what they need, and details.
2.Filled in
The email field must be a real address shape; the browser checks it before anything is sent. Details are optional and stop at 4,000 characters.
3.Sent
The page names the email we'll write to and the day a person replies (the next business day), and points to the samples and the security overview meanwhile. The buyer gets the same by email. Our team is emailed and the request waits in the ops console's Today, with the sample page the buyer started from when they came through one.
4.If it doesn't go through
Offline, or if our server fails, the form says so in place and keeps everything typed, so the buyer can send it again.
Good to know
- A repeat from the same email within 10 minutes (a double click, a resend) is saved once.
- One network can send 10 requests an hour; past that the form names the limit and when to try again.
- We don't work for organizations headquartered in, or at least 50% owned or controlled from, a country of concern. The form says so; our team confirms it before creating an account.
Signing in and creating an account
Everyone · Sign in (top right), /login, /signup
No passwords anywhere. One button signs in an existing account or starts a new expert account, with a six-digit code sent by email.
1.One button
Enter an email address. The same pop-up serves experts, clients and staff.
2.Ask for a code
Continue with email sends a six-digit code. The answer is the same whether or not the address has an account.
3.Enter the code
Six boxes: typing, pasting the code (spaces and dashes are dropped) or the phone's autofill from the email fills them. The code expires in 10 minutes. An existing account is signed in at once (two-factor follows if it's on).
4.A new address
A valid code for a new address opens one more step: the name, two confirmations (living and working in the United States, and the original-work policy), and agreeing to the terms of service, having read the privacy notice. Only then is the expert account created, and the acceptance kept with its version.
5.A wrong code
Says what to do: check it, or send a new one. A new code can be sent after 30 seconds, and it replaces the older one.
6.A session that ended
When a session expires, or is signed out from another device, the next click lands here with a line saying so. Signing in again returns to the page the person was on.
7.Signed out in another tab
Every open tab hears at once when the account signs out, signs in, or changes, and says so across the top with Reload. It doesn't reload by itself: that would throw away whatever was being typed.
Good to know
- Client accounts are created by our team after the first call; clients sign in the same way.
- Two-factor login (an authenticator app, with backup codes) is optional for everyone, under Account.
- Every change to how an account signs in (a passkey added or removed, two-factor turned on or off, new backup codes) is emailed to its owner, with what to do if it wasn't them. Removing a passkey and turning two-factor off ask once more first.
- A link opened while signed out goes to sign-in, then back to the page it pointed to.
- Signing out in one tab signs out the others: each says so at once, and its next click goes to sign-in. Signing in again in any tab makes the others right again without a reload.
- The header always names the account that's signed in: after a session ends it shows Sign in, never the name and menu of whoever was there.
Account security: two-factor login and passkeys; Delete my account asks for the account's deletion, done 7 days on unless Keep my account is pressed first (what was earned is still paid)
Anyone signed in · the account menu (top right, at every width) → Account security
Signing in needs only the emailed code. Two-factor login adds a code from an authenticator app; a passkey replaces the emailed code with a fingerprint, face or device PIN.
1.The account menu
Beside the role, at every width (initials on a phone): who is signed in, Profile for experts, Account security and Log out. Esc or a click outside closes it.
2.Account security
Passkeys, two-factor login, and signing out of other devices. There's no password to change: none exists.
3.Setting up two-factor login
Scan the QR code with an authenticator app, or type the key, then enter the code the app shows. A wrong code says so and nothing is turned on.
4.On, with backup codes
Ten backup codes, shown once: each works once if the phone is lost. An email tells the account's owner that two-factor login was turned on.
5.The next sign-in
After the emailed code, the authenticator's code. Lost your phone? Use a backup code instead.
6.Turning it off
Asks once more, in place, with the keyboard's focus on Keep it on. Turning it off is emailed too.
Good to know
- Every change to how an account signs in is emailed to its owner: a passkey added or removed, two-factor turned on or off, new backup codes. The email says what to do if it wasn't them.
- Sign out of other devices ends every other session at once; those devices land on the sign-in page with a line saying they were signed out.
Projects, quality and exports
A client · Projects (/client)
Each project shows where the work stands and how good it is, from the first approved answer.
1.Projects
Every project with its status and progress. On a phone each project is a card.
2.A project
Where things stand comes first: how many answers are approved, with reviewers and in progress, the recent pace, whether the deadline holds, and how the client hears when it's done. Then progress once: one bar of the planned answers by state, with the tasks behind them on their own line; accuracy on hidden known-answer tasks, agreement and what's billable so far, each saying what it measures; the latest approved answers; and which AI models may read the project.
3.A paused project
Paused says what it means: experts take no new tasks, approved answers stay ready to export, and an email comes when it resumes. Our team's note about why is in the email.
4.The quality report
For the whole project or one week: delivered answers against the plan, average review score, known-answer accuracy, agreement, rework and rejection rates, flags, check pass rate, time per task, turnaround, and delivery against the deadline.
5.Print or save as PDF
The report prints cleanly: the site's header and buttons drop out.
6.The first batch
A new project starts with a small batch: every expert's first task comes from it, a reviewer checks each answer, and the page shows every decided answer as the export will, with an evidence card per expert (pseudonym, years, tier, what they passed; never a name). When all are decided, the client approves the batch or asks for changes; nothing else starts until then, and the batch is billed after approval.
7.Approved
The rest of the tasks open at once and every expert with work in the project is emailed. A note asking for changes goes to ops, who answer within one business day.
8.The first task as an expert sees it
From the project page: the codename, the instructions and rubric, the first uploaded task, and what the answer form asks for. Checking the wording here saves a round of changes later.
9.Changing the instructions
Edit the instructions (and the rubric) with a line on what changed: each change is the next version, our team is told, and experts see what changed on their next task. An answer started under an earlier version is reviewed under it. Experts' notes on what's unclear show here under pseudonyms, with our team's replies.
10.Returning a delivered answer
All delivered answers, as the export shows them. For 14 days after delivery each can be returned with a reason and a note: a reviewer other than the one who approved it replaces it (the task reopens and what it was billed comes off the next invoice), lets it stand, or has the expert fix it; the client is emailed the decision with the reviewer's note.
11.A coding project
Each approved task is verified by the sandbox (tests that fail on the starting code and pass on the reference), and says how often a model solved it from the statement and starting code alone.
Good to know
- Export JSONL gives approved answers (never the hidden known-answer tasks); coding tasks carry their test results and pass_at_k (the model, attempts, how many solved it), environment tasks the engine's task file. Export preference pairs gives chosen and rejected answers for clear majorities.
- A task's difficulty is one named model in one setting: Claude Opus 5.5, five attempts of one response each without tools, the hidden tests run on each answer. It's measured only where you allow Claude to read the project's answers (AI models).
- Experts appear under per-project pseudonyms everywhere a client looks, and experts see the project only by its codename, shown under the project's name.
- Every export is in our audit log. A coding-task project also exports as a Harbor package (Export Harbor tasks): a zip of runnable task directories, one per approved answer, that harbor run and labs' harnesses load as they are. The project's header says how many approved experts its field has, and the first batch's evidence card carries the note our team wrote on how each expert's credentials were checked. Datasheet, beside the exports, gives the delivery's datasheet: what it is, where it came from, how it was checked, its quality figures, the rights in it and its retention.
- Everything a client sees is read through a database role that only sees their company's rows: a second lock behind the app's own checks.
- Delivery is on track, at risk or late, projected from the pace of the last two weeks with deliveries.
- Everyone at the company is emailed when a project starts (with the price), is paused or opens again, and when it's complete (how to export, and the date its data is deleted). Each email links to the project.
- Our team can add teammates to the company's account; each has their own sign-in and sees the same projects.
Starting a project
A client · New project
Upload tasks and instructions; our team confirms price and staffing before experts start.
1.The form
Name, task type, field, experts per task (2 or 3 gives agreement and majority votes), instructions, a rubric for grading tasks, skills, target volume and deadline.
2.A mistake in the data
Each row is checked against the task type; the message names what's missing and on which row.
3.Personal data in the file
Email addresses, phone numbers, Social Security and card numbers hold the upload: the message says how many rows and where. Remove them, or tick that they're made up or covered by our data processing agreement; our team sees that on the project.
4.Tasks as JSONL or CSV
Paste or upload. Rows with gold answers become hidden known-answer tasks.
5.Awaiting pricing
The page says what happens next and by when: our team agrees the price and picks the experts, usually within two business days, and emails when experts start. Nothing is needed from the client. A confirmation goes to their email, and our team is emailed that there's a project to price.
Good to know
- At most 5,000 rows per upload.
- Closing or reloading the tab with the form half filled asks first.
- Task types: preference comparison, rubric grading, ideal answer writing, coding task authoring, and environment task authoring.
Invoices and paying online
A client · a project's Invoices, an invoice
Invoices come from our team: a deposit when a pilot starts, then invoices for delivered work. Paying online uses Stripe's invoice page.
1.On the project
Issued, paid and voided invoices. Drafts never show.
2.An invoice
Our invoice with its number, bill-to, dates, lines and total, printable as a PDF, with Pay online.
3.Stripe's invoice page
Pay by bank (ACH) or card on Stripe's page.
4.Paid
Stripe's page opens in a new tab. Once the payment goes through, the invoice page left open shows it as paid by itself, and we email a confirmation.
Good to know
- The deposit is 40% of the pilot's estimate by default, due in 7 days, and credited against later invoices. Invoices for delivered work are due in 30 days.
- Without Stripe, we email the invoice ourselves: its lines, the total, the due date and how to pay, to the billing email or everyone at the company. Paid and voided invoices are emailed either way.
- Invoices stay as financial records after a project's data is deleted: they show amounts and the project's name, not tasks or answers.
AI models
A client · a project's AI models
Which AI providers' models may read the project's answers. Never for a company that builds AI models, and otherwise off unless the client turns it on: labs may not want their data going to another lab's models.
1.Off
No AI model reads this project's answers. Our reviewers decide either way.
2.On
With Anthropic allowed, Claude reads each submitted answer to give our reviewers a second opinion, and attempts each verified coding task so you learn how hard it is for the model. The card shows who changed it and when. Only for companies that don't build AI models: for one that does, the card stays off and says why, since the model vendors' terms forbid it even with the client's OK (the demo client is recorded as not building models).
Good to know
- Every change is in the audit log. Our team can change it only on the client's written word, and the client sees who did.
- Experts never see what a model says, and the client doesn't either.
Data retention and deletion
A client · a project's Your data
A completed project's tasks and answers are deleted 30 days after it ends; a client can ask for deletion sooner.
1.Your data
When the project's data will be deleted, and Request deletion behind a confirmation.
2.Requested
It's done within 7 days, and we email a written confirmation.
Good to know
- Database backups age out 7 days after the deletion.
- A request for all of a company's data also removes its accounts, API keys and webhook.
- Asking twice makes one request.
- A week before a finished project's data is deleted, everyone at the company is emailed the date, in time to export.
- If our team cancels a request, whoever asked is emailed that nothing was deleted, and why.
The API and delivery webhooks
A client · API (/client/api)
Pull approved answers into a pipeline, and hear when new ones are ready.
1.The API page
Keys, the webhook, recent deliveries and a reference with examples.
2.A new key
Read-only, for the company's projects. The key is shown once, with a copy button; we keep only a hash. Keys can be revoked at any time.
3.A webhook
One https endpoint and the events: an answer approved, a project completed. Its signing secret is shown once; it can be rolled.
4.Deliveries
Each event sent, its status and the endpoint's answer, with Send again for failures.
Good to know
- GET /api/v1/projects lists the company's projects; GET /api/v1/projects/{id}/export?format=raw|dpo|harbor&since=… returns the export, only answers approved after since; GET /api/v1/projects/{id}/datasheet returns the delivery's datasheet.
- Everything the API returns is read through a database role that only sees the key's company's rows.
- 60 requests a minute per key; past it the API answers 429 with Retry-After. Up to 10 active keys.
- Events carry IDs, never answers. They're signed like Stripe's (Canonset-Signature: t=…, v1=HMAC-SHA256), carry Canonset-Delivery to ignore repeats, and can arrive out of order.
- Failed deliveries are retried after 1, 5, 30, 120 and 720 minutes; the log keeps 30 days. When a delivery gives up after the last retry, the company is emailed (at most once a day) that the endpoint isn't answering.
- Webhook URLs must be https on a public server: our servers never connect to private addresses.