Earlier in 2026, Highline Beta conducted a client project, part of which explored a variety of concepts with US nurses. We decided to try to see how well we could replicate those findings using Candor's synthetic nurses. We set the pass/fail threshold ahead of running the synthetic studies based on agreement baselines between human nurses from our sample.
This is the first article in a series on how Candor's synthetic research results compare with results from human participants. It will answer the question we get asked by a lot of prospects: How well does Candor represent answers you would get from humans? This is only data from one study. We'll be running more of these tests with different study types and samples to generate a representative number for the system overall, but take this as early evidence. Although, full disclosure, we're not necessarily advocating for these numbers as the test of synthetic data's value. We're publishing them because we're getting asked and the industry has decided to use this as proof of value. For now. We actually believe other tests are more important when evaluating synthetic research tools as complements to human research. You can read our POV on that here.
Even though we built Candor and are invested in the outcomes of these tests, we want to report our findings as objectively as possible. Synthetic research is an emerging industry, and part of establishing where it fits rests on intellectual honesty. We applied these standards to the following tests by defining success and failure criteria before running any statistical tests, using blind AI raters, and committing to reporting unfavourable results. We're also using this information to learn how to iterate on the platform to make it more valuable to its users.
What we tested
The concept test results we tried to replicate with Candor were part of a broader program of research that started from ground-zero problem discovery.
The human benchmark
From the problem discovery interviews we conducted with human nurses, we ideated a host of solutions to those problems and turned them into concept statements to test with a second, independent sample of human nurses. We came out with 21 concepts across three broad categories, which is far too many to individually test in one-hour interviews. Instead, we ran it as a card sort exercise. This shows each concept as a short, high-level statement, and asks participants to sort them into piles (e.g. very interested in using, somewhat interested in using, not interested in using). We then deep-dived into the concepts sorted into the very interested or somewhat interested piles, depending on how many were in each pile.
Synthetic application
Candor was built to only enable testing up to 12 concepts at a time, so we selected one of the subcategories of concepts from the human interviews (tools) to test with the synthetic nurses. For each human nurse, we have data on how they sorted the 6 tool concepts and a transcript of the reasons why.
To run this replication test, we then built synthetic nurses on Candor using the same screening rules the recruiter used to recruit the human nurse sample. When building the Candor nurses, we uploaded documents from the project to ground the synthetic participants in relevant data:
- problem discovery interview synthesis
- 18 interview transcripts
- client's diary study results
- client's survey results
None of these documents included any indication of nurse preferences for the tools we tested with the human nurses, all of which were ideated after the uploaded documents were created.
Candor generated 16 synthetic nurses grounded in web search data (400 signals from 188 sources) and the uploaded documents (2155 signals). The system generates between 15 and 24 participants per audience, and the number depends on how many unique participants the uploaded documents and web search data can support. How evidence grounding works explains how those signals turn into participants.
Next we ran the card sort concept test on the 16 synthetic nurses. The concept descriptions were identical to the ones shown to the human nurses. Candor wrote the interview questions and ran the interviews, but was set up to mimic the human interviews as closely as possible. Because we built Candor to automatically transition to card-sort mode when testing 5 or more concepts, the format matched the format we used with human nurses.
The main discrepancy with the human nurses was that they sorted 21 concepts across three categories rather than just six tool concepts. The tool card sort was a separate card sort for the human nurses though, so the concepts were not mixed together in the human test. As such, we were able to extract the independent tool card sort from the human transcripts as a comparison.
How we set the pass/fail threshold
The first question we asked in order to establish a threshold to use as a pass/fail benchmark was how well do human nurses actually agree with other human nurses on this task?
For the card sort
Before we ran any synthetic interviews, we took the sample of 12 human nurses and split them into two groups of six. We compared the groups of six to determine how closely their results matched, one concept at a time. A score of 0 meant the two groups sorted the concepts completely differently. A score of 1 meant they sorted them identically. All numbers here are those scores turned into percentages. We repeated that for all 462 possible splits across the 12 nurses. Human nurses scored 68.8% on average across all 462 splits. The number established how much human nurses agreed with each other, which we adopted as our baseline.
We opted for a fail threshold below 58.3%, which represents the score only 1 in 20 human nurse pairings fell below. We did not hold synthetic nurses to the 68.8% bar that human nurses set because human nurses consistently disagree with each other, and this was just one sample. Another sample of nurses may have agreed at a higher or lower number. We'll be further testing this in future validation studies.
Setting a 58.3% pass threshold means our standard is that a group of six synthetic nurses passes if it agrees with the human nurses at least as well as the worst-matched pairs of human nurses agreed with each other.
For the sort reasoning and qualitative theme identification
We ran the same exercise on the qualitative reasons the human nurses provided for their sort decisions. For each of the 462 ways to split the 12 human nurses, we asked how many of one group's reasons the other group also raised. We did this by having a Claude session read the 12 transcripts and extract every reason, objection, and "I'd use it if..." condition human nurses gave, before any synthetic interviews had been conducted. 64 reasons were catalogued into a codebook. Then we ran the same split-in-two exercise on that list to see how much of each other's reasons two groups of human nurses would cover.
Two groups of six human nurses covered 75.3% of each other's reasons on average. We opted for a fail threshold below 68.9%, the score only 1 in 20 human pairings fell below. This was the same percentile threshold as the card sort.
Method and results
Card sort results
After Candor ran the synthetic interviews, we input each synthetic nurse's card sort placements into Claude, then compared every possible group of six synthetic nurses against every possible group of six human nurses. In total, we ran 7,399,392 comparisons. Each comparison produced an agreement score between 0-1. The average score across all those comparisons was .603, meaning that synthetic nurses overlapped with human nurses' concept sorts 60.3% of the time, on average. This compared to human-human nurse agreement at 68.8%, and passed our 58.3% threshold.
| Measure | Human vs human | Synthetic vs human |
|---|---|---|
| Comparisons | 462 | 7,399,392 |
| Mean | 68.8% | 60.3% |
| SD | 7.4% | 6.0% |
| 5th pct | 58.3% | 50.0% |
| Median | 69.4% | 61.1% |
| 95th pct | 80.6% | 69.4% |
Qualitative reasoning results
Scoring the rationale was more complex because it involved reading each transcript and coding which reasons from the fixed human nurses' codebook were also present in the synthetic nurse interviews. There were 64 unique themes identified from the human nurses. To determine overlap, we blinded the transcripts to hide which ones came from human versus synthetic nurses by converting every transcript to the same plain format: a speaker label on each question and response, names replaced with codes, disfluencies and timestamps removed.
Two AI models from different companies then read each of the transcripts and marked which reasons appeared with a yes or no verdict for all 12 human nurses and all 16 synthetic nurses. We ran one fresh transcript per session with no labels attached. Everything was run in chats with no memory (Claude Incognito and ChatGPT Temporary Chat).
Once all of the transcripts were coded for the presence or absence of the 64 themes, we split the groups into six synthetic nurses and six human nurses. For each group, if a reason was raised, it counted for the whole group. For instance, if one nurse in a group of six said the tool felt like a sales pitch, that reason went on the group's list, even if the other five never mentioned it. The list is what those six nurses said between them. We then compared the list of reasons generated by each group (one synthetic group versus one human group) to determine what share of the human group's reasons the synthetic group also raised, and vice versa.
More detail on the comparison. If a group of six synthetic nurses raised 20 reasons, a group of six human nurses raised 40, and 18 of them are shared, we looked at it from two directions: coverage and precision.
Coverage refers to how much of what the human nurses raised was also raised by the synthetic nurses. Low coverage means the synthetic participants missed reasons raised by the human participants. We calculated this as a percentage (i.e. if the synthetic group covered 18/40 reasons it was recorded as 45%).
Precision refers to how much of what the synthetic nurses raised was covered by human nurses. Low precision means the output contains material that the human nurses did not mention. We also calculated this as a percentage (i.e. if the human nurses raised 18/20 of the reasons raised by synthetic nurses it was recorded as 90%).
The score for that comparison is the average of the two.
We ran the same 7,399,392 cuts of the synthetic versus human sample again and averaged across all those comparisons. Each comparison produced an agreement score between 0-1. The average score across all comparisons was .757 (averages across both AI coders), meaning that synthetic nurses' and human nurses' reasons overlapped 75.7%, on average across all comparisons. This compared to human-human nurse agreement at 75.3%, and passed our 68.9% threshold.
| Measure | Human vs human | Synthetic vs human (ChatGPT) | Synthetic vs human (Claude) |
|---|---|---|---|
| Comparisons | 462 | 7,399,392 | 7,399,392 |
| Mean | 75.3% | 75.9% | 75.5% |
| SD | 3.8% | 4.2% | 4.2% |
| 5th pct | 68.9% | 68.8% | 68.3% |
| Median | 75.8% | 76.2% | 75.6% |
| 95th pct | 80.6% | 82.5% | 82.2% |
At face value, that's a 1:1 match, but when digging into the two different component metrics, it tells a slightly different story.
Because we were initially thinking about them as a combined score, we didn't set a pass threshold for coverage and precision ahead of time until we decided to separate them in analysis. The 68.9% threshold applies to the combined score, not to either component on its own. Compared against human nurses one direction at a time, the equivalent threshold is 65.4%, which sticks to the same 5th percentile range we used as the criteria for the predetermined thresholds.
Synthetic nurse precision was actually higher than human-human precision. 85.9% of what the synthetic nurses raised also showed up in the human transcripts. That's well above the 75.3% human nurses overlapped with each other. This means the synthetic nurses said fewer unusual things than human nurses did. About a quarter of what any six human nurses said didn't come up with another six human nurses, because humans raise idiosyncratic things.
In contrast, synthetic nurses' coverage was lower than human-human coverage. The synthetic nurses raised 66.0% of what the human nurses said under one coder and 65.1% under the other, compared to 75.3% for human nurses. One coder rated just over our directional threshold of 65.4% and the other just under.
Lower coverage means the synthetic nurses identified fewer reasons than the human nurses. It wasn't that in the comparisons, the synthetic nurses raised too few reasons. Individually they actually raised more reasons than the human nurses did, about 20 each versus 15. However, they raised the same reasons as each other. Six human nurses produced ~51 of the 64 reasons on our list. Six synthetic nurses produced ~39.
Five reasons were raised by at least three human nurses and by almost none of the synthetic ones. One of those reasons came up with more than half the human nurses and never came up once across all 16 synthetic interviews.
| Measure | Human vs human | Threshold | Synthetic vs human (ChatGPT) | Synthetic vs human (Claude) |
|---|---|---|---|---|
| Coverage | 75.3% | 65.4% | 66.0% | 65.1% |
| Precision | 75.3% | 65.4% | 85.9% | 85.9% |
Summary
Candor's synthetic nurses passed both tests we set in advance, and passed one test we came up with mid-analysis according to one AI-coder, but failed according to the other. We're chalking that up as an inconclusive result on the coverage metric.
On the card sort, synthetic participants overlapped with the human nurses at 60.3%, against a human-human baseline of 68.8% and a pass threshold of 58.3%. That means the synthetic nurses sorted the concepts less like the human nurses than the human nurses sorted them like each other, but still inside the range human nurses produce.
In terms of the reasons provided for the sort decisions, the synthetic nurses overlapped with the human nurses at 75.7%, against a human baseline of 75.3%. When we dug in more deeply, synthetic nurses raised 65.6% (averaged across two AI raters) of the reasons human nurses raised, against a human baseline of 75.3%. Precision across synthetic nurses was higher than human nurses, with 85.9% of the reasons synthetic nurses raised also appearing in the human transcripts, against the same 75.3% human baseline. This means that synthetic nurses rarely raised reasons that human nurses did not.
What this means for Candor users
Don't pick concept winners from a small sample, whether human or synthetic
On the card sort, synthetic nurses overlapped with human nurses 60.3% of the time, which passed the threshold we set, but still falls short of the 68.8% human nurses overlapped with each other.
Importantly, 68.8% of human nurse overlap just demonstrates that nurses aren't a monolith and that 12 people is not enough to decide what to build, regardless of whether they're synthetic participants or human participants.
The implication isn't that synthetic nurses are worse at picking the right concept, it's that we still don't know what the right concept is, and we can't know based on a small sample of interviews. What it can tell you is which ideas are worth further exploring, but this is where researcher and contextual judgement always come in. Even if a concept tests poorly (in synthetic OR human interviews), judgement can override that result and you can decide to pursue a concept anyway if there's another reason to have conviction.
If you need a ranking you can defend, ideally you run a sufficiently powered quantitative survey with a sample large enough to generalize. The qualitative round, synthetic or human, helps work out which concepts go into the survey and what to ask about them. When concept testing with synthetic users works covers where a synthetic concept test fits in that sequence.
Synthetic participants won't raise all the rationale raised by humans
In this example, synthetic nurses raised 65.6% of what human nurses raised, compared to human nurses overlapping with each other 75.3%.
Seventeen of the 64 reasons raised by human nurses did not come up in the 16 synthetic interviews. This is because certain reasons just weren't surfaced. 5/64 were raised by at least three human nurses. 1/64 was raised by seven of the twelve.
More synthetic participants are unlikely to fix that, because the problem is that synthetic nurses' reasons overlapped with each other more than human nurses'.
What the synthetic sample did well
85.9% of what the synthetic nurses raised also showed up in the human transcripts, well above the 75.3% human nurses manage with each other. Synthetic nurses rarely covered ground human nurses hadn't. If you read a Candor transcript, most of what's in it is the kind of thing human nurses said. That makes the output useful early in a project, when you're working out what objections to expect and what to put in front of people.
How to work around the gap
We're still deciding what this means for iterating on Candor (or not). One option is to push the interviewer to probe further on each concept to see if that surfaces more of the reasons provided by human nurses. This is something that naturally occurs when testing fewer than 5 concepts, because it tests each one individually and doesn't default to the card sort methodology. That follow-up is on our list of tests to run.
Limitations
This is only one validation study. We're planning to replicate this process on a number of different audiences, study types, and research questions to ultimately understand where synthetic participants fall short of replicating human participants and whether we can iterate on Candor to close that gap.
Twelve human nurses is a small sample size, which makes the pass/fail threshold and the verdict less precise than if we had a larger human sample.
The concept tests weren't identical. The human nurses sorted the tools either before or after sorting two other sets of concepts. The synthetic nurses only sorted the tools. We don't know how that may have impacted either human nurses' responses or synthetic nurses' responses.
The transcript blinding failed. Before comparing, we had one AI model run a test to see whether it could tell human transcripts from synthetic transcripts using nothing but the text. It was able to tell the synthetic transcripts 100% of the time, because the synthetic transcripts were longer and more polished. We assumed this is partly an artifact of the reality that the human transcripts were portions of longer transcripts from the full 21-concept concept test, whereas the synthetic transcripts spent more time on the six concepts. The models scoring the reasons didn't explicitly know that the transcripts were synthetic versus human, but could probably tell.
What comes next
We'll keep testing and publishing these types of validation studies, because it's what customers and the market are asking for. However, we think there are other more important metrics we should be evaluating synthetic output against. Our validation studies will focus on both threads, including how uploading documents versus not affects the audience and the interview quality; what synthetic audiences raise that human audiences may not; how distinct the participants are and how that plays out in the interviews; etc.
Where to go next
For the limits of synthetic research in general, read what synthetic research skeptics get right. For how to judge whether a synthetic transcript holds up, read how to spot convincing nonsense. And to run a test like this one yourself, see concept testing with Candor.