SurveyForge

A specialist-gated bottleneck, not a capacity problem.

At the AI-first brand research team where I was seconded, survey design was a three-week, specialist-gated process. Every client delivery waited for the survey team to translate a brief into a methodologically-sound questionnaire. The problem wasn't capacity — it was expertise. Designing a good survey requires methodology knowledge most people don't have: question order effects, Likert scale construction, loaded-question avoidance, branching logic, attention-check placement.

The obvious AI opportunity was to pave that path — let non-experts produce specialist-quality surveys. The obvious wrong answer was equally visible: an LLM chat UI that "helps you write survey questions" would produce the wrong outputs at scale, because the rules a good survey follows are not things LLMs stay consistent with when the user doesn't know what to ask for.

Three layers: intake, methodology, generation.

The tool runs in three internal stages:

1. Structured intake. Eight to ten questions about the research objective, audience, decision context, and target analysis plan. Required fields are required — you cannot proceed without declaring what you're measuring. This is where non-specialists get scaffolding a specialist would provide in person.
2. Methodology guardrails. A constraint layer that encodes survey design rules — question order effects, Likert construction, balanced response scales, avoiding double-barrelled items, matrix-vs-single-select decisions, attention-check placement. These rules live outside the LLM prompt and get enforced on the output.
3. LLM generation. Only at this point does a language model produce the actual questionnaire, given the intake and constrained by the methodology layer. The prompt includes the research context and the specific rules the output must satisfy.

After generation, the output is scored against the methodology rules before the user sees it. Failures get flagged. A user can override scores — but the tool makes them do so deliberately, with a note recording why. That's the honesty-loop that makes the tool trustworthy to the specialist team: anything non-specialist-produced is auditable.

The key technical decision: keeping the methodology layer outside the LLM prompt, not encoded as prose in the system message. Rules as code (a validator) are enforceable; rules as prose instructions (a prompt) drift when the model is creative.

The hardest part isn't the AI.

The product-level lesson worth keeping: the hardest part of an AI product isn't the AI. It's deciding which rules the AI isn't allowed to break.

If I'd started with "build the LLM pipeline," I'd have shipped an unreliable tool that produced plausible-looking but methodologically-wrong surveys — the worst failure mode, because the user can't tell. Starting with "what rules does a good survey follow" made the build tractable, the output defensible, and the adoption possible. The domain expertise was the product; the LLM was the execution layer.

The related lesson, which I'd now apply to any AI-product decision: if you can't write the quality rules down as code, you shouldn't be building the tool yet. If they're only in your head, you don't understand the problem well enough to constrain it.