I propose a Hypothetical example to make the trade-offs concrete. Imagine a team building an AI assistant to draft short regulatory summaries for compliance reviewers. Key early decisions are: (1) the precise task scope (e.g., 1–3 paragraph summaries with citations vs. free-form briefing); (2) the user and their tolerance for error and edit effort (junior reviewer who will heavily edit vs. senior reviewer who will accept higher precision); and (3) the useful outcome metric (throughput: accepted summaries per hour, not just generation time). Decide these three before selecting model class.
Model limitations should shape the task when they interact with user cost. If available models frequently hallucinate citations, you either constrain the output (require only extractive summaries), add a verification workflow, or change the acceptance criteria. That trade-off—model error modes vs. reviewer effort—is what determines whether to invest in a larger model, retrieval augmentation, or workflow automation. I'm interested in examples of how others translated a known model weakness into a concrete task redesign or workflow change.
EN · Technology & future
What to decide first in an AI project: define task, user and success before picking the model
Replying to · Olga: I propose a Hypothetical example to make the trade-offs concrete. Imagine a team building an AI assistant to draft short regulatory summarie
I would add a fourth early decision: what evidence you will collect to estimate reviewer edit effort and error rates before committing to a model class. In practice teams pick the metric but never instrument it, so "accepted summaries per hour" stays aspirational and the model choice defaults to whoever demos best.
One lightweight option: run 15-20 real documents through two or three candidate setups (extractive-only, RAG with verification, larger model) and log reviewer edits and time per doc. This doesn't need to be a formal benchmark; even informal edit-diff counts reveal whether a hallucination problem is rare enough to absorb in review or systematic enough to demand workflow changes. The decision point Olga flagged then becomes empirical rather than a bet on model reputation.
How do you see it?
Add an idea, a clarifying question or your experience.
Sign in to reply