Bad survey responses: how to detect speeders, straight-liners, and duplicates

The dirty secret of survey research

Somewhere between 5 and 30 percent of the responses in a typical online panel study are junk. Not maliciously fabricated, mostly: just people clicking through as fast as possible to collect an incentive, bots that slipped past the panel's own screening, or the same person entering twice through different links. If you run analysis on the raw file, that junk does not average out. Speeders cluster on the first response option, straight-liners inflate correlations between grid items, and duplicates double-weight whatever opinions they carry. Cleaning is not optional, and it should not be improvised after the data arrives.

1. Speeders: too fast to have read the questions

Reading a question, considering it, and answering takes time. A respondent who finishes a 12-minute survey in 3 minutes did not do those things. The standard practitioner threshold is a fraction of the median completion time: flag anyone below 40 to 50 percent of the median, and treat anyone below a third of the median as presumptively invalid.

Two refinements matter. First, use the median, not the mean: completion times are heavily right-skewed by people who left the tab open over lunch. Second, per-question timing beats total duration when you can get it, because some speeders slow down for the screener (where they know they are being watched) and sprint through the body.

2. Straight-liners: the same answer down every grid

Straight-lining is picking the same scale point for every row of a matrix question. Some of it is legitimate (a genuinely delighted customer may rate everything a 5), which is why the signal is variance across a whole grid, and across multiple grids, not a single flat row. A respondent who straight-lines one grid is unremarkable. A respondent whose answers show near-zero variance across three separate matrices, including one with reverse-keyed items, is not giving you data.

Reverse-keyed items are your best trap here: if someone strongly agrees that the product is "easy to use" and also strongly agrees it is "difficult to learn," the flat line is exposed as inattention rather than enthusiasm.

3. Inconsistent answers: contradictions the respondent cannot explain

Logical consistency checks compare answers that constrain each other. A respondent who says they have no children and later describes their youngest child's school. Someone who selects "never purchased" in the screener and then rates their purchase experience. An age that does not match a stated graduation year. Each contradiction alone might be a misread; two or more in one interview is a strong signal the respondent is not reading.

Design these checks in before fielding. The classic instrumented version is the attention check ("select the third option to show you are reading"), but natural cross-question consistency is harder for professional survey-takers to game, because they cannot recognize it as a trap.

4. Duplicates and near-duplicates

Exact duplicates (same person, same answers, submitted twice) are easy: match on token, device fingerprint, or identical answer vectors. The harder and increasingly common case is the near-duplicate: two submissions with almost identical closed-ended answers and lightly reworded open ends. This pattern shows up with respondents re-entering through multiple panel routers, and lately with bots paraphrasing their own output. Detection means measuring similarity across the full answer vector and across open-text responses, and flagging pairs above a high similarity threshold for human review.

Why panel data specifically needs this

Panel respondents are compensated per complete, which creates a rational incentive to maximize completes per hour. Panels do their own quality enforcement, but their checks run at the panelist level and across studies; only you can check consistency within your questionnaire. Most panel providers also let you reject and replace flagged completes within a reconciliation window, so cleaning promptly has direct financial value: cleaning a week after field close usually means paying for the junk.

Build the workflow before you field

• Pre-register your rules. Decide thresholds (speed cutoff, straight-line variance floor, number of consistency failures that disqualify) before seeing the data. Cleaning rules invented after looking at results are indistinguishable from cherry-picking.
• Score, then decide. Use multiple weak signals rather than one hard rule. A respondent who trips one check gets flagged; one who trips three gets excluded.
• Flag first, exclude reversibly. Keep excluded cases in the file with a status marker so the decision can be audited and reversed. Never delete rows.
• Report the cleaning rate. If you excluded 12 percent of completes, say so in the report. It is a credibility marker, not an embarrassment.

A cleaning rule you can state before fielding is methodology. The same rule applied after peeking at the results is bias with a spreadsheet.

Doing this without a nightly script

Most teams run these checks in a spreadsheet or an R script after export, which means they run late, inconsistently, and only when someone remembers. Sinova360 runs the whole battery server-side across every response as it arrives: speeder detection against the survey's own timing distribution, straight-line variance scoring across grids, cross-question consistency checks, and near-duplicate detection over full answer vectors. Each check produces a flag rather than a silent deletion, and the response grid filters let you view, exclude, or re-include flagged respondents in one step, so your analysis and your exports reflect the cleaned set while the audit trail stays intact. The thresholds are yours to set; the point is that they get applied uniformly, to every response, from the first complete onward.

However you implement it, the principle is the same: data quality is a fielding-time process, not an analysis-time apology.