Question-Analysis Annotation Rubric¶
This rubric documents the per-turn fields stored by projects whose
annotation_type is question_analysis.
Fields per turn (one row per message_id + annotator_id)¶
| Field | Type | Description |
|---|---|---|
label |
string | Free-form turn-grouping label, semantically like a thread id. |
trigger_marker |
boolean | True when the turn is unambiguously question-form (see rubric below). |
borderline |
boolean | True when the case is fronteiriço / ambiguous. |
multiform |
boolean | True when the turn mixes multiple question-form features. |
trigger_marker rubric¶
A turn is question-form if its surface wording is unambiguously interrogative. Mark only forms that cannot reasonably be read as non-questions. Do not mark a turn merely because it could be heard as a question with the right intonation or context.
The check has two steps: the question mark is the primary check, and the no-question-mark features are the secondary band.
Primary check — the question mark¶
If the turn carries a question mark, it is question-form. This is the main, high-frequency path.
Secondary band — no question mark¶
WhatsApp-style group chat often omits punctuation, so a turn with no question mark can still be unambiguously interrogative in its wording. A no-question-mark turn is question-form if it carries one of the five features below. Each feature comes with an exclusion — the exclusion is what makes the feature reliable enough to settle the turn for sure.
-
Bare interrogative fragment. The whole turn is an interrogative pro-form or a short interrogative phrase — "Porquê", "Como assim", "Qual deles", "Em que sentido". No declarative reading exists; no exclusion is needed.
-
Turn-initial interrogative wh-word. A wh-word — quem, o que / o quê, qual, quando, onde, como, porquê, quanto — opens the turn in matrix interrogative use: "Como definimos beleza", "Quem disse isso". Does not count when the wh-word is relative ("a forma como pensamos") or embedded under a verb of saying, thinking, or knowing ("não sei o que pensar").
-
Dedicated interrogative construction. The será que… / não será (que)… frame, or subject–verb inversion where it occurs: "Será que a beleza é universal", "Não será isso cultural". Does not count when será is a plain future declarative ("isso será difícil") — the signal is the será que / (não) será + X frame, not the word será on its own.
-
Disjunctive choice question. The turn presents a choice between alternatives as its point: "Universal ou diversa", "Concordas ou discordas". Does not count when ou is additive or hedging ("ou pelo menos…", "duas ou três").
-
Conventionalised answer-soliciting formula. The turn matches a recognised question formula — "alguém sabe…", "o que acham", "quem concorda", "podem explicar". Keep this a closed list, confirmed against actual DebaQi turns; do not extend it case by case.
Inter-annotator agreement (IAA)¶
For question_analysis projects the IAA combines:
- Label one-to-one accuracy — Hungarian-algorithm matching on the
labelfield, identical to disentanglement projects (0–100). - Trigger / borderline / multiform agreement — percentage of common messages where the boolean matches (0–100).
- Combined IAA — unweighted mean of the four metrics above.
Only messages annotated by both annotators contribute to the per-pair
metric (common_message_count is reported alongside).