contentbring your own workflow

Content moderation

Several independent yes/no facts about one post, each with its own probability.

Moderation is a bundle of separate questions, not one label. Ask them separately and route on the combination. noul questions are the natural fit: each is a statement about the post that is true or false.

Use this as the shape to fit to your own moderation policy: your categories, your thresholds, your escalation rules.

The questions

harassmentnoulis the statement true?

The post harasses, insults or demeans a person or group.

falseno, the statement does not hold

trueyes, the statement holds

threatnoulis the statement true?

The post contains a credible threat of harm.

falseno, the statement does not hold

trueyes, the statement holds

piinoulis the statement true?

The post exposes personal contact information.

falseno, the statement does not hold

trueyes, the statement holds

actionchoicepick one option

What should the moderation system do?

allowLeave it up.

flagLeave it up but flag for a moderator.

redactRemove the sensitive part and keep the rest.

removeTake the post down.

tonescorepick a level on an ordered scale

How hostile is the tone?

0Friendly.

1Neutral.

2Irritated.

3Hostile.

4Abusive.

Acting on the answers

pii true: redact automatically, whatever else is true.
threat above 0.5: a human sees it within minutes; do not auto-remove on the model alone.

request

{
  "state": "Post in #general: 'lol if you idiots merge that again I'm going to lose it. seriously. anyway here's my number 555-0142 if anyone wants to grab a beer after the release'",
  "questions": {
    "harassment": {
      "type": "noul",
      "instructions": "The post harasses, insults or demeans a person or group."
    },
    "threat": {
      "type": "noul",
      "instructions": "The post contains a credible threat of harm."
    },
    "pii": {
      "type": "noul",
      "instructions": "The post exposes personal contact information."
    },
    "action": {
      "type": "choice",
      "instructions": "What should the moderation system do?",
      "criteria": {
        "allow": "Leave it up.",
        "flag": "Leave it up but flag for a moderator.",
        "redact": "Remove the sensitive part and keep the rest.",
        "remove": "Take the post down."
      }
    },
    "tone": {
      "type": "score",
      "instructions": "How hostile is the tone?",
      "criteria": [
        "Friendly.",
        "Neutral.",
        "Irritated.",
        "Hostile.",
        "Abusive."
      ]
    }
  }
}

call it

import { loadEngine } from "nirnaya";     // the same call this site makes

const engine = await loadEngine();          // loads once, then cached by the browser
const answers = await engine.decide(
  "Post in #general: 'lol if you idiots merge that again I'm going to lose it. seriously. anyway here's my number 555-0142 if anyone wants to grab a beer after the release'",
  {
    "harassment": {
      "type": "noul",
      "instructions": "The post harasses, insults or demeans a person or group."
    },
    "threat": {
      "type": "noul",
      "instructions": "The post contains a credible threat of harm."
    },
    "pii": {
      "type": "noul",
      "instructions": "The post exposes personal contact information."
    },
    "action": {
      "type": "choice",
      "instructions": "What should the moderation system do?",
      "criteria": {
        "allow": "Leave it up.",
        "flag": "Leave it up but flag for a moderator.",
        "redact": "Remove the sensitive part and keep the rest.",
        "remove": "Take the post down."
      }
    },
    "tone": {
      "type": "score",
      "instructions": "How hostile is the tone?",
      "criteria": [
        "Friendly.",
        "Neutral.",
        "Irritated.",
        "Hostile.",
        "Abusive."
      ]
    }
  }
);
for (const a of answers) console.log(a.qid, a.answer, a.p);

curl and Python target nirnaya_serve, the Jev-compatible server in the release; the browser snippet is exactly what this site does.