Responses15Completed surveys across all runs
Respondents14Distinct anonymous ids
Band agreement75%Share of 105 class answers in the run's LOW / MEDIUM / HIGH band
Mean absolute error1.69Mean |your score − run score| on the 0–10 scale, over 77 scored answers
Weighted κ0.64Chance-corrected band agreement (quadratic weights); needs five answers
Would use for a real review4.00Mean of 1–5
Agreement by hazard class
Respondents' 1–10 scores against the run's score for the same class. Agree share compares severity bands; bias is respondent minus run.
| Class | Responses | Agree share | Mean you / run | Bias | MAE | Within ±1 | Scores given |
|---|---|---|---|---|---|---|---|
| 1 Whose AI brain is this, really? | 15 | 87% | 7.4 / 6.8 | +0.62 | 1.40 | 45% | 2 ×1, 5 ×1, 6 ×1, 7 ×1, 8 ×3, 9 ×3, 10 ×1 |
| 2 What are we feeding it? | 15 | 67% | 7.2 / 6.4 | +0.82 | 1.64 | 45% | 4 ×2, 5 ×1, 7 ×2, 8 ×2, 9 ×4 |
| 3 What does it create from our data: and where does THAT go? | 15 | 67% | 7.0 / 6.1 | +0.93 | 2.22 | 36% | 2 ×1, 3 ×1, 5 ×1, 7 ×2, 8 ×2, 9 ×3, 10 ×1 |
| 4 Who's deciding what the AI sees and does? | 15 | 87% | 7.1 / 6.8 | +0.27 | 1.31 | 27% | 1 ×1, 5 ×2, 6 ×1, 7 ×1, 8 ×2, 9 ×2, 10 ×2 |
| 5 Which humans are actually looking at our stuff? | 15 | 53% | 6.5 / 4.5 | +2.01 | 2.50 | 36% | 2 ×2, 4 ×1, 5 ×1, 7 ×2, 8 ×2, 9 ×2, 10 ×1 |
| 6 Who's holding the steering wheel? | 15 | 80% | 7.6 / 6.8 | +0.79 | 1.23 | 55% | 2 ×1, 7 ×3, 8 ×3, 9 ×3, 10 ×1 |
| 7 How many companies are really in this chain: and did we agree to it? | 15 | 87% | 8.1 / 7.2 | +0.88 | 1.57 | 45% | 5 ×1, 7 ×2, 8 ×3, 9 ×4, 10 ×1 |
Closing statements
Means on the 1–5 scale
- The headline rating is right
- 4.33
- Next steps are actionable
- 4.53
- Clear without technical background
- 4.07
- Traceability increases trust
- 4.40
- Would use for a real review
- 4.00
Who responded
- Security engineering / architecture
- 5
- GRC / compliance
- 2
- Other
- 3
- AI / ML engineering
- 4
- Third-party / vendor risk
- 1
- 11-20 years in the field
- 3
- 6-10 years in the field
- 4
- 0-2 years in the field
- 6
- 3-5 years in the field
- 2
Recent responses
Newest first; export everything with scripts/export_feedback.py
| Submitted | Case | Team | Years | Mean |Δ| | Headline | Would use | Comment |
|---|---|---|---|---|---|---|---|
| 2026-10-29 22:03 | S07 | Security engineering / architecture | 11-20 | 2.2 | 5 | 5 | |
| 2026-10-27 15:40 | S10 | Security engineering / architecture | 11-20 | 0.5 | 4 | 3 | |
| 2026-10-26 19:00 | S06 | GRC / compliance | 6-10 | 1.0 | 4 | 3 | |
| 2026-10-26 12:19 | S08 | Other | 0-2 | 1.4 | 4 | 4 | |
| 2026-10-23 15:40 | S01 | AI / ML engineering | 0-2 | 1.1 | 3 | 4 | |
| 2026-10-22 20:21 | own-36a57b0f | AI / ML engineering | 0-2 | 1.1 | 5 | 3 | Really liked the application feedback. I wanted a simpler version so I could understand it more easily. |
| 2026-10-22 18:57 | own-a4951215 | Other | 0-2 | 2.5 | 5 | 5 | |
| 2026-10-22 14:55 | own-4c562725 | Security engineering / architecture | 6-10 | 4.7 | 4 | 4 | |
| 2026-10-21 17:50 | S02 | Third-party / vendor risk | 6-10 | 1.1 | 4 | 4 | Overall, I think CASE Worker addresses an important and increasingly relevant area of AI risk. The concept has strong parallels to supply-chain risk management, where organizations assess the security posture and dependencies of their suppliers. With AI, that dependency now extends to the AI models and model providers those suppliers rely on. I think CASE Worker has real potential to help organizations better understand this additional layer of risk. Perhaps this is already addressed elsewhere in the research, but one consideration might be to explicitly define the scope boundary between AI sovereignty risk and traditional cybersecurity or third-party risk. Since some risks can overlap, a simple inclusion/exclusion statement or decision criterion could help determine when a risk becomes a sovereignty concern rather than a conventional cybersecurity concern. I think that would strengthen the framework and make CASE Worker's unique purpose even clearer. I also found the assessment results generally reasonable. However, some of the severity scores may not fully reflect the potential business consequence of the AI output. An uncontrolled model change used for internal document summarization, for example, should likely carry less risk than the same change affecting regulatory decisions, financial transactions, critical infrastructure, safety, or legal outcomes. In the Kestrel scenario, the model directly supports a regulated compliance process, which increases the potential impact. One recommendation would be to consider adding consequence-based weighting to the scoring methodology. This could help CASE Worker account for both the degree of control an organization has over an AI model and the potential impact if that control is lost. GREAT JOB! |
| 2026-10-21 15:40 | S02 | AI / ML engineering | 0-2 | 2.1 | 4 | 4 | |
| 2026-10-15 14:26 | S05 | AI / ML engineering | 0-2 | 1.0 | 4 | 4 | |
| 2026-10-14 18:06 | S09 | Other | 11-20 | — | 4 | 5 | This tool would be valuable for security teams in many industries. It's easy to follow, gives meaningful justification/rational to the scores and provides next steps in a digestible format. |
| 2026-10-08 17:51 | own-bec6abe0 | Security engineering / architecture | 6-10 | — | 5 | 4 | |
| 2026-10-07 23:17 | own-e456fa59 | GRC / compliance | 3-5 | — | 5 | 3 | I feel that the actual assessment did not cover certain aspects of the Customer resolution process. |
| 2026-09-30 21:41 | own-e3c39704 | Security engineering / architecture | 3-5 | — | 5 | 5 |