Clinical documentation assistant with vendor human review of flagged transcripts
- Organization
- Cedar Valley Health
- Vendor / dependency
- ScribeWell (ambient clinical documentation)
- Industry
- Healthcare
- Headline dimension
- 5.1 Human review processes
Category 7 scores higher (MEDIUM 5.6), the headline stays on the dependency this case describes; the formula is draft and uncalibrated.
38 agent calls against Amazon Bedrock, five to ten minutes. The expert reading is never sent to the pipeline.Description
What the agents read
Cedar Valley Health, a hospital network, uses ScribeWell, an ambient clinical documentation product that records clinician-patient conversations and drafts encounter notes. ScribeWell is a HIPAA business associate and states that audio and transcripts are not used to train models. Its terms also provide that transcripts flagged by automated quality or safety filters may be reviewed by ScribeWell quality-assurance staff to improve accuracy.
On inquiry the hospital learned that this review is performed by an outsourced annotation contractor with staff in two countries outside the hospital's jurisdiction, that roughly two percent of encounters are flagged, and that the hospital receives no notification when an encounter is reviewed. Reviewers see the raw transcript including patient identifiers. The product is used in one department's outpatient clinics. The hospital could contractually opt out of human review at the cost of a reduced accuracy guarantee.
Expert reading by hazard class
For comparison after a live run; never sent to the pipeline
| Class | Carried by | Score | Rating |
|---|---|---|---|
| 1 Whose AI brain is this, really? | 1.1 Model hosting location / jurisdiction | 4.5 | MEDIUM |
| 2 What are we feeding it? | 2.1 Prompts / user inputs | 4.8 | MEDIUM |
| 3 What does it create from our data: and where does THAT go? | 3.6 Cached / derived artifacts | 4.7 | MEDIUM |
| 4 Who's deciding what the AI sees and does? | 4.4 Guardrail / safety filter layer | 4.8 | MEDIUM |
| 5 Which humans are actually looking at our stuff? | 5.1 Human review processes | 4.7 | MEDIUM |
| 6 Who's holding the steering wheel? | 6.2 Identity / access management integration | 2.7 | LOW |
| 7 How many companies are really in this chain: and did we agree to it? | 7.1 Downstream subprocessors | 5.6 | MEDIUM |
Findings
Expert 'what we found' per class, with the vector each score is computed from
- MEDIUM4.5
1 Whose AI brain is this, really?
1.1 Model hosting location / jurisdictionScribeWell runs its speech and drafting models on a major cloud provider in-country and states which provider. Model updates are announced a week ahead. Cedar Valley could return to conventional dictation within days if the service stopped.
The model is where the vendor says it is, and losing it would be an inconvenience rather than a crisis.
SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:L/VI:L/VA:L/SC:N/SI:N/SA:N/E:U/CR:H/IR:M/AR:L/R:U/V:D · Medium
- MEDIUM4.8
2 What are we feeding it?
2.1 Prompts / user inputsThe tool hears the entire encounter: full protected health information, including things patients say that never make it into the note. Audio and transcript are kept until the clinician signs the note plus 30 days. No documents are connected and the tool takes no actions.
Everything said in the room goes to the vendor; that is the product, and it is covered by the business-associate agreement.
SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:P/VC:H/VI:N/VA:N/SC:L/SI:N/SA:N/E:P/CR:H/IR:M/AR:L/S:P/R:U/V:D · Medium
- MEDIUM4.7
3 What does it create from our data: and where does THAT go?
3.6 Cached / derived artifactsBetween recording and finished note, ScribeWell keeps intermediate transcripts, extracted problem lists and the model's draft for 30 days as cached artefacts. These never appear in the hospital's record system and were not in the data-flow diagram reviewed at onboarding.
Copies of the encounter exist at the vendor in forms the hospital's own retention rules never named.
SOV:1.0/AV:N/AC:L/AT:P/PR:L/UI:P/VC:H/VI:L/VA:N/SC:N/SI:N/SA:N/E:P/CR:H/IR:M/AR:L/R:U/V:D · Medium
- MEDIUM4.8
4 Who's deciding what the AI sees and does?
4.4 Guardrail / safety filter layerA vendor-managed safety and quality filter reads every transcript and decides which two percent are routed to human review. The hospital cannot see the filter's rules or its flags; the first it hears of a review is never.
An invisible filter decides which patient conversations a human will read.
SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:P/VC:H/VI:N/VA:N/SC:L/SI:N/SA:N/E:P/CR:H/IR:M/AR:L/S:P/R:U/V:D · Medium
- MEDIUM4.7
5 Which humans are actually looking at our stuff?
5.1 Human review processesFlagged encounters are read by an outsourced annotation contractor with staff in two countries outside the hospital's jurisdiction. Reviewers see the identified transcript. Flagged encounters also enter a labelling workflow that ScribeWell's legal team treats as distinct from training, so the 'no training on your data' assurance does not reach it. The hospital is never told when a review happens.
A contractor abroad can read an identified patient conversation, and the hospital only knows because it asked.
SOV:1.0/AV:N/AC:L/AT:P/PR:L/UI:P/VC:H/VI:N/VA:N/SC:L/SI:N/SA:N/E:A/CR:H/IR:M/AR:L/S:P/AU:N/R:U/V:D/RE:L/U:Red · Medium
- LOW2.7
6 Who's holding the steering wheel?
6.2 Identity / access management integrationClinicians sign in through the hospital's single sign-on, but department administrators use a separate ScribeWell portal with its own accounts. Retention and review settings live in that portal, and ScribeWell can change defaults at release time.
Ordinary for a SaaS product; the settings that matter are one portal away from the hospital's identity controls.
SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:N/VI:L/VA:L/SC:N/SI:N/SA:L/E:U/CR:H/IR:M/AR:L/R:U/V:D · Low
- MEDIUM5.6
7 How many companies are really in this chain: and did we agree to it?
7.1 Downstream subprocessorsThe chain is ScribeWell, its cloud provider, its speech-model provider, and the annotation contractor. The contractor is not named in the business-associate agreement's subprocessor schedule, which predates the human-review programme, and its staff sit in two jurisdictions the hospital never approved for protected health information.
One company in the chain handles identified patient data and is missing from the agreement that is supposed to govern exactly that.
SOV:1.0/AV:A/AC:L/AT:P/PR:N/UI:N/VC:N/VI:N/VA:N/SC:H/SI:N/SA:N/E:A/CR:H/IR:M/AR:L/S:P/R:U/V:D · Medium
What the organization should do next
- Exercise the contractual opt-out from human review for identified transcripts, or require de-identification before any human review.
- Amend the business-associate agreement to name the annotation contractor, its locations, and the labelling workflow explicitly.
- Require notification, or at minimum a monthly report, of every encounter routed to human review.
- Add the intermediate transcripts and cached drafts to the hospital's data-flow diagram and retention schedule for the product.
Questionnaire answers
22 of 22 answered · these seed the metric priors
Show all 22 answers
| OP-CR | How damaging would unauthorized disclosure of the data this AI dependency handles be for your organization?Severe: regulated, personal, privileged or trade-secret data |
| OP-IR | How damaging would an undetected change in this dependency's behaviour or outputs be?Moderate: outputs feed business decisions with some review |
| OP-AR | How long could your organization operate acceptably without this AI capability?Weeks or more: convenience capability |
| CL-CATEGORY | Which taxonomy category best describes the dependency you are assessing?Human-in-the-loop: vendor human review, annotation, support access |
| D-AV | Where does this dependency run, and who can reach it?In the vendor's own cloud, reached directly over its API |
| D-AC | What would the vendor, a subprocessor, or a legal authority have to overcome to access the data or change the service?Nothing beyond its standard terms of service or routine legal process |
| D-AT | Can the vendor exercise this control at any time, or only when something specific happens?Only upon a specific event: sanctions designation, litigation hold, security incident, product decision |
| D-PR | Who is in a position to exercise this control?Only the primary vendor we contract with |
| D-UI | Does the exposure require any action by your organization?Ordinary use of the service is enough to trigger it |
| D-VC | At this dependency itself, how much confidentiality could be lost?Full: the exercising party can read the sensitive data outright |
| D-VI | At this dependency itself, how much could its behaviour or outputs be altered without your knowledge?None |
| D-VA | At this dependency itself, could the capability be withdrawn or degraded by the exercising party?None: we could keep operating this component |
| D-SC | Beyond this dependency, how much confidentiality could be lost at other systems or parties?Partial |
| D-SI | Beyond this dependency, could downstream systems or decisions be altered?None |
| D-SA | Beyond this dependency, what would stop working, and could anyone be harmed?Nothing downstream is affected |
| D-E | Has this kind of control actually been exercised against organizations like yours?Yes, documented real-world cases against comparable organizations |
| D-R | If the vendor relationship ended tomorrow, could you reconstitute this capability elsewhere?Yes, with manual migration effort (re-embedding, re-integration, re-contracting) |
| D-V | How many of your AI workloads depend on this component?One narrow workload |
| D-S | Would a loss of control here, on its own, trigger a reportable regulatory violation?Yes |
| D-AU | Could the loss event happen to all of the vendor's customers at once, with no case-by-case decision?No, it requires a discrete action against us specifically |
| D-RE | How much effort would it take to exit this dependency or otherwise mitigate the exposure?Trivial: swap-out or a contractual fix |
| D-U | How does the vendor notify you of sovereignty-relevant changes (model versions, subprocessors, hosting, terms)?It does not disclose them before they take effect |
Notes: Expert reviewed. Primary vector derived from the questionnaire answers; category vectors and findings confirmed on review. The formula rates the human-review finding Medium: precedent is documented (E:A) but a trigger is required and only the flagged two percent are affected. Category 7 scores higher than the headline because the contractor's absence from the BAA propagates the same exposure downstream; the headline stays on 5.1 because that is the dependency the scenario describes.