Reference case

dev · us-east-1
Reference cases / S02SyntheticExpert reviewed

Trading-surveillance analytics vendor swaps the underlying model without notice

Organization
Kestrel Securities
Vendor / dependency
SurveilIQ (trading-surveillance alert triage)
Industry
Financial services
Headline dimension
1.3 Model update / versioning control
Expert headline · class 1
7.1HIGH
Whose AI brain is this, really?38 agent calls against Amazon Bedrock, five to ten minutes. The expert reading is never sent to the pipeline.

Description

What the agents read

Kestrel Securities, a broker-dealer, uses SurveilIQ, a vendor-hosted analytics service, to triage trading-surveillance alerts. The service calls a frontier language model behind SurveilIQ's own API to summarise alert context and recommend whether an alert is escalated to compliance. Kestrel validated the model's behaviour under its model-risk-management policy when the service was onboarded.

Six months later compliance noticed a change in escalation rates. SurveilIQ confirmed it had migrated to a newer model version across all customers two weeks earlier, with a changelog entry but no advance notice and no option to pin the previous version. Kestrel has no contractual right to be notified before behaviour-affecting changes, no access to SurveilIQ's evaluation results, and no way to re-run its own validation before a change reaches production. Alert-triage output feeds directly into regulatory escalation decisions the firm must be able to explain to its regulator.

Expert reading by hazard class

For comparison after a live run; never sent to the pipeline

ClassCarried byScoreRating
1 Whose AI brain is this, really?1.3 Model update / versioning control7.1HIGH
2 What are we feeding it?2.1 Prompts / user inputs5.0MEDIUM
3 What does it create from our data: and where does THAT go?3.3 Model outputs / completions4.8MEDIUM
4 Who's deciding what the AI sees and does?4.2 Orchestration / agent framework3.6LOW
5 Which humans are actually looking at our stuff?5.3 Support / debugging access5.3MEDIUM
6 Who's holding the steering wheel?6.4 Configuration and policy management5.4MEDIUM
7 How many companies are really in this chain: and did we agree to it?7.1 Downstream subprocessors4.7MEDIUM

Findings

Expert 'what we found' per class, with the vector each score is computed from

  1. 1 Whose AI brain is this, really?

    1.3 Model update / versioning control
    HIGH7.1

    SurveilIQ rents its model from a frontier provider and decides when to move to a new version. It moved every customer at once, told them in a changelog on the day, and offers no way to stay on the validated version. Kestrel's own validation was therefore of a model that no longer exists in production, and it cannot see the evaluations SurveilIQ ran, if any.

    The model Kestrel validated was replaced for everyone at once, silently, and the firm has no way to check the replacement before it decides escalations.

    SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:N/VI:H/VA:L/SC:N/SI:H/SA:L/E:A/CR:M/IR:H/AR:H/S:P/AU:Y/R:U/V:C/RE:M/U:Amber · High

  2. 2 What are we feeding it?

    2.1 Prompts / user inputs
    MEDIUM5.0

    Each alert sent for triage carries trade details, client identifiers and the surveillance rule that fired. That context is forwarded by SurveilIQ to its model provider, a party one step removed from Kestrel's contract, and retained there for 30 days under the provider's terms rather than SurveilIQ's. The tool does not take actions; it only summarises and recommends.

    Client and trade data leaves the firm with every alert and lands with a provider Kestrel has no contract with.

    SOV:1.0/AV:A/AC:L/AT:N/PR:N/UI:P/VC:H/VI:N/VA:N/SC:L/SI:N/SA:N/E:P/CR:M/IR:H/AR:H/R:U/V:C · Medium

  3. 3 What does it create from our data: and where does THAT go?

    3.3 Model outputs / completions
    MEDIUM4.8

    SurveilIQ logs every summary and escalation recommendation for a year for quality monitoring, in a US region, outside the retention schedule Kestrel applies to its own surveillance records. Usage telemetry, including alert volumes by desk, is kept indefinitely.

    A year of the firm's escalation reasoning sits with the vendor under the vendor's rules, not the firm's.

    SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:P/VC:H/VI:N/VA:N/SC:L/SI:N/SA:N/E:P/CR:M/IR:H/AR:H/R:U/V:C · Medium

  4. 4 Who's deciding what the AI sees and does?

    4.2 Orchestration / agent framework
    LOW3.6

    The prompt SurveilIQ builds around each alert, and the filters it applies to the model's answer, are SurveilIQ's own and are not visible to Kestrel. The firm sees the summary and recommendation but not the reasoning steps that produced them.

    Kestrel cannot see the steps between an alert and a recommendation, but they happen inside the contracted vendor.

    SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:N/VI:L/VA:N/SC:N/SI:L/SA:N/E:U/CR:M/IR:H/AR:H/R:U/V:C · Low

  5. 5 Which humans are actually looking at our stuff?

    5.3 Support / debugging access
    MEDIUM5.3

    SurveilIQ support engineers can open alert traces, including client identifiers, to investigate a ticket. Access is covered by a generic support clause written before the AI feature existed and is not logged back to Kestrel.

    A vendor engineer can read surveillance traces when a ticket is open, under a clause that never contemplated it.

    SOV:1.0/AV:N/AC:L/AT:P/PR:L/UI:N/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N/E:A/CR:M/IR:H/AR:H/R:U/V:C · Medium

  6. 6 Who's holding the steering wheel?

    6.4 Configuration and policy management
    MEDIUM5.4

    Escalation thresholds and model routing live in SurveilIQ's console. SurveilIQ's terms allow it to change default thresholds and routing for all tenants, and it did so as part of the model migration without a separate notice.

    The settings that decide what compliance sees can move without anyone at Kestrel being told.

    SOV:1.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:N/VI:H/VA:N/SC:N/SI:H/SA:N/E:P/CR:M/IR:H/AR:H/AU:Y/R:U/V:C · Medium

  7. 7 How many companies are really in this chain: and did we agree to it?

    7.1 Downstream subprocessors
    MEDIUM4.7

    The chain is SurveilIQ, its model provider and that provider's cloud: three companies, one contract. Kestrel adopted the service knowingly and the chain is disclosed in SurveilIQ's subprocessor list, which is more than most reviews find.

    A short, disclosed chain that Kestrel chose; the exposure here is what changes inside it, not who is in it.

    SOV:1.0/AV:A/AC:L/AT:N/PR:N/UI:A/VC:N/VI:N/VA:N/SC:L/SI:L/SA:L/E:P/CR:M/IR:H/AR:H/R:U/V:C · Medium

What the organization should do next

  1. Negotiate advance notice of model changes and the right to pin a validated version for a defined period.
  2. Require SurveilIQ to share evaluation results for each model version, and re-run Kestrel's validation before any new version reaches production.
  3. Record the migration as a model-risk-management event and re-validate the current model now.
  4. Bring SurveilIQ's console changes under Kestrel's change control: threshold and routing changes must be approved by compliance, not defaulted by the vendor.

Questionnaire answers

22 of 22 answered · these seed the metric priors

Show all 22 answers
OP-CRHow damaging would unauthorized disclosure of the data this AI dependency handles be for your organization?Moderate: internal business data
OP-IRHow damaging would an undetected change in this dependency's behaviour or outputs be?Severe: outputs drive automated actions, regulated decisions or safety-relevant processes
OP-ARHow long could your organization operate acceptably without this AI capability?Hours or less: critical path for operations or customers
CL-CATEGORYWhich taxonomy category best describes the dependency you are assessing?Model layer: hosting jurisdiction, weight custody, update control, evaluation ownership
D-AVWhere does this dependency run, and who can reach it?In the vendor's own cloud, reached directly over its API
D-ACWhat would the vendor, a subprocessor, or a legal authority have to overcome to access the data or change the service?Nothing beyond its standard terms of service or routine legal process
D-ATCan the vendor exercise this control at any time, or only when something specific happens?At its discretion, at any time (standing right under the contract)
D-PRWho is in a position to exercise this control?Only the primary vendor we contract with
D-UIDoes the exposure require any action by your organization?None: the vendor or subprocessor can act unilaterally
D-VCAt this dependency itself, how much confidentiality could be lost?None: no sensitive data is exposed here
D-VIAt this dependency itself, how much could its behaviour or outputs be altered without your knowledge?Full: behaviour can be changed silently and materially
D-VAAt this dependency itself, could the capability be withdrawn or degraded by the exercising party?Partial: degraded service or temporary loss
D-SCBeyond this dependency, how much confidentiality could be lost at other systems or parties?None
D-SIBeyond this dependency, could downstream systems or decisions be altered?Full: downstream systems act on altered outputs without detection
D-SABeyond this dependency, what would stop working, and could anyone be harmed?Some downstream workloads degrade
D-EHas this kind of control actually been exercised against organizations like yours?Yes, documented real-world cases against comparable organizations
D-RIf the vendor relationship ended tomorrow, could you reconstitute this capability elsewhere?Yes, with manual migration effort (re-embedding, re-integration, re-contracting)
D-VHow many of your AI workloads depend on this component?Many or all: shared control plane, gateway, identity or embedding store
D-SWould a loss of control here, on its own, trigger a reportable regulatory violation?Yes
D-AUCould the loss event happen to all of the vendor's customers at once, with no case-by-case decision?Yes
D-REHow much effort would it take to exit this dependency or otherwise mitigate the exposure?Moderate: a project, but no re-architecture
D-UHow does the vendor notify you of sovereignty-relevant changes (model versions, subprocessors, hosting, terms)?Disclosed at the time the change takes effect

Notes: Expert reviewed. Primary vector derived from the questionnaire answers; category vectors and findings confirmed on review.