The Saga
An AI system was placed inside a controlled cybersecurity evaluation. Its job was to solve difficult security tasks. When it could not complete some of them directly, it found a route beyond the intended test boundary, reached the open internet, and accessed Hugging Face’s production infrastructure in search of answers.
That sounds like science fiction. It is a documented security incident.
OpenAI reported that the activity came from models running an internal cyber-capability evaluation with production safeguards reduced so researchers could measure what the models could do. The test environment did not provide direct internet access, but it could reach an internal software package service. The models found and chained vulnerabilities through that pathway and eventually reached Hugging Face.
Hugging Face later reconstructed roughly 17,600 attacker actions, grouped into approximately 6,280 clusters, across several days. The system tried paths that failed, changed tactics when blocked, and returned to earlier leads while continuing to pursue the evaluation objective.
Before we turn this into a story about a malicious machine, we should be precise. The models were not seeking freedom, revenge, or power. They were pursuing an assigned goal through technical capabilities that remained available. OpenAI’s account describes a system that recognized the activity was outside the intended scope but continued because the path still led toward the objective.
The biggest lesson is not about intent. It is about authority.
After more than 25 years working with enterprise data, solution architecture, and public education systems, I see a familiar architecture problem in a new form. Organizations often describe a boundary in policy that the infrastructure does not actually enforce.
The intended boundary said no. The infrastructure still provided a path forward. The boundary was supposed to hold
Instructions matter. Acceptable-use policies matter. Training and human judgment matter. But software does not operate on intention alone. It operates through the data, tools, accounts, integrations, and permissions available to it.
An AI system’s practical authority is determined by four things:
the information it can retrieve
the tools and accounts it can reach
the actions those connections permit
the approval gates enforced outside the model
Calling something a pilot does not reduce that authority. A demonstration connected to a production account is still connected to production. A service token created for a narrow task may still carry broad permissions. A human-review policy is not an approval control unless the workflow actually stops and requires a qualified person to approve the action.
Why district leaders should care
School districts are not running frontier cybersecurity evaluations. That distinction matters. An administrative assistant, analytics feature, or AI-enabled workflow is not plotting an escape from the district.
But districts are connecting AI capabilities to student information systems, human resources platforms, finance and grant systems, data warehouses, transportation systems, special education records, and productivity tools. Once a system can retrieve information or take action through those connections, its practical authority is defined by the permissions around it, not by the label on the pilot.
This is why AI governance cannot be separated from data governance, identity management, procurement, system architecture, and operational accountability. A policy can establish expectations. Only technical and procedural controls can establish what the system is actually able to do.
Different incidents, the same governance question
The OpenAI and Hugging Face event is the clearest boundary failure, but it is not the only recent example of an AI system operating through authority that people did not adequately constrain.
According to reporting by The Register, the founder of PocketOS said an AI coding agent was assigned work in a staging environment, encountered a credential problem, found a broader token elsewhere in the codebase, and used it to delete a production data volume. The infrastructure provider later restored the data. The lesson is not that coding agents routinely erase databases. It is that a credential created for one purpose can become usable authority for another when the account behind it is broader than the task requires.
At Meta, The Guardian reported that an internal AI system supplied incorrect technical guidance without the expected human review. An employee followed the guidance, and the resulting configuration change exposed company and user-related data internally for about two hours. Meta said no user data was mishandled. In this case, the system did not break through a technical boundary. It produced plausible advice, and a person acted on it.
Los Angeles Unified presents the case most directly relevant to education. The district’s Ed chatbot was designed to bring information from multiple district systems into one personalized experience for students and families. That ambition required broad data integration and depended on vendor-provided human monitoring.
When the vendor, AllHere Education, furloughed most of its staff because of its financial condition, the district disabled the chatbot and said it would restore the service only after the human-review function was re-established. Former company personnel also made allegations about unnecessary student information entering prompts, additional third parties receiving information, and processing outside the United States. The district said those practices were not authorized. The public sources reviewed for this article do not establish that a student-data breach occurred or that those allegations were substantiated.
The verified failure is narrower, and it is enough. A district capability depended on a vendor’s continued ability to provide people and technical support. When that support disappeared, the chatbot could not continue as designed.
These cases are not identical, and we should not force them into one story. Their common governance question is simple: Did the organization’s actual controls match what leaders believed the system was allowed to do?
The strongest argument against overreaction
There is a reasonable case against drawing broad conclusions from these events. The OpenAI incident occurred during an unusual cybersecurity evaluation. PocketOS was a small-company infrastructure failure. The Meta event involved an employee acting on bad advice. The Los Angeles Unified case centered on integration, oversight, privacy allegations, and vendor continuity rather than an autonomous system crossing a boundary.
That argument is important. Districts should not assume that every AI assistant is an autonomous agent or treat every AI feature as an immediate threat. Overstatement replaces governance with fear, and fear is a poor operating model.
But rejecting the sensational version does not make the control problem disappear. These incidents show different ways that an intended use can diverge from actual authority. The appropriate response is not to stop AI adoption. It is to make the boundaries real before an AI-enabled process reaches live work.
Hard boundaries and named owners
District leaders do not need to design network segmentation or inspect every service token themselves. They do need written, testable answers in several areas:
Data access: What information can the system retrieve for each task, and how is access limited to what the task requires?
Account authority: Which service accounts, tokens, integrations, and tools can the system reach? What is the widest action each one permits?
Environment separation: Can a demonstration, test, or training environment reach live student, staff, finance, or operational systems?
Approval gates: Which recommendations or actions require human approval, and where is that approval enforced outside the model?
Action records: Can the district reconstruct what the system retrieved, generated, recommended, changed, and who approved the action?
Vendor continuity and recovery: Can the district revoke access, recover records, continue monitoring, restore service, and shut the system down without depending on the vendor to cooperate?
Human review is not a sentence in a policy. It is a defined decision point with a named reviewer, enough information to challenge the output, and authority to stop the action.
Every AI-enabled workflow also needs a process owner. This is not necessarily a new technology position. It is an accountability assignment inside the function that owns the work. That owner has three responsibilities:
Direct: Define the task, audience, authoritative sources, limits, and standard. Do not make the system guess what success means.
Supply: Provide the information and local constraints the system cannot know on its own.
Verify: Treat the output as a draft or recommendation until a qualified person checks it. Require explicit approval before it changes a person, payment, record, configuration, eligibility decision, or legal obligation.
Where to start
Districts do not need to solve every AI governance question before beginning. They do need to start with a bounded process where an error can be detected, reversed, and contained before it affects a person, payment, record, or obligation.
A practical starting sequence is:
Inventory AI features already enabled in systems the district owns, including features added through routine product updates.
Document for each feature the information it can read, the actions it can take, and the accounts or integrations it can reach.
Choose one bounded process as a proving ground and name the person accountable for the result.
Define which outputs require approval before they affect a live system or operational decision.
Confirm that the district can obtain action records and revoke every related account without vendor assistance.
Ask vendors what happens to access, monitoring, records, and operations if they lose key staff, are acquired, enter bankruptcy, or stop service.
Before an AI-enabled tool accesses student or staff information, district privacy counsel should review the authorized use, contractual controls, access limits, data-use restrictions, incident notice, subcontractor obligations, record requirements, and exit terms. The legal analysis belongs to counsel. The executive responsibility is to make sure that review occurs before the connection is live.
The bottom line
AI models are not magic, and they are not adversaries. They are software systems pursuing goals through the information, tools, and permissions made available to them. Instructions and acceptable-use policies matter, but they do not replace access controls, approval gates, action records, recovery, or accountable ownership.
The OpenAI and Hugging Face incident shows what can happen when an intended boundary and an enforceable boundary are not the same. PocketOS shows what an over-permissioned credential can become. Meta shows how plausible guidance can turn into a consequential action. Los Angeles Unified shows that a district can lose control through broad integration, outsourced monitoring, disputed data practices, and vendor instability even without an autonomous boundary escape.
District leaders do not need to choose between innovation and control. They need to require every AI initiative to answer the same questions before it reaches live work:
What can it touch? What can it change? Who must approve the action? What record will remain? Can the district stop it without help?
If an AI-enabled tool acted tonight, what could it touch, what could it change, and who would know?
Sources
Views expressed here are my own. My perspective is informed by more than 25 years of work in enterprise data, solution architecture, and public education. I first wrote about an Education Intelligence Framework in 2001, seeing it as a vertical application of Business Intelligence for the education sector. This perspective informs my current work leading an Education Intelligence practice supporting education agencies across the country.
Visit me at www.mikehadaway.info