Has AI Already Written a Law without Anybody Noticing?

As I started working on the LangGraph Agent Security blog series, a thought came to my mind that I haven’t been able to shake: has AI already written and passed a law without anybody noticing?

Here’s a scenario how it could happen.

In the final hour before a vote on a massive, contentious bill, congressional staff are exhausted, running on coffee and are under tremendous deadline pressure. A last minute addendum needs to be drafted, and it’s something technical, something that needs to sound like legislation. A staff member uses AI to generate the addendum text quickly, he reviews it very briefly and inserts it into a 1,000-page document that nobody will read in full before the vote is called. The bill passes, the president signs the bill into law, but nobody notices that the AI generated addendum actually includes something that nobody has intended.

I don’t have any proof that this has ever happened, but it’s entirely plausible.

Why This Matters More Than It Sounds

Most AI risk discussions focus on dramatic scenarios: autonomous weapons, economic collapse, existential threats. This risk is quieter and in some ways more insidious precisely because it doesn’t require anyone to act maliciously. It just requires people to be busy, tired, and reaching for a useful tool.

But a more malicious version of the above scenario may happen. For example, the owner of the proprietary AI model could anticipate that congressional staff will be using it to draft bills in the upcoming legislature session, and instructs the AI, either through its training or its system prompt, to add a small clause that’s self-serving when generating bill language: a subtle carve-out in a regulatory bill; a definition written just broadly enough to exclude a competitor. Nothing dramatic enough to trigger scrutiny. Just enough to matter.

I have no public evidence that this has occurred, but it’s plausible.

Three Recommendations

1. Require disclosure when AI is used to draft legislative text.

The simplest intervention is transparency. If a congressional office uses AI to generate or substantially edit legislative language, that should be disclosed in the legislative record; just as we require disclosure of lobbyist involvement. This doesn’t prevent AI use; it creates accountability for it. Staffers who know their AI use will be disclosed are more likely to review the output carefully. Knowing which AI model is used will make it clear if there’s any model self-serving in the bill’s language. This is inline with many other industries’ (e.g. video gaming, publishing) AI disclosure requirements.

2. Mandate open-source or government-hosted models for official bill drafting.

The intentional manipulation scenario is most dangerous when the AI being used is a closed, proprietary system operated by a private company with its own interests. A government-hosted open-source model that’s auditable, with no external operator removes that attack surface. Yes, this is more complex and expensive than using a commercial API, but the alternative is drafting laws on infrastructure controlled by entities with active interests in the outcome of those laws, and that could be more expensive in the long run.

3. Establish an AI output review process for late-stage bill amendments.

The highest-risk moment is the last-minute addendum under deadline pressure — exactly the scenario where review is most likely to be skipped. A simple procedural requirement that any AI-generated text in a bill amendment be flagged and reviewed by at least one additional staffer before insertion would significantly reduce the unintentional risk. This requires no new technology, just a process.

The Broader Point

We are at a moment when AI is becoming embedded in the machinery of governance faster than the rules governing its use are being written. The question of who controls the AI tools used to draft legislation is not a technical question; it is a question about who has influence over the laws that govern all of us.

Congressional staff are under-resourced, overworked, and increasingly reliant on AI tools to keep up. That is not a criticism; it is a structural reality. The right response is not to prohibit AI use in legislative drafting; that ship has likely already sailed. The right response is to build the transparency, accountability, and procedural safeguards that make that use trustworthy.

The good news is that none of the recommendations above require solving hard technical problems. They require political will and process design; exactly the kind of intervention that Congress can actually make.