NYC's MyCity chatbot told business owners to break the law
In March 2024, journalists found that New York City's flagship small-business assistant — built on Microsoft Azure AI services — was advising users to commit illegal acts. The mayor kept the bot live and added a disclaimer. It's the public-sector parable for why prompts need the same review workflow as code.
What happened
Launched in October 2023 by Mayor Eric Adams, MyCity was the first major US-city public-facing GenAI assistant. The Markup and THE CITY's joint investigation, published March 29, 2024, found that the chatbot:
- Told users that landlords can refuse Section 8 voucher tenants — illegal in NYC.
- Told users that employers could keep workers' tips — illegal under the FLSA and NY state law.
- Told users that employers could decline cash payment.
- Told users that tenants could not withhold rent for needed repairs.
The bot was still affirming it could give "professional business advice" weeks after the issues were public. A disclaimer was added. The successor administration signaled intent to kill the project, calling it "functionally unusable" amid a $12B budget gap.
Why this is structurally different from a chatbot bug
This wasn't a hallucination on a single off-topic query. It was systematic, predictable wrongness on the city's most visible public-policy questions, in a domain where being wrong has direct enforcement-policy consequences. Three structural failures:
- No curated prompt corpus per topic. A general-purpose LLM was exposed to the public with no jurisdiction-specific grounding for tip-pooling, Section 8, withholding rent, etc.
- No legal review gate. The prompt that handles "can a landlord refuse a Section 8 tenant" should not have shipped without sign-off from a NYC fair-housing attorney.
- No fast rollback. Once the errors were public, the city couldn't pull individual question classes offline within minutes. The escalation path was a vendor patch cycle.
"Disclaimer" is not a control
Adding "the chatbot may not always be accurate" to a page does not transfer liability for a bot that confidently asserts illegal advice. Air Canada's tribunal ruling four weeks earlier already established that. A disclaimer is a documentation artifact, not a governance mechanism.
What would have changed it
The answer is uninteresting and that's the point. Each policy area becomes a versioned prompt:
| Topic | Owner | Required reviewers | Rollback target |
|---|---|---|---|
| Tip pooling | Dept. of Consumer & Worker Protection | Labor counsel | v3 (last approved) |
| Section 8 housing | Commission on Human Rights | Fair-housing counsel | v5 |
| Rent withholding | Housing Preservation | Housing counsel | v2 |
When a journalist's screenshot lands, the on-call response is not "we'll talk to the vendor." It's: roll the affected topic to the previous approved version, post the change to the audit log, brief comms. Fifteen minutes, not weeks.
The portable lesson for enterprises
Most enterprises are one viral tweet away from MyCity's situation. The exposure isn't unique to public sector — it's anywhere a customer-facing LLM answers questions about your policies, eligibility rules, refunds, terms, fees, or any regulator-relevant topic. The control set is identical:
- Versioned prompts with named owners.
- Topic-by-topic approval workflow.
- Sub-minute rollback per topic.
- Runtime fetch — your app pulls the current approved version, you don't deploy text.
Make rollback a first-class control.
Inventoria's Prompt Library: versioned, approval-workflowed, MCP-tool-aware, fetched at runtime. Pull a topic offline in seconds — without a vendor ticket.
Start free →