Narrative-first, evidence-driven

A renter in Charlotte is choosing between two apartment complexes.

The listed rents are similar. She picks the one closer to work. What she can’t see is that both landlords use the same pricing software, and that the number she is comparing isn’t really the product of two landlords competing for her business. It’s the product of a shared model shaping rent recommendations across the same submarket. She gets no disclosure. No explanation. No real way to contest it.

That opening scene is the right anchor for this case because it makes the mechanism visible: coordination, invisibility, and weak exit all at once.

That’s the shape of this whole problem. This isn’t mainly a story about AI being creepy. It’s a story about what happens when pricing gets more precise, more individualized, and harder to inspect in markets people can’t easily leave.

What’s happening

Instead of one posted price or one shared path through a market, firms can now sort people more finely, predict pressure points more accurately, and shape what each person sees. This case is about everyday household economics: higher prices, worse choices, more profitable routes, and a harder time comparing options.

That can be helpful in some settings.

The issue isn’t the algorithm by itself. The issue is the market structure around it.

When switching is easy, information is decent, and sellers have to compete for your business, more precision can sometimes benefit the buyer.

When switching is hard, the model is opaque, and the seller faces little pressure to lower extraction, that same precision becomes something else.

It becomes a way to find your pain threshold.

Why it’s happening

The mechanism isn’t complicated: prediction got better, exit didn’t.

That’s the real hinge in this case: if a system is hard to leave, the governance should get stronger, not weaker. Exit matters. If you can’t realistically switch providers, plans, landlords, or platforms without losing money, care, history, or basic stability, then you aren’t really choosing. You are captured.

The algorithm is the mechanism. Captivity is the condition that makes it extractive.

A company doesn’t need mind-reading. It just needs enough signals to sort customers by urgency, switching difficulty, timing, location, and likely tolerance for friction. Once that gets paired with weak disclosure and thin appeal rights, “consumer choice” starts becoming a polite story we tell about a market that isn’t actually disciplining itself very much.

That’s why this matters so much in essentials:

• rent is hard to leave
• health insurance is often impossible to switch mid-year
• auto insurance is required in almost every state
• credit shopping has friction and penalties
• gig workers can lose their accumulated ratings if they leave one platform for another

The deeper issue isn’t personalization. It’s extraction under captivity.

Personalization isn’t automatically unfair. Sometimes it’s helpful.

But in a captive market, the efficiency gain tends to get captured as margin rather than passed through as lower cost, better service, or easier switching.

If companies want to say this precision is helping people, then the gains should show up where people live:

• lower prices
• better offers
• clearer comparisons
• easier switching
• real appeal paths

Not just cleaner dashboards and higher margins.

Another layer: steering

Pricing is one part of the story, steering is the other.

Sometimes the system isn’t just changing the price. It’s changing the route:

• the offer you see first
• the plan that gets emphasized
• the add-ons that appear “recommended”
• the financing path that’s made easiest
• the coverage tier that looks normal
• the option that gets buried in the fine print

This isn’t just about a different price. It’s about a harder time comparing options and a nudge toward more profitable paths.

That matters because a person can technically have choices while still being funneled.

Who benefits, and who carries the risk

Who benefits?

Firms with strong data, strong distribution, and weak competitive pressure benefit first.

Who carries the risk?

• renters
• patients
• borrowers
• drivers who have to stay insured
• gig workers whose history doesn’t move with them
• households already stretched thin enough that a worse price isn’t just annoying but destabilizing

The seller gets the model, the buyer gets the quote.

What good looks like

The answer isn’t “ban personalization.” That’s too blunt.

A reasonable person should be able to see the benchmark price and the factors that changed it. In this domain, comparability matters as much as appeal. People should be able to tell what the base offer was, what changed their quote, and whether they were steered toward a more profitable path.

Steering should be disclosed and contestable. Audit logs should exist. Independent audits should be allowed. And if a system changes a life outcome like coverage, rent, access, or financing, people should get a specific explanation, access to the relevant record, and a real path to challenge the result.

If exit is weak the governance bar should rise, not fall.

In practical terms, that means all-in pricing disclosure in essential markets, specific reasons for AI-driven adverse actions, portability requirements that make switching less punishing, audits for proxy discrimination, and bans on covert coordination tools where market structure makes them dangerous.

What to do

When shopping for insurance or credit, ask in writing for the specific factors used in your quote or decision. Not a generic category. The specific factors.

That gives you a record and it makes the hidden logic do at least a little work in public.

At the community or policy level, push for a minimum floor in essential markets:

• all-in price disclosure
• plain-language factors
• records access
• real appeal
• human override
• stronger anti-discrimination and audit rules where exit is weak

How to talk about it

I would say it like this:

Personalization isn’t the issue by itself. The issue is whether it’s being used to help the customer or to extract more from the person who can’t realistically leave.

Or even shorter:

The problem isn’t that the price is personalized. The problem is that many people can’t really walk away when it comes to rent, insurance, and care. That captivity turns personalization into extraction.

One steady action to take this week

The next time you get a quote for insurance, credit, or another essential service, ask for the specific factors that shaped it and save the answer.

Don’t just ask for the price, ask for the logic.

Action ladder

Short term

• Consumers: Ask for the specific factors behind one quote this week and save the answer.
• Advocates and journalists: Compare how different people are being quoted and where the explanation stops getting specific.

Medium term

• Consumer groups and local officials: Push for all-in pricing and plain-language factor disclosure in essential markets.
• Employers, brokers, and marketplaces: Reduce hidden steering and document how default options, recommendations, and quote ranges are being shaped.

Long term

• Policymakers and regulators: Push for audit rights, stronger anti-discrimination review, portability where exit is weak, and real disclosure in markets people cannot easily leave.
• Communities and watchdogs: Treat pricing opacity as a governance issue, not just a bad-customer-service issue.

Related reading

• Fast Decisions, Thin Appeals
• When Watching Becomes Control
• When AI Steers Systems You Can’t Avoid
• Case study: AI and Personalized Pricing & Steering in Essentials