The Internet Archive faces a new threat: Wary publishers who opt out to stop scraping by AI bots

Ruins of the Library of Pantainos in Athens, Greece. Photo (cc) 2018 by Michael Kogan.

Has the Internet Archive reached the end of the line? The 30-year-old nonprofit, which has saved and made searchable more than a trillion webpages, has proved itself to be of enormous value over the years.

I’ve used it to track changes in reporting, including this blog post about The New York Times’ shifting coverage of an explosion at Ahli Arab Hospital in Gaza City in the days after Hamas’ October 2023 terrorist attack on Israel. The Times and other news organizations initially reported that Israeli forces had bombed the hospital, but they later had to walk back that unverified claim.

Follow my Bluesky newsfeed for additional news and commentary. And please join my Patreon for just $6 a month. You’ll receive a supporters-only newsletter every Thursday.

The Internet Archive is also home to The Boston Phoenix’s online digital and print archives thanks to an agreement that it made with Northeastern University, which acquired the Phoenix’s intellectual property after the legendary alt-weekly went out of business in 2013. (Note: I was a longtime staff columnist for the Phoenix, and I helped arrange the donation to Northeastern.)

Now, though, the Internet Archive and its Wayback Machine, which reproduces web content from years past, are facing an existential threat. News organizations ranging from the Times to USA Today are inserting code into their sites that blocks the Archive from crawling their content, mainly to prevent AI companies from accessing their journalism without permission.

As Katie Knibbs reports for Wired, the irony is that USA Today recently published an important piece of investigative journalism documenting ICE detention statistics that wouldn’t have been possible without the Archive. Knibbs writes:

According to analysis by the artificial-intelligence-detection startup Originality AI, 23 major news sites are currently blocking ia_archiverbot, the web crawler commonly used by the Internet Archive for the Wayback project. The social platform Reddit is too. Other outlets are limiting the project in different ways: The Guardian does not block the crawler, but it excludes its content from the Internet Archive API and filters out articles from the Wayback Machine interface, which makes it harder for regular people to access archived versions of its articles.

The Electronic Frontier Foundation, which is helping to lead a signature drive in support of the Archive, compares the publishers’ actions to “a newspaper publisher announcing it will no longer allow libraries to keep copies of its paper,” according to a recent EFF article by Joe Mullin, who writes:

For nearly three decades, historians, journalists, and the public have relied on the Internet Archive to preserve news sites as they appeared online. Those archived pages are often the only reliable record of how stories were originally published. In many cases, articles get edited, changed, or removed—sometimes openly, sometimes not. The Internet Archive often becomes the only source for seeing those changes. When major publishers block the Archive’s crawlers, that historical record starts to disappear.

This is not the first time the Archive has run into legal problems. One major challenge was of its own making: a project begun during the COVID pandemic to make books available for free without permission and without any compensation to publishers or authors. Not surprisingly, the Archive lost that case in a federal appeals court in 2024. As I wrote in describing that decision: “The Archive claimed that it was in compliance with copyright law because it limited e-book borrowing to correspond with physical books that it had in its collection or that was owned by one of its partner libraries. That’s not the way it works, though.”

The current threat involves the right of publishers’ to make the content available as they see fit, which they have a legal right to do. They are under no obligation to let the Internet Archive repurpose it. Ideally, they will come to understand the incalculable damage they are doing.

As EFF’s Mullin puts it: “There are real disputes over AI training that must be resolved in courts. But sacrificing the public record to fight those battles would be a profound, and possibly irreversible, mistake.”

A Muzzle for Teresa Riley, the chief immigration judge, for her silence over a censorious firing

ICE goons grab Rümeysa Öztürk near Tufts.

Eleven months ago, I handed a New England Muzzle Award to Donald Trump’s thuggish immigration czar, Stephen Miller, for the arrest and detention of Rümeysa Öztürk. The Tufts University Ph.D. student’s only offense was to help write an op-ed piece in The Tufts Daily that was critical of Israel and sympathetic to the pro-Palestinian cause.

Follow my Bluesky newsfeed for additional news and commentary. And please join my Patreon for just $6 a month. You’ll receive a supporters-only newsletter every Thursday.

Now Roopal Patel, the Boston immigration judge who ended deportation proceedings against Öztürk, has been fired. And thus I’m awarding another New England Muzzle, this one to Teresa Riley, the chief immigration judge who was appointed to her position by the Trump regime. I don’t know whether she was involved in Patel’s firing. What I do know is that Riley has neither resigned in protest nor raised her voice in outrage since Patel was dismissed on Friday.

Patel was actually one of two immigration judges fired Friday who had been involved in high-profile immigration cases. The other, Nina Froes, had ruled similarly that Trump officials had no right to detain Mohsen Mahdawi, a green card holder who’d been involved in pro-Palestinian protests at Columbia University. There’s another New England angle as well — Froes’ court is based in Chelmsford, Massachusetts.  The New York Times reports:

Ms. Patel, like many immigration judges interviewed by The Times, said the Trump administration had made it clear that it wanted more immigrants ordered deported.

“It was a pressure I at least tried to actively resist,” she said in an interview. “All people in the United States are entitled to due process, and everyone deserves to have their cases adjudicated fully and fairly.”

According to The Boston Globe (sub. req.), Patel “was nearing the end of her two-year probationary period” when she was fired. “Even though I was expecting it, it was still sort of shocking,” Patel told the Globe. “The consequences are immediate.”

The Globe reports that 113 immigration judges out of more than 700 have been fired since January 2025. The paper quoted Patel as saying:

It’s creating this climate of fear where judges are worried that if they misstep and do something that’s out of line with what the administration wants, they’re more subject to firing. That can erode judicial independence, it can erode due process, and it can make people more likely to be ordered removed from this country.

Unlike most judges, who are part of the independent judiciary, immigration judges are considered members of the executive branch and are appointed by the attorney general. “The judges there need more judicial independence,” Patel told the Times in speaking about her former colleagues.

This is the way repression works. Just as international students learned from the Öztürk and Mahdawi cases that the price of avoiding arrest and detention is to refrain from their First Amendment-protected rights to write and to protest, immigration judges have learned from Patel and Froes that they should place Trump’s agenda above the law if they want to hold onto their jobs.

The Society of Professional Journalists blasts Massachusetts for a ‘troubling lack of transparency’

The Massachusetts Statehouse. Photo (cc) 2024 by Dan Kennedy.

The Society of Professional Journalists has named the State of Massachusetts recipient of its 2026 Black Hole Award, an annual dishonor recognizing government entities that demonstrate a troubling lack of transparency and disregard for the public’s right to know.

This article is a press release from the Society of Professional Journalists, republished here by permission.

The award is presented each year during Sunshine Week, a national initiative promoting open government and access to public information.

The SPJ Freedom of Information Committee selected Massachusetts for deficiencies in the state’s public records law, including broad exemptions, weak enforcement mechanisms and persistent delays that limit access to government information.

“Access to public records is not optional — it is a cornerstone of a functioning democracy,” said FOI Committee Chair Michael Morisy. “When broad swaths of government operate outside public records laws, or when compliance is routinely delayed or obstructed, the public’s right to know is fundamentally compromised.”

Despite a legal framework that purports to guarantee access to public records, Massachusetts remains one of the few states in which the governor’s office, legislature and judiciary are largely exempt from public records requirements — leaving significant portions of the state government shielded from public scrutiny.

In addition, reporting has shown that:

  • Requests for public records are frequently delayed or ignored, despite statutory deadlines
  • Excessive fees are sometimes used to discourage or block access to records
  • Enforcement mechanisms are limited, often leaving requesters with no option but costly and time-consuming litigation
  • Compliance is inconsistent across agencies, with little centralized oversight or accountability

The committee also noted that Massachusetts is not alone in facing transparency challenges, with similar issues emerging in states across the country. However, the scope and persistence of these issues within Massachusetts make it a particularly clear example of the systemic barriers that continue to limit public access to government information.

“The public should not have to fight, wait or pay exorbitant costs to understand how their government operates,” said SPJ National President Chris R. Vaccaro. “Transparency delayed or denied is accountability denied — and that undermines the very foundation of public trust.”

The Black Hole Award is intended to call attention to actions and policies that restrict transparency and to encourage reforms that strengthen access to public records at all levels of government. Massachusetts was nominated by the SPJ New England Chapter.

SPJ presents the award annually to highlight the importance of open records, open meetings and the free flow of information in a democratic society.

Last year’s recipient was the Utah State Legislature for repeatedly undermining transparency by amending the state’s Government Access and Management Act to block the release of public records — even after court orders mandated their disclosure.

The Washington Post stands firm in a vital First Amendment battle over its Pentagon tip line

The Pentagon. Photo (cc) 2009 by Rudi Riet.

If you’ve canceled your subscription to The Washington Post because of the rightward lurch of its opinion section, the decimation of the newsroom or both, I have news that might surprise you: The paper is involved in a vitally important First Amendment battle over its right to report on the Pentagon.

Follow my Bluesky newsfeed for additional news and commentary. And please join my Patreon for just $6 a month. You’ll receive a supporters-only newsletter every Thursday.

Erik Wemple, himself a Post alumnus, reports in The New York Times that the Trump regime’s objection to a tip box the Post has been publishing has emerged as an issue in a lawsuit brought by the Times over the Pentagon’s restrictions on journalists.

Continue reading “The Washington Post stands firm in a vital First Amendment battle over its Pentagon tip line”

The Mass. public records law needs some teeth. Will 2026 be the year that it happens?

The Massachusetts Statehouse. Photo (cc) 2024 by Dan Kennedy.

Massachusetts has long been notorious for being one of the least progressive states with regard to government transparency. The state’s public records law is alone in exempting the governor’s office, the Legislature and the judiciary, leaving cities, towns, counties and the state’s executive agencies as the only government bodies that may be compelled to produce documents when requested to do so by journalists or members of the public.

Follow my Bluesky newsfeed for additional news and commentary. And please join my Patreon for just $6 a month. You’ll receive a supporters-only newsletter every Thursday.

What’s worse, there are few penalties for failing to comply with the law. As John Hilliard observes (sub. req.) in The Boston Globe:

In Massachusetts, the state law’s deadlines for fulfilling records requests can be ignored, workers can conspire to overestimate costs, elected officials can spend years fighting requests in court, or not bother releasing records at all. No one tracks whether local governments like cities and school districts follow the law; state agencies self-report requests, but not the reasons why they refuse them.

Michael Morisy, the chief executive of Boston-based MuckRock, who’s been helping people file public records requests for years, told the Globe: “It’s among the worst states when it comes to public records access.”

Continue reading “The Mass. public records law needs some teeth. Will 2026 be the year that it happens?”

A reporter’s home is raided, and the Justice Department is admonished for withholding information

U.S. Justice Department. Photo (cc) 2006 by Coolcaesar.

Should a judge be expected to know when a prosecutor’s request is illegal? I would have thought so. But that turns out to be not the case with regard to a Washington Post reporter whose home was raided by the FBI last month as part of a leak investigation targeting a Pentagon contractor.

Follow my Bluesky newsfeed for additional news and commentary. And please join my Patreon for just $6 a month. You’ll receive a supporters-only newsletter every Thursday.

New York Times reporter Charlie Savage reported recently that the Justice Department had failed to tell a judge that a 1980 federal law prohibited the government from seeking a journalist’s reporting materials in most instances. Because of that failure, the judge issued a warrant to search the home of Post reporter Hannah Natanson — a shocking move given that journalists are generally summoned to court and given an opportunity argue against being forced to turn over their documents.

Continue reading “A reporter’s home is raided, and the Justice Department is admonished for withholding information”

Northeastern journalism faculty members condemn the arrests of Don Lemon and Georgia Fort

As current and former faculty members at Northeastern University’s School of Journalism, we condemn the unconstitutional arrests of independent journalists Don Lemon and Georgia Fort. We are instructors, mentors and colleagues of young journalists, and we believe it is imperative that we stand up for the vital role of a free and unfettered press in a democratic society.

The Justice Department has filed charges against Lemon and Fort for the crime of committing journalism when they accompanied activists who entered Cities Church in St. Paul, Minnesota, on Jan. 18. The activists were there to protest the pastor’s alleged employment by ICE. The journalists were there to observe, to live-stream the proceedings and to interview participants, church members and the pastor before leaving the church. In so doing, they engaged in activities protected by the First Amendment with the goal of informing the public about the Trump administration’s deadly and illegal occupation of the Twin Cities.

As Amnesty International put it: “Journalism is not a crime. Reporting on protests is not a crime. Arresting journalists for their reporting is a clear example of an authoritarian practice.” We call on the Justice Department to drop all charges against Lemon and Fort and to acknowledge the centrality of journalism in holding the government and other powerful institutions to account.

Note: Our statement was published earlier this morning by The Huntington News, Northeastern’s independent student newspaper.

Belle Adler
Rahul Barghava
Mike Beaudet
Matt Carroll
Myojung Chung
Ellen Clegg
Charles Fountain
John Guilfoil
Meg Heckman
Carlene Hempel
Marcus Howard
Jeff Howe
Dan Kennedy
William Kirtz
Catherine Lambert
Laurel Leff
Peter Mancusi
Meredith O’Brien
Jody Santos
Alan Schroeder
Jeb Sharp
Dan Zedek

Note: The list of signatories has been updated.

A stunning violation of the First Amendment as the Trump regime arrests two journalists

Don Lemon. Photo (cc) 2019 by Ted Eytan.

The Trump regime has taken direct aim at the First Amendment, arresting four people — including two journalists — for their role in a protest at a church in St. Paul, Minnesota, earlier this month. The journalists are Don Lemon, a former CNN anchor who’s now independent, and Georgia Fort, an independent reporter based in the Twin Cities who recorded her thoughts just before she was arrested and posted a video on Facebook.

Lemon and Fort accompanied protesters as they made their way inside a church to protest what they claimed were the pastor’s ties to ICE. They were there to live-stream and report on what happened, and the Department of Justice hasn’t produced a scintilla of evidence that their activities went beyond that. This is a grotesque violation of the First Amendment. I’m tempted to say that it’s shocking, but it’s not. Stunning?

Here is yet another independent journalist, former Washington Post reporter Philip Bump, on Bluesky:

The thing about this is that I don’t know a single journalist who will be intimidated by Don Lemon being arrested? But, then, that’s not why the administration is doing this. The founders couldn’t have imagined a president violating core freedoms for the purposes of social-media accolades.

Philip Bump (@pbump.com) 2026-01-30T13:42:06.336Z

The Committee to Protect Journalists weighed in earlier today.

The Committee to Protect Journalists (CPJ) strongly condemns the arrests of journalists Don Lemon and Georgia Fort for their reporting on a protest in Minnesota, arrests which mark a serious escalation of attacks on the press in the United States.

“This is an egregious attack on the First Amendment and on journalists’ ability to do their work,” said CPJ CEO Jodie Ginsberg. “As an international organization, we know that the treatment of journalists is a leading indicator of the condition of a country’s democracy. These arrests are just the latest in a string of egregious and escalating threats to the press in the United States — and an attack on people’s right to know.”

Lemon, who formerly reported for CNN and now publishes on Substack, was arrested Thursday night; Fort, an Emmy-winning filmmaker, was arrested Friday morning. Both arrests were in relation to their coverage of a protest at a Minnesota church led by local activists against immigration enforcement operations in the state.

Lemon was arrested in Los Angeles, where he was reporting on the Grammy Awards. Here is what his lawyer, Abbe Lowell, had to say:

Don has been a journalist for 30 years, and his constitutionally protected work in Minneapolis was no different than what he has always done. The First Amendment exists to protect journalists whose role it is to shine light on the truth and hold those in power accountable.

Instead of investigating the federal agents who killed two peaceful Minnesota protesters, the Trump Justice Department is devoting its time, attention and resources to this arrest, and that is the real indictment of wrongdoing in this case. This unprecedented attack on the First Amendment and transparent attempt to distract attention from the many crises facing this administration will not stand. Don will fight these charges vigorously and thoroughly in court.

Update, Jan. 31. The Washington Post has published a visual investigation of Lemon’s activities at the church. As you’ll see, it’s pretty unremarkable — Lemon enters, interviews people and leaves.

By raiding a reporter’s home, Trump and his thugs have escalated their attacks on a free press

Barack Obama’s administration threatened reporters with jail if they refused to turn over their confidential sources. But he didn’t order raids on reporters’ homes. Photo (cc) 2024 by Gage Skidmore.

Back in 2012, I wrote an opinion piece for The Huffington Post (now just HuffPost) that I headlined “Obama’s War on Journalism.” The premise was that Barack Obama, like George W. Bush and other presidents before him, was disrespecting the First Amendment’s protection of independent journalism by taking reporters to court and theatening them with jail if they didn’t reveal the identities of White House sources leaking to them.

At least Obama, Bush et al. were following a legal process. As The Associated Press reports, Donald Trump’s FBI, headed by the buffoonish but dangerous Kash Patel, raided the home of a Washington Post journalist to grab what they claimed were classified documents provided by a Pentagon contractor.

Continue reading “By raiding a reporter’s home, Trump and his thugs have escalated their attacks on a free press”

One good reason the shutdown should have continued; plus, a settlement in Kansas, and Kara Miller’s new podcast

Senate Democratic Leader Chuck Schumer has been harshly criticized for his handling of the government shutdown. Photo (cc) 2024 by the Jewish Democratic Council of America.

We’ve been hashing out the pros and cons of ending the government shutdown on Facebook this week. My position has been that the Democrats shouldn’t have caved, but that it was a close call. Certainly the shutdown couldn’t have gone on too much longer, especially with families in danger of going hungry and federal workers not receiving paychecks.

Sign up for free email delivery of Media Nation. You can also become a supporter for just $6 a month and receive a weekly newsletter with exclusive content.

More than anything, I didn’t see any possible way that the Democrats could achieve their stated objective of forcing Donald Trump and the Republican Congress to extend health-care subsidies. The government could have stayed shut for six more months and that wouldn’t have changed.

Continue reading “One good reason the shutdown should have continued; plus, a settlement in Kansas, and Kara Miller’s new podcast”