Twitter can’t afford to be one of the world’s most influential websites

The Washington Post / Elizabeth Dwoskin, Joseph Menn, Cat Zakrzewski

11:56 JST, September 5, 2022

In the weeks leading to Twitter’s launch of a new fact-checking program to combat misinformation, experts at the company warned managers that the project could be easily exploited by conspiracy theorists.

Those warnings – which went unheeded – almost came true. The night before the invitation-only project, called Birdwatch, launched, in 2021, engineers and managers learned that they had inadvertently accepted a proponent of the violent conspiracy theory QAnon into the program, which would have enabled them to publicly annotate news-related tweets to help people determine their veracity.

The details of Twitter’s near miss with Birdwatch came to light as part of an explosive whistleblower complaint filed in July by the platform’s former head of security, Peiter Zatko. Zatko had commissioned an external audit of Twitter’s capabilities to fight misinformation and it was included in his complaint. The Post obtained the audit and the complaint from congressional staff.

While Zatko’s allegations of Twitter’s security failures, first reported last month by The Post and CNN, have received widespread attention, the audit on misinformation has gone largely unreported. Yet it underscores a fundamental conundrum for the 16-year-old social media service: despite its role hosting the opinions of some the world’s most important political leaders, business executives and journalists, Twitter has been unable to build safeguards commensurate with the platform’s outsized societal influence. It has never generated the level of profit needed to do so, and its leadership never demonstrated the will.

Twitter’s early executives famously referred to the platform as “the free speech wing of the free speech party.” Though that ethos has been tempered over the years, as the company contended with threats from Russian operatives and the relentless boundary-pushing tweets from former president Donald Trump, Twitter’s first-ever ban of any kind of misinformation didn’t take place until 2020 – when it prohibited deep fakes and falsehoods related to covid-19.

Former employees have said that privacy, security, and user safety from harmful content were long seen as afterthoughts for the company’s leadership. Then-CEO Jack Dorsey even questioned his most senior deputies’ decision to permanently suspend Trump’s account after the Jan. 6, 2021, riot at the U.S. Capitol, calling silencing the president a mistake.

The audit report by the Alethea Group, a company that fights disinformation threats, confirms that sense, depicting a company overwhelmed by well-orchestrated disinformation campaigns and short on engineering tools and human firepower while facing threats on par with vastly better-financed Google and Facebook.

The report described severe staffing challenges that included large numbers of unfilled positions on its Site Integrity team, one of three business units responsible for policing misinformation. It also highlighted a lack of language capabilities so severe that many content moderators resorted to Google Translate to fill the gaps. In one of the most startling parts of the report, a head count chart said Site Integrity had just two full-time people working on misinformation in 2021, and four working full-time to counter foreign influence operations from operatives based in places like Iran, Russia, and China.

The report validates the frustrations of outside disinformation experts who have labored to help Twitter identify and reduce campaigns that have poisoned political conversations in India, Brazil, the United States and elsewhere, at times fueling violence.

“It has this outsized role in public discourse, but it’s still staffed like a midsize platform,” said Graham Brookie, who tracks influence operations as head of the Atlantic Council’s Digital Forensics Research Lab. “They struggle to do more than one thing at one time.”

The result of Twitter’s chaotic organizational structure, the Alethea report found, was that the experts on disinformation had to “beg” other teams for engineering help because they largely lacked their own tools, and had little guarantee that their safety advice would be implemented in new products such as Birdwatch.

The report also exposed slapdash technological workarounds that left experts using five different types of software to label a single tweet as misinformation.

“Twitter is too understaffed to be able to do much other than respond to an immediate crisis,” the 24-page report concluded, noting that Twitter was consistently “behind the curve” in responding to misinformation threats.

“Organizational siloing, a lack of investment in critical resources, and reactive policies and processes have driven Twitter to operate in a constant state of crisis that does not support the company’s broader mission of protecting authentic conversation,” it found.

Alethea declined to comment on the report.

Twitter disputes many details in the 2021 report, arguing that it depicted a moment in time when the company had far less staff, and that by focusing on a single team, it portrayed a misleadingly narrow picture of the company’s broader efforts to combat misinformation.

A senior company official, who spoke on the condition of anonymity because of ongoing litigation with billionaire Elon Musk, told The Post that the report – which was based on interviews with just 12 Twitter employees – tended to blow individuals’ concerns out of proportion, including worries about the Birdwatch launch. He said the report’s staffing numbers referred only to senior policy experts – the people who set the rules – while the company currently has 2,200 people, including dozens of full-time experts and thousands of contractors, to actually enforce them.

“To successfully moderate content at scale, we believe companies – including Twitter – can’t invest in head count alone,” Yoel Roth, Twitter’s head of safety and integrity, said in an interview. “Collaboration between people and technology is needed to address these complex challenges and effectively mitigate and prevent harms – and that’s how we’ve invested.”

Nonetheless, at the time that Twitter had just six full-time policy experts tackling foreign influence operations and misinformation, according to the report, Facebook had hundreds, according to several people familiar with internal operations at Meta, Facebook’s parent company.

Twitter is vastly smaller, in terms of revenue, users, and head count, than the other social media services it’s compared to, and its ability to combat threats is proportionally smaller as well. Meta, which owns Facebook, Instagram, and WhatsApp, for example, has 2.8 billion users logging in daily – more than 12 times the size of Twitter’s user base. Meta has 83,000 employees; Twitter has 7,000. Meta earned $28 billion in revenue last quarter; Twitter earned $1.2 billion.

But some of the issues confronting Twitter are worse than Facebook and YouTube, because the platform traffics in immediacy and because people on Twitter can face broad attacks from a public mob, said Leigh Honeywell, chief executive of Tall Poppy, a company that works with corporations to mitigate online abuse of their employees. She added that Twitter users can’t delete negative comments about them, while YouTube video providers and Facebook and Instagram page administrators can remove statements there.

“We see the highest volume of harassment in our day-to-day work on Twitter,” Honeywell said.

“It isn’t a sound defense to say we’re really small and we’re not making that much money,” said Paul Barrett, deputy director of the Stern Center for Business and Human Rights at New York University. “You’re as big as your impact is, and you had that obligation, while you were becoming so influential, to protect against the side effects of being so influential.”

To be sure, wealthier companies, including Facebook and YouTube, face similar problems and have made halting progress in combating them. And Twitter’s size, experts said, has also accorded it a certain nimbleness that enables it to punch above its weight. Twitter was the first company to slap labels on politicians for breaking rules, including putting a warning label on a May 2020 tweet from Trump during the George Floyd protests.

Twitter was also the first company to ban so-called “deep fakes,” the first company to ban all political ads, and, at the onset of the Ukraine war, the first to put warning labels on content that mischaracterizes a conflict as it evolves on the ground.

The company was also first to launch features that slowed the spread of news on its service in an effort to prevent misinformation from quickly spreading, such as a prompt that asked people if they’d read an article before they retweeted it. And it published a first-ever archive of state-back disinformation campaigns on its platform, a move researchers have praised for its transparency.

Frances Haugen, a Facebook whistleblower who raised the alarm about the shortcomings of Meta’s investments in content moderation and has been highly critical of technology companies, has said that other companies should copy some of Twitter’s efforts.

“Because Twitter was so much more thinly staffed and made so much less money, they were willing [to be more experimental],” Haugen said in an interview.

But nation-backed adversaries such as Russia’s Internet Research Agency could adapt quickly to such changes, while Twitter lacked tools to keep up.

“There is an enormously vulnerable landscape that is infinitely manipulatable, because it’s easy to evolve and iterate as events occur,” Brookie said.

Twitter employees made much the same point, according to the Alethea report, complaining that the company was too slow to react to crises and other threats and sometimes didn’t have the organizational structure in place to respond to them.

For example, the report said that Twitter delayed responding to the rise of QAnon and the Pizzagate conspiracy theory – which falsely alleged that a Democrat-run pedophile ring operated out of a pizza shop in Washington – because “the company could not figure out how to categorize” it.

Executives felt QAnon didn’t fall under the purview of the disinformation team because the movement wasn’t seeded by a foreign actor, and they determined that the conspiracy wasn’t a child exploitation issue because it included false instances of child trafficking. They did not deem it to be a spam issue despite the aggressive, spamlike promotion of the theory by its proponents, the report said. Many companies, including Facebook, faced similar challenges in addressing QAnon, The Post has previously reported.

It was only when events forced the company’s hand, such as the celebrity Chrissy Tiegen threatening to leave Twitter because of harassment from QAnon devotees, that executives got more serious about QAnon, the report said.

“Twitter is managed by crisis. It doesn’t manage crisis,” a former executive told The Post. The executive was not interviewed by Alethea for its report, and spoke on the condition of anonymity to describe sensitive internal topics.

Twitter’s lack of language capabilities figure prominently in the Alethea report. The report said that the company was unprepared for an election in Japan in 2020 because there were “no Japanese speakers on the Site Integrity team, only one [Trust and Safety] staff member located in Tokyo, and severely limited Japanese-language coverage among senior [Twitter Services] Strategic Response staff.”

In Thailand, the report said, Twitter moderators are “only able to search for trending hashtags . . .. because they do not have the language or country expertise on staff” to conduct actual investigations.

The Twitter executive who spoke on behalf of the company said the report painted a misleading picture about its response to threats internationally. He said Twitter maintains a large office in Japan, which is a huge market for the company, and had employees who consulted on misinformation issues during the election there. He pointed to the company’s record of taking down influence operations in Thailand, including the suspension, in 2020, of thousands of murky accounts that appeared to be tied to a campaign to mar opponents of the Thai monarchy.

Some former insiders told The Post that aspects of their experience at Twitter echoed the report. Edwin Chen, a data scientist formerly in charge of Twitter’s spam and health metrics and now CEO of the content-moderation start-up Surge AI, said that the company’s artificial intelligence technology to tackle hate speech was typically six months out of date. He said it was often difficult to get resources for projects related to creating a healthier discussion on the platform.

“You have to kind of convince this other team to do this work for you because there’s a lack of strong leadership,” he said.

He also noted that there’s always tension between those who work in safety and security and those responsible for other aspects of the business. “There’s an inevitable tradeoff between growth and security, and there’s always going to be something missing,” he said.

Rebekah Tromble, director of the Institute for Data, Democracy, and Politics at George Washington University, noted in an interview that because of the public and political nature of the Twitter platform, operatives see it as ideal for sowing disinformation campaigns.

“Though Twitter has a minuscule number of users compared to YouTube, Facebook, and TikTok, because it is such as public platform, those who seek to spread misinformation and undermine democracy know that Twitter is one of the best places to increase the likelihood of their messages spreading widely,” she said. “The folks that they hire are good, and earnest, and really want to make a difference – but Twitter is just an under-resourced company compared to the outsized impact they have on the larger information ecosystem.”