China’s Social Credit System: AI-driven panopticon or fragmented foundation for a sincerity culture?

China’s Social Credit System, a government initiative which aims to assign a “social credit” rating to every citizen based on their financial behavior, personal information, and online activity, has earned a bad reputation abroad, including comparisons to an episode of “Black Mirror.” According to critics, the data-based, AI-driven system is more suited for comprehensive social control than keeping tabs on individuals’ financial state like credit score rankings devised in the West.

The trouble with most of this characterization is that it fails to recognize just how fragmented digital credit score-keeping is in China: it is not a single system, but many.

“I think the biggest misconception about the Social Credit System is that it is this evenly implemented all-seeing central bureaucratic surveillance apparatus,” Shazeda Ahmed, a Berkeley Ph.D. student researching the topic, told TechNode.

Social credit systems are currently being trialed by several provinces and cities, with Shanghai being one of the more famous ones with its “Sincere Shanghai” app. Each area decides its own rules for score keeping.

The ultimate goal is to lay out foundations for an encompassing Social Credit System by 2020 which will integrate not only individual, but also government, legal, and enterprise scoring. The system aims to compensate for underdeveloped credit scoring, but it also aspires to establish a “sincerity culture” by addressing rampant fraud, corruption, and mistrust in the country. And unlike traditional credit scoring which was established by financial agencies and institutions, a significant part of China’s Social Credit System is being built with technology.

Alibaba and Tencent—The social credit vanguard

After the Chinese State Council laid out an outline for building a Social Credit System in 2014, China’s central bank authorized eight Chinese tech companies, including Alibaba and Tencent, to conduct social credit pilot testing. According to Pennsylvania State University Ph.D. candidate Wang Keren, China’s tech giants were the vanguard in building “social trust” by developing e-commerce and online payments rating systems.

“The real drivers of this social credit reform are corporate actors—most notably this emergent generation Chinese ‘new-industrialists’ such as Jack Ma [founder and chairman of Alibaba] and Ma Huateng [founder and chairman of Tencent],” Wang told Technode.

Sesame Credit, run by Alibaba’s Ant Financial, is the now biggest social credit pilot built on the company’s huge trove of information on consumers. The platform, which has seen massive growth in recent years, provides credit information on individuals, as well as enterprises.

Alibaba has been promoting its Sesame Credit by offering users with good scores discounts on air tickets and hotels, deposit waivers on bike and car rentals, and even fast-tracked visas from countries such as Singapore and Luxembourg. The service is also being used as an authentication method for users of the Chinese version of Airbnb and dating site Baihe.

“It’s pretty useful because it is a reliable system which is connected to all the big platforms,” said Sesame Credit user Wen Sida. “Personally, I trust their system better than the government’s so-called credit system.”

But the fact that a private company is collecting so much different data has led to concerns that recently prompted China’s central bank to withdraw tech companies’ licenses for conducting social credit pilots. Some of the reasons behind the decision are the lack of consumer privacy protection, overreach of data collection, as well as conflict of interests, Ahmed explained.

“My own thoughts on this are that the government probably didn’t expect that something like Sesame Credit would have hundreds of millions of users in just two years and be so deeply embedded in people’s lives,” said Ahmed.

Her views were echoed by Pen State Law School research assistant Zhu Shaoming:

“The leading authority of the world has been transferred from religious leaders to politicians, and now to data owners. The social credit system is not market-oriented; it requires the government’s macro-control, especially given that the legal framework of the social credit system has not yet been established.”

Subsequent media reports of hackers claiming they earned thousands of dollars by manipulating Sesame Credit scores (in Chinese) have further cemented fears over data leaks. And this isn’t the only case—according to a report from Caixin (paywall), China’s entire data industry is trading personal information in a way that causes systemic abuses of privacy.

But neither the government’s decision to pull back on the pilot nor mounting concerns over data privacy has stopped Alibaba from moving forward with its social credit scheme. In fact, its biggest rival, Tencent, announced the launch of its own credit system in early August signaling that private social credit systems are alive and well.

This shouldn’t surprise us. Society in the West has already accepted a fairly extensive social credit system for many decades, including private credit ratings, employee assessments which can be shared, customer cards and reward credit cards, Professor of Law and International Affairs at Penn State Law Larry Cata Backer told TechNode.

Massive data collection

While many reports have focused on how social scoring will affect individuals, a more interesting aspect might be its implications for doing business in China, including foreign companies. According to an analysis published by Mercator Insititute for China Studies (MERICS), China’s Social Credit System has the potential to become the most globally sophisticated and fine-tuned model for IT-backed and big data-enabled market regulation.

“Functionally speaking, the Chinese social credit system has a wider goal than the Western credit system. It also aims to standardize industries, especially professional industries, into the system,” Zhu Shaoming said.

The MERICS report states that China is currently implementing a highly innovative approach to monitoring, rating, and regulating the behavior of market participants, including individuals, companies, and other institutions, such as NGOs, which will fall under the Social Credit System. The idea is to create a highly effective and adaptive economy capable of outperforming slower and more fragmented Western economies. At its heart: massive data collection.

The data will be collected from multiple sources by the so-called “National Credit Information Sharing Platform” and used to generate ratings for each company. These scores will have an immediate impact on business opportunities for companies incentivizing them to self-restrict their behavior by following regulations and government industrial targets.

A good example of this is real-time measuring of emission data for polluting industries which is already being piloted. But this is also where the first problems are visible. In April, media reported that one-third of manufacturers in northern China fabricated real-time pollution data in the government-installed monitoring systems.

If left unchecked, data manipulation, as well as data hacking, could turn the heart of the Social Credit System into one of darkness. But even with transparent data, the AI algorithms that will eventually be used for calculating credit scores might remain opaque.

“As an aid to disciplining discretion, AI has potentially useful applications,” said Larry Cata Backer. “As a means of avoiding responsibility, or of hiding behind machine decision-making to avoid individual responsibility (or governmental responsibility), AI poses a danger to the integrity of any system that would so ‘wash their hands’ of governance.”

Opposite of lacking oversight, another potential risk is too much state control over the economy that could reduce the capacity for autonomous business decisions or disruptive business models that fall out of the standard, the MERICS report notes. In order to avoid bad scores, companies might be pressured to comply with political targets even if it doesn’t make sense for them business-wise.

For now, this is all just speculation. If implemented correctly, the system will have many positive effects on China’s market, including fewer lawbreakers and more transparency. It could also make China a leader in big data-based technologies and a role-model for other economies in the world.

“In my opinion, social credit is already poised to become a dominant mode of managing behavior all over the world,” said Cata Backer, adding that the system would first have to be effectively implemented and transposed to other national cultures.

As for the fear of IT authoritarianism and China’s poor track record in keeping its citizens’ data private, this question may be more connected with the love of secrecy for which the Chinese state is known for.

“In public affairs, China does not have a good track record of keeping data transparent,” said Zhu. “Compared with keeping data private or transparent, it is more important to categorize data, keep the data secure, i.e., prevent abuse and fraud, and use the data wisely. Some data needs to private; some data needs to be transparent.”

This article was updated on August 23rd to correct the statement that users with low scores on Sesame Credit may affect receiving loans or refunds for online purchases. Sesame Credit currently does not have any punitive measures for users with low scores, nor does the Sesame Credit score affect receiving loans.