Surveillance and automated social credit systems have been praised by some individuals as efficient ways to reward good behaviour and punish bad behaviour. But who decides what good and bad behaviour are? If a teacher or policeman tallies the merit and demerit points, we are entitled to some “reprieve” because these folks are not omnipresent. As humans, we forgive and ignore little bad things that good people sometimes do. However, if points were awarded or deducted by an intelligent machine that can monitor all CCTV footage and mobile phone activity of each and every individual under it’s purview, there will be no escape from the smallest “sin”. The goodness and badness of a complex human being will be numerically defined with perfect “accuracy” as if God himself were watching.
Nexus by Yuval Noah Harari:
“For example, social credit systems could create a new underclass of “low-credit people.” Such a system may claim to merely “discover” the truth through an empirical and mathematical process of aggregating points to form an overall score. But how exactly would it define pro-social and antisocial behaviors? What happens if such a system deducts points for criticizing government policies, for reading foreign literature, for practicing a minority religion, for having no religion, or for socializing with other low-credit people? As a thought experiment, consider what might happen when the new technology of the social credit system meets traditional religions.
Religions like Judaism, Christianity, and Islam have always imagined that somewhere above the clouds there is an all-seeing eye that gives or deducts points for everything we do and that our eternal fate depends on the score we accumulate. Of course, nobody could be certain of their score. You could know for sure only after you died. In practical terms, this meant that sinfulness and sainthood were intersubjective phenomena whose very definition depended on public opinion.
What might happen if the Iranian regime, for example, decides to use its computer-based surveillance system not only to en- force its strict hijab laws, but to turn sinfulness and sainthood into precise inter-computer phenomena? You didn’t wear a hijab on the street-that’s -10 points. You ate on Ramadan before sunset- another 20 points deducted. You went to Friday prayer at the mosque, +5 points. You made the pilgrimage to Mecca, +500 points. The sys- tem might then aggregate all the points and divide people into “sin- ners” (under 0 points), “believers” (0 to 1,000 points), and “saints” (above 1,000 points). Whether someone is a sinner or a saint will depend on algorithmic calculations, not human belief. Would such a system discover the truth about people or impose order on people?”