The term has usually been associated with increased volume, variety and velocity of data and is defined as extremely large data sets which reveal patterns, trends and associations, especially relating to human behaviour and interaction. It is a novel idea insofar as it is a breakaway from more traditional data-gathering practices.
Despite the legitimate concerns surrounding big data, recent criticism has relied on a veritable army of strawmen. All too often the attacks are weak and misguided, focusing unduly on extreme cases of bad practice. Sometimes iterative prototypes of big data projects are treated as the final product and their flaws exaggerated.
Concerns have been amplified, as ever, by the moral panic of technophobes. Unfortunately, those too quick to condemn promising ideas are themselves condemned to complacency.
Take Place Pulse, for example, a project to map people’s perception of a city’s safety and wealth. Despite pictures being taken at arbitrary times of the day, new data and algorithms have helped overcome the initial issues of inconsistency in data collection. It has the potential to revolutionise the study between urban perception and other datasets, such as violent crime, creativity or economic growth.
Progress is not, however, just about GDP growth. Big data has also helped make finance more efficient and safe. Opposition to the proliferation of data-driven technology in this sector is hard to disentangle from the general distaste many have about the finance industry— viewing it as immoral and corrupt. But that is a whole other conversation.
The transition towards technology in finance may be seen as a form of Joseph Schumpeter’s ‘creative destruction’; change inevitably means disruption, but it is better for all of us in the long run. There’s a reason we often trust a machine to take out cash rather than an employee at a cashier. It is only a matter of time before cognitive AI feels the same.
It is not just finance where big data can be put to use. Take the conundrum of genetic modification. Like a Russian doll, there are many layers. One of these involves big data. In July 2018, 23andme (a DNA health and ancestry service) and pharma giant GlaxoSmithKline began using genetic records to help research and develop new medicine. 23andme provides DNA information on individuals’ health and ancestry—its terms of service clearly state that customers may have their data used for research purpose. Customers can opt out if they choose to do so, but some 80 per cent decide not to.
We should be kissing the feet of those 80 per cent, as the data they are providing could prove transformational for genetic research and, ultimately, curing diseases.
Joint action gene variants are the cause of many common illnesses and data is scarce. As there are so many variants and their causes and effects are so numerous and hard to pinpoint, large-scale data is required to identify disease risk combinations. Certainly, GSK has a private interest in gathering such data, but that should not obscure its potential contribution to the advancement of science itself.
The GSK/23andme partnership calls not only for a defence of big data, but also the profit motive; without GSK seeing the potential commercial gains to be made from the investment, 23andme might not have been an anchor for potentially revolutionary research.
As well as helping to take advantage of some of the more futuristic opportunities, big data can also help us solve some age-old problems, not least building houses. Companies like LandInsight and UrbanIntelligence are using online, accessible land data to create maps for property developers (and hopefully estate agents in due course). The data allows for a streamlined process, saving time and energy by taking the role of data sharing from the state into private hands.
Of course, this kind of innovation must go hand in hand with the kind of wide-scale planning reform that so many are now arguing for to free up supply, reduce house prices and provide the kind of homes people actually want to live in.
While the opportunities are undoubtedly exciting, there are obviously significant pitfalls that come with the advent of new technology. Many are rightly concerned about the possibility of an unholy alliance between big data and Big Brother. The textbook example is China’s burgeoning digital dictatorship. Its “social credit” system brings privileges for the loyal and punishment for those who don’t toe the Communist Party line. To help keep the people under control, the Chinese government employs a vast network of 200 million CCTV cameras to monitor its 1.4 billion citizens.
Advocates of centrally planned economies have also contended that the rise of big data will help solve Hayek’s ‘knowledge problem’ of socialism by providing information about demand and allowing the state to supply. But we should be sceptical of technocrats who claim new technology can overcome the inherent unworkability of a state-directed economy.
As von Mises explains: “One cannot add up values or valuations. One can add up prices expressed in terms of money, but not scales of preference”. Simply put, central planners cannot take advantage of the price mechanism which accounts for these factors in the real economy, however much data they have at their disposal.
There are also cultural hurdles to the proliferation of big data. Often there is an unwillingness to encourage collecting or sharing data, in case it reveals socioeconomic trends that unsettle us. The archetype of this is the French government, which refuses to collect data on its citizens based on race or ethnicity, ostensibly to maintain the idea that all French citizens are created equal, whatever evidence there is to the contrary. Another good example is the Nordic paradox, which reveals often surprising outcomes for egalitarians.
The rising pressure to publish pay differences between men, women and ethnic minorities is also not helpful. Some 43 per cent of businesses cite corporate privacy as the reason they would not be prepared to open up access to others to their own data. If the government coerces businesses to do so, they may decide against collecting the data in the first place.
Certainly we ought not to dismiss people’s legitimate concerns about privacy and it would be ludicrous to claim some kind of moral imperative for individuals to deliver up their data in the name of science and progress. At the same time the vilification of big data risks obscuring the many potential benefits it promises to bring.
Just as with the Luddites wrecking handlooms, we may look back on the moral panic around big data and wonder why we were so pessimistic. We should proceed with caution, but also optimism – big data has and will be an instrument for positive change if harnessed with the appropriate technology.
CapX depends on the generosity of its readers. If you value what we do, please consider making a donation.