Big Data, Small World

When a magazine like Foreign Policy comes up with an article titled 'The Rise of Big Data' you know that something has tumbled out of geekdom to the world of ordinary mortals.

The article talks about 'incredible new uses (of data) with the assistance of inexpensive computer memory, powerful processors, smart algorithms, clever software, and math that borrows from basic statistics'.  Instead of trying to “teach” a computer how to do things, the new approach is to feed enough data into a computer so that it can infer the possible answers on its own. 

India's ambitious UID project does not find mention here - perhaps its not big enough - but it does describe some amazing new uses of data. One of these is Prof. Shigeomi Koshimizu's "bottom-up" approach to establishing unique identities.

Prof. Shigeomi's team at AIIT-Tokyo placed 360 pressure-sensors on car-seats and found that when a person is seated, the contours of the body, its posture, and its weight distribution can all be quantified and tabulated and converted to a digital code that is unique to each individual!

Perhaps the day is not far when Indian's lining up for their UID cards will be spared the trouble of scanning fingerprints and of staring into machines for iris patterns. They would just have to just sit on a chair while somebody verifies their application and simply walk out with a biometric-id card..  :-)

---------------------------------------------------------

LINKS & REFERENCES

Cukier, Kenneth Neil  and Viktor Mayer-Schoenberger (2013): The Rise of Big Data - How It's Changing the Way We Think About the World, Foreign Policy, May/June 2013 -- url -- http://www.foreignaffairs.com/articles/139104/kenneth-neil-cukier-and-viktor-mayer-schoenberger/the-rise-of-big-data?page=show

Japanese boffins crack arse-based ID recognizer -- url -- http://www.theregister.co.uk/2011/12/27/japanese_bum_id_recognizer/

UID Project: The Task of Verifying Authenticity

.
Under Budget 2012, the Finance Ministry has allocated Rs.17.58 billion for enrolling 400 million more people into India's Unique Identity Project (UID). If all goes well we hope to have a national database covering the entire 1.20 billion population of India. If it succeeds we expect to have a more efficient public distribution system (PDS) for those who need it most - the poorest of India's poor.

It is a big 'if' -  especially when you look at the challenges of validating such a huge database. Even if the state manages to reach out and persuade each citizen to apply for a UID card, how will the database be managed and maintained?

The process of authentication would require each and every applicant to be cross-checked for duplications amongst all previous records. With each individual's biometric data taking about 5MB the entire database is expected to have at least six petabytes (6,000 terabytes, or 6,000,000 gigabytes), making it among the world’s largest databases.

The dairy industry provides an interesting analogy. In large collection centers, milk is stored in vertical silos. In order to avoid wastage of storage-space to froth & foam, milk is pumped in from the bottom of the tank .With each additional liter pumped in, you have to deal with the pressure exerted by the weight of milk already in the tank. Thus it takes a greatest amount of power to pump in the 99999th litre into such a tank.

How much computing power - and time - would it take to verify the authenticity of the 1,199,999,999,999 UID applicant in India?

-------------------------------------------------------
Context of the UID Project:

• India has a population of about 1.2 billion,  spread across 32,87,263 sq. km.; more than 35.16 per cent being illiterate (2001 census) and speaking 22 different languages, and following various religions
• The cost of the project without considering the recurring cost at around INR 15 lakh crore (US $33.33 billion) (Hindu, 2009)
• A guesstimate as reported by the  Frontline magazine put the cost of the project without considering the recurring cost at around INR 15 lakh crores (US $33.33 billion)
• The Indian government is expected to spend as much  as $250 billion over five years on programmes aimed at the poor, including subsidies for food, diesel, fertilizer, and jobs. But 40 per cent of the benefits, as the system now stands, will go to the wrong people or to "ghosts" with fake identification papers.
• The estimated cost of the Phase-I and Phase-II of the project spread over five years is Rs.3170.32crore. Of this amount, Rs.147.31 crore comprises the cost of Phase-I of the project and Rs.3023.01 crore is the estimated cost of Phase-II of the project.
• Around five megabytes of data will be required to store the compressed fingerprint images (of all the 10 fingers) of each individual, requiring the size of the entire database to be at least six petabytes (6,000 terabytes, or 6,000,000 gigabytes), making it among the world’s largest databases

-------------------------------------------------------
LINKS & REFERENCES

Dass, Rajanish (2012): Unique Identity Project in India: A Divine Dream or a Miscalculated Heroism? IIMA Working Papers, 4 March 2011

Jebaraj, Priscilla (2012): Double boost for Aadhaar: funds and PM's endorsement, The Hindu, 17 March 2012, URL - http://www.thehindu.com/news/national/article3003882.ece