QUANTS: A shorthand name for “quantitative analysts”. At the pinnacle of geekdom lie those mathematical savants, prized by financial and web institutions, who make a living examining measurable and verifiable data in order to extrapolate predictions. That is, they create sophisticated formulas (formally known as algorithms) to analyze vast amounts of big data (both structured, that which can fit neatly into the columns and rows of relational database - like sales, shipments & addresses; and unstructured - such as customer complaints or blog comments). The results of these algorithms may dictate decisions (say, buying or selling stocks or adding or modifying product features), often in milliseconds (by using powerful computers). Most retailers now have a predictive analytics department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them. Moreover, the science of habit formation has become a major field of research in neurology and psychology departments at hundreds of major medical centers and universities, as well as inside extremely well financed corporate labs. Quants who specialize in consumer technology are sometimes known as WANTS because it is their job to troll through data, hunt for trends, identify types of personalities and lifestyles, all to put the right ad in front of the right person at the right time (what they want) and have them click on it and buy a product or service. Think of a bunch of guys in a room, parsing information about what you’ve searched for on Google, what you wrote about on g-mail, the websites you like to visit, the geostamps on your photos, the places you and your GPS-enabled phone or car have visited, where you have flown, what purchases or services you’ve charged and the like, all in order to hit you with a customized ad when you sign on to your browser. That’s what quants do.
This is becoming big business. Gartner says databases alone brought in $24 billion in 2011, and predicts a market surpassing $120 billion by 2015. And all of this requires powerful computers. Tracking, say, thousands of ankle bracelets of convicted criminals and their whereabouts (SecureAlert), patients’ biologies to detect cardiac or other threatening abnormalities (IRythm’s Zio patch), trucking company or ambulance vehicles in real time or retail sales (Amazon, eBay, Macy’s) and the like, mostly over the Internet, requires tremendous computing power.
The sea change occurred in 2006, when a software consultant working for Yahoo named Doug Cutting released an open-source system known as Hadoop (named after his son’s toy elephant, below left) which dramatically changed data analytics. Old-school business intelligence software required the loading of huge relational databases using rigid, slow schemas to sift through queries one at a time on a single mainframe computer. Big data analytics, on the other hand, handles variably structured data in real time from many sources without the delays inherent in the static schemas of the old relational databases. Rather than running massive data through one huge computer, it chops up the data into bite-sized chunks and distributes them among thousands of smaller (but still large) computers, then probes that pool of data for answers to various queries. As a result, companies can now, for example, look at recent point-of-sale transactions alongside clickstreams, online enrollment and social media chatter simultaneously to draw analytic conclusions about their customers, often in real time. This software became an instant hit at companies like Google, Yahoo!, Facebook (especially for the Like button), Walt Disney and Dell, which either used Hadoop or crafted their own software. Oracle, IBM, Cisco, EMC and Sun later joined the fray. A 2013 study found that 20% of businesses with 1,000 employees or more and at least 100 servers have implemented Hadoop, another 20% are in the process and more than 30% are considering it. In addition, dozens of startups like Datameer, MapR, Cloudera, HortonWorks, Splunk, Infoblox and ServiceNow are refining the software at a lower cost than the major players. And, in 2012, MemSQL claims to have developed database software 30 times faster than the others. Go to the Hadoop definition for more.
Want to see how this works for yourself? Click on this link to the Wolfram Alpha Facebook Data Tool, which can provide a detailed report that can analyze data about you and find patterns and stats about your online life or that of your friends. It’s free.
Quant analysis is not like the old number crunching. (Click HERE for more about the bell vs. bathtub curve.) It’s much more. For example, companies like RetailNext collect and integrate data from sales, surveillance video, RFID tags and motion sensors in order to detect not just how often a particular brand of cereal is picked up and purchased (or not), how long a purchaser stands in front of a particular display showing interest or purchasing, how often the shoppers turn left or right afterwards or when entering a store, how store traffic and patterns influence purchases, how the colors, size, shape and location of displays and product itself influence interest and purchases, how often they purchase the same product from an end cap, which size boxes are more popular, and a host of other variables which influence purchases. Add to this cellphone tracking technology like that of Path Intelligence (FootPath), which uses the phone’s cellular GPS signal to follow its owner throughout a store or building. Although Path is located in England, it has been tested in the U.S. in Richmond, Va., Temecula, CA and at J.C. Penney at one location, but it has been pretty much abandoned in the U.S. due to privacy concerns, even though signs were posted identifying the service and advising persons to shut off their phones if they wanted to opt out of the tracking. See also, beacons.
Taking this a step further is machine learning, in which computers with AI are learning on their own, eliminating the human evaluation of big data patterns and, instead, are performing neural analysis and adjusting the program algorithms as the data analysis evolves. This process is already in play, and is becoming more developed every day.
Not to be left out, the U.S. Government has purchased (or otherwise obtained) such software to track everyone’s moves on the Internet. Federal intelligence agencies such as NSA and DARPA have already spent millions to grab essentially every kind of data there is and to track the spread of ideas on networks such as Facebook to find people participating in persuasion campaigns and develop countermeasures, they say. For more, see Privacy,Are You Being Watched, Whistleblowers & NSA.