The ever-growing stream of sensor information, photographs, text, voice and video data being created, and the new ways in which we can process data, are playing a major role in the process of digital transformation across all industries.
Companies are able to understand and anticipate what segment of customers will buy what product, and when. Big data also enables companies to manage their operations more efficiently. Big data is changing the world in other ways, not only in businesses:
Improving healthcare. By analysing a big volume of medical records and images, patterns that can help spot disease early and develop new medicines can be identified.
Predicting and responding to natural and human-made disasters. Data from sensors are being analysed to predict earthquakes and tsunamis. To improve the relief to survivors, organisations are looking for patterns of human behaviour. Big data technology is also being used to try to understand the complex dynamics of migration by compiling and analysing migration statistics.
Big Earth data. The advancements in remote sensing, social networking, high-performance simulation modelling and in-situ monitoring provide unprecedented big data about our planet. The large volume and variety of data offer an opportunity to better understand the Earth by extracting pieces of knowledge from these data.
Traditional sectors such as transport, health or manufacturing will greatly benefit from properly analysing and processing data, especially big data. Data has the potential to:
revamp Europe’s service industries by helping them generate innovative information products and services based on their customer’s preferences and needs
boost productivity by making available improved business intelligence to every sector
provide us with more thorough knowledge and understanding of how to address multiple societal challenges
advance research and speed up innovation
reduce costs by helping companies offer more personalised services or products
increase efficiency in the public sector
Some examples of big data usage are:
personalised marketing
sentiment analysis (product review)
consumer growth to guide product growth (consumer behaviour)
biomedical applications (personalised medicine and treatment)
big data-driven cities (smart cities)
Smart health: improving healthcare and public health
It took 10 years to decode the human genome. Today, the computing power of big data analytics enables us to decode entire DNA strings in minutes and it is believed that this will enable us to cure, understand and predict disease patterns. If all the individual data from smart watches and wearable devices is put together it can be applied to millions of people and their diseases. It can make clinical trials much more powerful and insightful.
For example, Apple's new health app, called ResearchKit, can turn your phone into a biomedical research device. Researchers can now collect data and input from users' phones to run health studies. Your device (like a smart watch) might track how many steps you take in a day, how you slept, ask cancer patients how they feel after a chemotherapy session, or track how someone’s Parkinson's disease is progressing. This can dramatically increase the volume of studies and trials as well as the veracity of the data.
In another example, big data techniques are being used to monitor premature babies. Premature newborns have to complete their maturation outside the mother's womb while their immune system is not completely operational, which increases the risk of infection. Quick diagnosis without invasive monitoring is crucial. Digi-NewB is working on a non-invasive monitoring system which will help doctors and nurses make better decisions in a shorter time to prevent infections. The system was developed based on a large database of over 700 newborns' recordings of vital signs data, clinical health records data, and data coming from video and sound.
Between technological advance and ethics
Big data opens the door to unprecedented insights and opportunities, but what is the other side of the coin? There are important concerns and questions that must be addressed to ensure our fundamental rights and democratic system are protected.
Data privacy. The volumes of data we generate contain a tremendous amount of information about our personal lives, which we should have the right to keep private. When data that should be kept private gets in the wrong hands, bad things can happen. A data breach at a school could put students’ personal identities in the hands of criminals who could commit identity theft. A breach at a hospital or doctor’s office can put patients’ information in the hands of those who might misuse it.
Data security. Even when we decide to share our data to monitor our health, for example, can we trust it is kept safe? Data is an incredibly important asset, and collecting and sharing data can be big business in today’s digital economy. With a heavier reliance on computers, there are a number of potential threats to the data being stored. Data can get lost due to system failure, corrupted by a computer virus, deleted or altered by a hacker.
Data discrimination. If everything is known, will it become acceptable to discriminate against people based on data we have on their lives? Like offering less advantageous medical insurance to overweight people? Or offering a higher interest credit to those whose parents had financial difficulties? Both banking and insurance are very data-driven and people can expect to be analysed and assessed in greater detail. How can we ensure that this won’t be done in a way that contributes to making life more difficult for those who already have less resources and access to information?
At the EU level, big data ethics is supported by regulation and documentation, which seeks to find concrete solutions to maximise the value of big data without sacrificing fundamental human rights. The European Data Protection Supervisor (EDPS) supports the right to privacy and the right to the protection of personal data in the respect of human dignity.
What exactly is personal data?
The General Data Protection Regulation (GDPR) reads as follows:
“’personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person.”
You can read more about personal data and the data protection rules on the GDPR page.
Considering and knowing more about these challenges is an important part of using big data. Ethics and individual rights must be imperative to organisations who want to take advantage of data. Failure to do so can leave our society in a vulnerable position, and damage businesses at a reputation, legal and financial level.
There are many other concerns regarding ethics and the use of data, such as the right to non-discrimination based on personal attributes, personal profiling and predictive behavioural analysis used to influence democratic processes, fairness and respect for human autonomy, and so on. Therefore, more than ever before, it is important for all of us to be aware of what, why and how data is being used in our society to be able to understand and recognise when data is misused.
Big data skills and the future of jobs
As we have seen, delving into data involves many processes, each of which demand unique skill sets. Working with big data requires both technical and non-technical skills, like the ability to operate the technologies that make it possible, an understanding of data analytics techniques, and the creativity and story-telling skills to create powerful data visualisations.
It requires the right skill set and mindset. You need to evaluate what is the best solution for the kind of data you have, what your needs are (meaning what you want to achieve by analysing that data), and choosing the right tools in order to create value. It is not just having the data, but having people who can find insights from it.
Big data professionals must also understand the legal and economic aspects of data, so they can turn insights into value and communicate frequently with executives and product managers about what they’ve learned and how that could be used towards new business directions such as new services and products, more efficient operational models or new markets.
That’s the key for working in the field: being curious, looking for new things to discover, creating and testing theories and finding patterns that allow you to predict results.
Eventually, all jobs of the future will require a mix of sector specific skills and data science competences, as data becomes an important driver for all activity sectors:
Entrepreneurs will use data to do behaviour analysis, variety and price optimisation, labour inputs optimisation, distribution and logistics optimisation and demand forecasting, but it also requires a sound industry and business knowledge to correctly implement a data-driven strategy.
Doctors analysing large data sets of patients’ information can help identify patients who are likely to suffer from a particular disease. Healthcare professionals also have a key role in helping to train brain scan AI with their expert feedback on the results.
Banking professionals can apply sentiment analysis and predictive analysis techniques to predict who are the potential customers and additionally offer targeted products to customers.
And so on.
No matter how advanced they are, data science and analytics do not remove the need for human insights. On the contrary, there is a compelling need for skilled people with the ability to understand data, but also add the sector point of view, and come up with insights.
Data engineers are some of the most sought-after data professionals
Data engineers are vital members of any enterprise’s data analytics team, responsible for managing, optimising, overseeing and monitoring data retrieval, storage and distribution throughout the organisation.
Data engineers are responsible for finding trends in data sets and developing algorithms to help make raw data more useful to the enterprise. This IT role requires a significant set of technical skills, including a deep knowledge of SQL database design and multiple programming languages. But data engineers also need communication skills to work across departments and understand what business leaders want to gain from the company’s large datasets.
Skills required for the job include: Apache Spark, Scala, Hadoop, Python, Hive, Amazon Web Services, Apache Kafka, Big Data, Extract/Transform/Load (ETL), SQL and Machine Learning. Sectors that hire these profiles: information technology and services, internet, automotive industry, computer software, management consulting, mechanical and industrial engineering, financial services, hospital and healthcare and telecommunications.
Some other job opportunities in the big data field are: data scientist, big data developer and data consultant.
