People often ask questions like “I’m still confused about the difference between data analyst, data engineer, and a data scientist. Can you give me an example of their work (not definitions)?” All these jobs – and a few more besides – are related to big data. Even as late as 2020 a survey by New Vantage Partners revealed that more than 90% executives said they were increasing their investment in Big Data. For the job market this means more jobs – but what exactly are those jobs and what would you expect to do? Let’s take a look.
What is big data?
If you operate within the IT industry, you probably know the answer to that question but for the uninitiated, here is a simplified version of what big data is. When unstructured data – think random information – arrives in large volumes from multiple sources at great speed, traditional databases and software are unable to handle it. This – in very simple terms – is big data. Big data requires specialized software and large storage spaces – a.k.a. databases.
What does big data contain?
About a decade ago when digitization was going viral, organizations and businesses began realizing they could gather vital information about consumers that could help them understand consumer wants, adapt their products and services to these wants and drive sales. So they began recording this data. That answers your questions about where big data comes from and why big data is important.
However the data came in different formats, at different speed, and above all in large volumes. Sorting through all this data, structuring, organizing, and analysing it, and then making sense of it became a humungous task that could not be handled by traditional databases and software – ergo Big Data software was developed.
But it also required a great deal of human effort and this gave rise to new job profiles, careers, and multiple jobs. Here are some of the top jobs in Big Data that you can hope to get if you are studying computer science or a taking a big data course.
Data Scientist
A data scientist mines – digs – through the voluminous and complex data. To do that he takes the help of a data analyst and data engineer. We’ll talk about these roles in a bit. For now suffice it to say that the data scientist receives the data in a relatively sorted and organized form. It is his job to run statistical analysis – such as deviation and regression analysis – on this data, interpret it, and make recommendations to organization about how they can utilize the information to advantage. Obviously, a data scientist must have knowledge not only about IT and software but also statistics. He must be able to translate statistical formulae into database instructions, understand the results, and then put his findings into words. Employers therefore look for a Masters or Ph.D in computer science when hiring data scientists. Obviously, too, they look for experience because the more you work with large volumes of data the more you learn about its vagaries. Similarly, the more you deal with big data the faster you can interpret it.
As far as salary is concerned, Glassdoor estimates an average salary of USD 108 thousand per annum and rates the role at 4.3. Of course, a lot depends upon your location and the technological advancement of the region but big data is fairly prevalent in most regions around the world. The job role does involve considerable stress as well as performance pressure – particularly in terms of speed.
Data Engineer
A data engineer is someone who facilitates the use, analysis, and interpretation of big data for others. The job role includes creating data structures and data sets, developing code for sorting through the data, developing processes, dashboards, and other tools to access the data to end users, and designing reports. It is the role of the data engineer to assist the data scientist by facilitating easy access to the data in an easily digestible format.
Employers hiring data engineers usually look for at least a bachelor’s degree in computer science, a B.Tech. or BE graduate. Once again, experience matters because speed is of paramount importance and an experienced person can do the job faster and more efficiently than a novice who may fumble at the basics. Some of the more popularly sought areas of expertise include knowledge about ad proficiency in designing and navigating SQL databases, knowledge of coding – such as knowledge of Python, java, PHP, Hive, Kafka, or Storm. Of course, given the rate and speed at which technology is advancing, it’s important to keep abreast of emerging trends. For instance SQL is being replaced by NoSQL in many applications. Apart from this as a data engineer you should also possess good written and verbal communication skills along with a few other soft skills such as leadership and presentation skills.
You can expect to earn about 106 thousand USD a year. The job satisfaction of a data engineer is rated – by Glassdoor – at 3.9 because of the extreme levels of concentration required for the job.
Data Analyst
This job profile ranks lower on the totem pole as far as Big Data jobs are concerned but still earns a decent income. The job scope includes collecting data – actionable information – about different aspects of the business. Data analysts do this by conducting large scale consumer surveys, tracking consumer behaviour, and techniques such as social listening. They present the data collected in the form of charts and reports as well as other digital formats to the data engineer who then gives it structure and meaning.
While you do not need a degree in computer science to become a data analyst, you should nevertheless have extensive knowledge about software like MS Excel, MS Access, PowerPoint, MS Word, SQL, SharePoint, and other data related software. A data analyst should also possess good communication skills and the ability to categorize and translate large volumes of complex data into structured information – albeit in the form of spread sheets or simple databases. Once again, the more experienced you are the better you would be suited to the job.
As we said, the job role of a data analyst does not require you to be highly qualified yet pays you a decent salary – Glassdoor estimates the average salary at USD 30 thousand per annum and rates the job satisfaction at 3.9 based on employee reviews.
Security Engineer
This is a significant role in any IT department. As the name suggests, the security engineer is responsible for the security of the data in the company’s database. Loss of even the smallest amount of data – or a leak – can mean a loss in millions for the business but that’s not the only reason to hire a data security engineer. These days, governements around the world are implementing data security laws and regulations to ensure that private data about consumers – such as social security number, aadhar card details, and national insurance number – is not compromised. It is the responsibility of the data security engineer – sometimes called the IT security officer – to minimize the exposure of data, protect the company data bases from infiltration and breach, and ensure compliance with various laws and regulations such as the GDPR (General Data Protection Regulation) in the UK. Specific tasks include building firewalls, ensuring that the antivirus software is up to date, conducting periodic site audits, and running security checks.
Employers prefer candidates having a bachelor’s degree in computer science, engineering or similar field. Again, experience matters because the more exposure you have the better you will be able to perform. Certification in specific industry security is an added advantage as is the knowledge of multiple coding language, operating systems, and databases.
As a data security engineer you can expect to earn USD 102 thousand per year. Glassdoor rates the job satisfaction at 3.8.
Database Manager or DBA
A Data Base Administrator (DBA) or database manager is someone who is trained in project management and diagnostics of data bases. Your role as a DBA will involve managing the database, ensuring its integrity – purity of data – and maintaining its structure. As a DBA you’ll work closely with the data analyst sorting the large volumes of data pour in through multiple sources and creating structures to store it. You will also be responsible for tracking singular elements and data records as well as ensuring that there is no loss of data. You’ll help design and install hardware for storage of data, set up cloud storage, and manage other aspects of storage.
Hiring managers usually ask for a bachelor’s degree in IT and a minimum of 5 years of experience. Knowledge and understanding of multiple databases such as Oracle, SAP, and AWS is a definite advantage as is knowledge about cloud databases and cloud computing in general. You should also be well versed in more than one database access tools such as MySQL if you want a decent chance to be selected for the role of DBA.
You can expect a pay package of over seventy thousand USD per year says Glassdoor.
Data Architect
Data architects organize and maintain databases. They are adept at developing data structures and strategies to store and retrieve data from computers. This may sound similar to the role of a DBA but there’s a subtle difference. A DBA does more than simply create a data structure and store data in it. He also possesses certain analytical skills and an understanding of how the data should be sorted and categorized so that it brings value to the business. A data architect performs the actual task of creating data structures and having the data fed into the computer under the guidance and instruction of the DBA.
A DBA or data engineer may be the one seeking to hire a data architect and may look for skills such as knowledge of SQL and XML along with a working knowledge of analytics, problem solving skills, attention to detail, and reasoning capabilities. Data architects usually come from the field of computer science but some employers may welcome diploma or certificate holders if you can demonstrate your capability and skill.
You can hope to earn upward of a hundred thousand USD (one lakh) per annum.
The above are essentially the main job roles in the area of Big Data. One other job role that you might want to consider if you want to work on the periphery of Big Data or IT is that of a technical recruiter. With your knowledge and understanding of IT you can help organizations source the right candidate for the right job – something that a recruiter lacking IT acumen may not be competent to do. As a knowledgeable recruiter, you will be better able to explain the job profile to candidates and assess their capability to fulfil the role for which you are hiring. There are a few other jobs that you may want to consider – such as Data Entry Operator (DEO) and tester also on the periphery of the industry.
The IT industry is vast in its scope encompassing much more than big data job roles. Indeed, entire organizations providing IT and other related services have a plethora of jobs on offer. BYN is here to review the various job roles in the IT and other industries for you bringing you updates and a wealth of information. Do join us on our FB community and sign up for our weekly newsletter. Let us know if you would like us to review a specific job role for you meanwhile stay tuned and stay connected. We are on LinkedIn, Twitter, Instagram, and Quora apart from FB – so don’t forget to follow us. Just type blogyourniches (don’t forget the s) into your search bar to find us.