Google Cloud Dataproc: Launch Hadoop & Spark Cluster in Google Cloud Platform

Dataproc is fast, fully managed and easy to use Cloud Services for running Apache Spark and Hadoop clusters. It lets you take advantage of open-source data tools for batch processing, querying, streaming and machine learning.

What is Spark?

Apache Spark is a fast, open-source,  and general-purpose engine for large scale data processing for Big Data Workloads. You can write code and it will automatically parallelize itself on top of Hadoop.

What is Hadoop?

Hadoop is a software library or you can say a collection of open-source software which is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

Features 

• Super Fast: Dataproc clusters are quick to start, scale, and shutdown, with Spark and Hadoop clusters, taking 90 seconds or less, on average. That means you can spend less time waiting for clusters and more hands-on time working with your data.

  
  

• Automated Management: Use Spark and Hadoop clusters without the assistance of an administrator or any software. Automatically managed deployment, signing and monitoring let you focus on your data, not on your cluster. Easily interact with clusters and Spark or Hadoop through the Google Cloud Console, the Cloud SDK, or the Dataproc REST API. Pay only on the basis of  Cluster usage, pay-as-you-go model. Need not worry about losing data because Dataproc is integrated with Cloud Storage, BigQuery, and Cloud Bigtable.

  
  

• Low Cost: Pricing at only one cent per virtual machine in your cluster per hour, on top of the other Cloud Platform resources you are using. In addition to this low price, It can include preemptible instances that have lower compute prices, reducing your costs even more.

  
  

• Autoscaling Clusters: This feature provides a mechanism for automating cluster resource management, and enables automatic addition and subtraction of cluster workers (nodes).

  
  

• Cloud Integrated: It has inbuilt integration with Google Cloud Platform services, such as Cloud Storage, BigQuery, Cloud Bigtable, and Cloud Monitoring, so you have more than just a Spark or Hadoop cluster which implies you have a complete data platform.

  
  

• Flexible and Familiar: Clusters can use custom machine types and preemptible virtual machines to make them the perfect size for your needs. You don’t need to learn new things or API’s to use Dataproc. Easy to move existing workloads into Dataproc without redevelopment.