• +1 510-870-8668, 510-298-5936, 510-796-2400
  • Login

SparkMLlib

An AWS product Spark Mllib Hadoop Scala powered by Miri Infotech. MLlib is Spark's machine learning library, focusing on learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives.

We are launching a product which will configure and publish Spark MLlib, an open source software solution which is embedded pre-configured tool with Ubuntu OS and ready-to-launch AMI on Amazon EC2 that contains Spark MBlib, Hadoop 2.7, Scala, Linux, PHP (LAMP).

MLlib fits into Spark's APIs and interoperates with Scala. You can use any Hadoop data source (e.g. HDFS, HBase, or local files), making it easy to plug into Hadoop workflows.

Why MLlib? It is built on Apache Spark, which is a fast and general engine for large scale processing. Supposedly, running times or up to 100x faster than Hadoop MapReduce, or 10x faster on disk. Supports writing applications in Java, Scala, or Python.

MLlib contains many algorithms and utilities
  • Classification: logistic regression, naive Bayes
  • Regression: generalized linear regression, survival regression
  • Decision trees, random forests, and gradient-boosted trees
  • Recommendation: alternating least squares (ALS)
  • Clustering: K-means, Gaussian mixtures (GMMs)
  • Topic modeling: latent Dirichlet allocation (LDA)
  • Frequent itemsets, association rules, and sequential pattern mining
  • MLlib will still support the RDD-based API in spark.mllib with bug fixes.
  • MLlib will not add new features to the RDD-based API.
  • In the Spark 2.x releases, MLlib will add features to the DataFrames-based API to reach feature parity with the RDD-based API.
  • After reaching feature parity (roughly estimated for Spark 2.2), the RDD-based API will be deprecated.
  • The RDD-based API is expected to be removed in Spark 3.0.
  • DataFrames provide a more user-friendly API than RDDs. The many benefits of DataFrames include Spark Datasources, SQL/DataFrame queries, Tungsten and Catalyst optimizations, and uniform APIs across languages.
  • The DataFrame-based API for MLlib provides a uniform API across ML algorithms and across multiple languages.
  • DataFrames facilitate practical ML Pipelines, particularly feature transformations. See the Pipelines guide for details.
  • Data types
  • Classification and regression
  • Collaborative filtering
  • Clustering
  • Dimensionality reduction
  • Feature extraction and transformation

 

SparkMLlib live cast:
 

You can subscribe to an AWS Marketplace product and launch an instance from the product's AMI using the Amazon EC2 launch wizard.

To launch an instance from the AWS Marketplace using the launch wizard
  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/
  • From the Amazon EC2 dashboard, choose Launch Instance.
    On the Choose an Amazon Machine Image (AMI) page, choose the AWS Marketplace category on the left. Find a suitable AMI by browsing the categories, or using the search functionality. Choose Select to choose your product.
  • A dialog displays an overview of the product you've selected. You can view the pricing information, as well as any other information that the vendor has provided. When you're ready, choose Continue.
  • On the Choose an Instance Type page, select the hardware configuration and size of the instance to launch. When you're done, choose Next: Configure Instance Details.
  • On the next pages of the wizard, you can configure your instance, add storage, and add tags. For more information about the different options you can configure, see Launching an Instance. Choose Next until you reach the Configure Security Group page.
  • The wizard creates a new security group according to the vendor's specifications for the product. The security group may include rules that allow all IP addresses (0.0.0.0/0) access on SSH (port 22) on Linux or RDP (port 3389) on Windows. We recommend that you adjust these rules to allow only a specific address or range of addresses to access your instance over those ports.
  • When you are ready, choose Review and Launch.
  • On the Review Instance Launch page, check the details of the AMI from which you're about to launch the instance, as well as the other configuration details you set up in the wizard. When you're ready, choose Launch to select or create a key pair, and launch your instance.
  • Depending on the product you've subscribed to, the instance may take a few minutes or more to launch. You are first subscribed to the product before your instance can launch. If there are any problems with your credit card details, you will be asked to update your account details. When the launch confirmation page displays

About

MLlib is Spark's machine learning library, focusing on learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives.

MLlib fits into Spark's APIs and interoperates with Scala. You can use any Hadoop data source (e.g. HDFS, HBase, or local files), making it easy to plug into Hadoop workflows.

Guidelines

Development is open and available to any member of the Spark MLlib community. All fixes and improvements are done through pull requests to the code. This code is open source and publicly available.

Limitations

MLlib uses the linear algebra package Breeze, which depends on netlib-java, and jblas. netlib-java and jblas depend on native Fortran routines. You need to install the gfortran runtime library if it is not already present on your nodes. MLlib will throw a linking error if it cannot detect these libraries automatically. Due to license issues, we do not include netlib-java’s native libraries in MLlib’s dependency set under default settings. If no native library is available at runtime, you will see a warning message. To use native libraries from netlib-java, please build Spark with -Pnetlib-lgpl or include com.github.fommil.netlib:all:1.1.2 as a dependency of your project. If you want to use optimized BLAS/LAPACK libraries such as OpenBLAS, please link its shared libraries to /usr/lib/libblas.so.3 and /usr/lib/liblapack.so.3, respectively. BLAS/LAPACK libraries on worker nodes should be built without multithreading.

Usage / Deployment Instruction

Step1 : Open Putty for SSH

Step2 : Open Putty and Type <instanceID>> at “Host Name”

Step3 : Open Conncetion->SSH->Auth tab from Left Side Area

Step 4 : Click on browse button and select ppk file for Instance and then click on Open

Step 5 : Type "ubuntu" as user name Password auto taken from PPK file

Step 6 : Use following Linux command to start Hadoop

Step 6.1 : sudo vi /etc/hosts

Take the Private Ip address from your machine as per the below screenshot and then replace the second line of your command screen with that Private ip address

Step 6.2 : ssh-keygen -t rsa -P ""

This command is used to generate the ssh key.

Step 6.3 : cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

This command is used to move the generated ssh key to the desired location

Step 6.4 : ssh localhost

Step 6.5 : hdfs namenode –format

You have to write “yes” when it prompts you – Are you sure you want to continue?

Step 6.6 : start-all.sh

Step 6.7 : After the above command executes successfully, you should check the below urls in the browser -

http://<instanceId>:8088

http://<instanceId>:50070

http://<instanceId>:50090

Step 7 : Use following Linux command to start Scala and Spark

Step 7.1 : cd spark-2.1.0/

Step 7.2 : ./bin/spark-shell

Step 7.3 : You can check the spark by going on to the following url in your browser –

http://<instanceId>:4040

Step 7.4 : Now you can execute your scala programs as below –

 
Live Demo

Our Rating

5 star
0
4 star
0
3 star
0
2 star
0
1 star
0

Submit Your Request

First Name:*
Last Name:*
Company/Organisation:*
Email Address:*
Phone Number:*
Message:*