Krutik Jayswal

Apache spark is an interesting big data framework using which we can do amazing things when we are dealing with big data.In journey of learning spark many developers face difficulties when to use java and when to use spark framework.There where plenty of scenarios In the journey of development there scenarios where it was effcient/effective to use java instead of using spark.

Spark is needed in a place where there are complex and manipulative operation on large dataset.A simple case where only lifting and shiftin of data is there in that case it can be achieved using other technologies also.Let's review the certain pointers which I have learned during the spark application development.

Packging spark application using Java

Using maven shade plugin is not encouraged, unless and until there is no other option apart from using the maven shade plugin. Below is the informative link for not to use the shade plugin.

Downsides of using Shade plugin relocation feature

From my perspective for running spark application, spring boot maven plugin is the best option for managing everything, including library dependencies. Even if your application is used as a scheduled job, in which you need to use spark-submit/spark launcher for launching the application.

In other case where your java app using spark and spring both but have controller/api to use the application in that case also spring boot maven plugin is the best one.

There are only 2 types of Challanges when we use spark-submit/spark launcher to launch the application which is created using spring boot maven plugin.

1. Main Class

When we package fat jar/uber jar using spring boot maven plugin, it is packging the class files and java libraries in spring way and not the way sprak is expecting it.Inside the uber jar generated by the spring boot maven plugin, we have boot-inf,meta-inf and org folder.So when we give mainclass in spark submit or spark launcher as a parameter,It will not able to find that class as it will not be able to locate package/path specified in parameter, due to change of structure in jar file.Even after you specify the correct location which is starting with BOOT-INF for main class, it will not work because the way spring launches the application is using differrent main class.

Below is the link which shows main class that should be used for launching the fat jar generated by spring boot maven plugin.

https://docs.spring.io/spring-boot/docs/current/reference/html/executable-jar.html#appendix.executable-jar.launching

On high level if I inform ,File named as MANIFEST.MF inside the uber jar contains below entries.Where Main-Class is the actual main class which is used for initialization of spring related stuff, after which customized main class should get started which is Start-Class entry.

Main-Class: org.springframework.boot.loader.JarLauncher Start-Class: com.mycompany.project.MyApplication

So as a conclusion specifiying main class as "org.springframework.boot.loader.JarLauncher" in spark-submit or spark-launcher will resolve our issue for this problem.This will only work if you are using sparing boot maven plugin for packagin the jar.

External common libraries used in pom.xml+spark installation. Another issue which might occure while using spark-submit or launcher application while launching uber jar, packaged using the spring boot maven plugin is jar conflicts.So the problem will be when we package jar using spring boot maven plugin, it is coping depedencies inside BOOT-INF/lib folder, let's say if you are using below depedency in pom.xml.

<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.10</version>
</dependency>

Now let's say this depedency already exist with differrent version in spark installation, in that case it will give conflict in classes.As it will be an issue of class loader ,its better not to use those which might create conflict due to which application might fail.I have faced this kind of issues related to logger classes or json libraries, as there are multiple json/logger library options are available.As a resolution you can excluded those classes or libraries or you can replace that library with alternative one.

Deployment/Triggering Spark Applciation

There are plenty of ways using which we can trigger or deploy the application in spark server.We will see two scenarios using which we can use the spark in the application development.

Deploy as Job In kubernates cluster

We can deploy the spark application as a job in kubermates cluster.If we choose to deploy application using this approach , in that case, main method will be triggered if you are using java.Because that is the entry point of the application.The application will not be running 24*7 , we can schdule this application in kubernates cluster as well to periodically trigger the job.In this case we can use below command to trigger the spark job.

 
  
  ./bin/spark-submit \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key<=<value> \
  --driver-memory <value>g \
  --executor-memory <value>g \
  --executor-cores <number of cores>  \
  --jars  <comma separated dependencies>
  --class <main-class> \
  <application-jar> \
  [application-arguments]

Deploy as Rest In kubernates cluster

Another way to trigger the business logic of the spark aplication is to have one rest application deployed and running in the cluster.Using that application we will be ablt to trigger the business logic or the task.In this case we can pass the job parameter as the url parameter in to the rest api application.Below is sample code for same.

 

curl -X POST [http://sparkendpoint.com]/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "appResource": "file:/home/user/spark_pi.py",
  "sparkProperties": {
    "spark.executor.memory": "8g",
    "spark.master": "spark://192.168.1.1:7077",
    "spark.driver.memory": "8g",
    "spark.driver.cores": "2",
    "spark.eventLog.enabled": "false",
    "spark.app.name": "Spark REST API - PI",
    "spark.submit.deployMode": "cluster",
    "spark.driver.supervise": "true"
  },
  "clientSparkVersion": "2.4.0",
  "mainClass": "org.apache.spark.deploy.SparkSubmit",
  "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
  "action": "CreateSubmissionRequest",
  "appArgs": [ "/home/user/spark_pi.py",  "80" ]
}'

This is one of the easiest and interesting design pattern among other creational design pattern. With use of builder design pattern we can create and object by passing several optional parameters. If the object creation is complex and it requires several parameters for creating an object in that case we can use the builder pattern for creating an object.

Idea of using the builder pattern is we initially create the blank object with no parameters or with the minimum number of parameter and letter on we will add the optional parameter whenever required and will return the created object. In java StringBuffer and StringBuilder objects are created using the builder pattern.

As image describes using HouseBuilder class we can directly create the object of furnished or unfurnished house without exposing the complex object creation login. Let's check the example with code.

House.java

 package com.design.pattern.builder;

public class House {
	private String window;
	private String door;
	private String bathroom;
	private String bedroom;
	private String furniture;

	public String getWindow() {
		return window;
	}

	public void setWindow(String window) {
		this.window = window;
	}

	public String getDoor() {
		return door;
	}

	public void setDoor(String door) {
		this.door = door;
	}

	public String getBathroom() {
		return bathroom;
	}

	public void setBathroom(String bathroom) {
		this.bathroom = bathroom;
	}

	public String getBedroom() {
		return bedroom;
	}

	public void setBedroom(String bedroom) {
		this.bedroom = bedroom;
	}

	public String getFurniture() {
		return furniture;
	}

	public void setFurniture(String furniture) {
		this.furniture = furniture;
	}

	@Override
	public String toString() {
		return "House has "+this.bedroom+", " +this.bathroom+", "+this.getDoor()+", "+this.getWindow()+ (this.getFurniture()!=null?", "+this.getFurniture():"");
	}
}

FurnishedHouseBuilder.java

 package com.design.pattern.builder;

public class FurnishedHouseBuilder {

	private House house;
	
	public House getHouse() {
		return house;
	}

	public void setHouse(House house) {
		this.house = house;
	}

	public FurnishedHouseBuilder() {
		house = new House();
	}

	public FurnishedHouseBuilder buildWindow() {
		house.setWindow("window");
		return this;
	}

	public FurnishedHouseBuilder buildDoor() {
		house.setDoor("door");
		return this;
	}

	public FurnishedHouseBuilder buildBedroom() {
		house.setBedroom("bedroom");
		return this;
	}

	public FurnishedHouseBuilder buildBathRoom() {
		house.setBathroom("bathroom");
		return this;
	}

	public FurnishedHouseBuilder buildFurniture() {
		house.setFurniture("furniture");
		return this;
	}
}

UnfurnishedHouseBuilder.java

 package com.design.pattern.builder;

public class UnfurnishedHouseBuilder {

	private House house;

	public House getHouse() {
		return house;
	}

	public void setHouse(House house) {
		this.house = house;
	}

	public UnfurnishedHouseBuilder() {
		house = new House();
	}

	public UnfurnishedHouseBuilder buildWindow() {
		house.setWindow("window");
		return this;
	}

	public UnfurnishedHouseBuilder buildDoor() {
		house.setDoor("door");
		return this;
	}

	public UnfurnishedHouseBuilder buildBedroom() {
		house.setBedroom("bedroom");
		return this;
	}

	public UnfurnishedHouseBuilder buildBathRoom() {
		house.setBathroom("bathroom");
		return this;
	}

}

HouseBuilder.java

 package com.design.pattern.builder;

public class HouseBuilder {
	public House buildFurnishedHouse() {
		return new FurnishedHouseBuilder().
				buildBathRoom().
				buildBedroom().
				buildDoor().
				buildWindow().
				buildFurniture().
				getHouse();
	}
	
	public House buildUnfurnishedHouse() {
		return new UnfurnishedHouseBuilder().
				buildBathRoom().
				buildBedroom().
				buildDoor().
				buildWindow().				
				getHouse();
	}
}

HouseBuilderDemo.java

 package com.design.pattern.builder;

public class HouseBuilderDemo {
	public static void main(String[] args) {
		House furnishedHouse=new HouseBuilder().buildFurnishedHouse();
		House unfurnishedHouse=new HouseBuilder().buildUnfurnishedHouse();
		
		System.out.println("Furnished House : "+furnishedHouse );
		System.out.println("Unfurnished House : "+unfurnishedHouse );
	}
}

Output:

Abstract factory design pattern is factory of factory design pattern. It is a creational design pattern in which we create multiple factory class and for creating factory object as well there will be main factory class. This is most complicated design pattern among other creational design patterns. Its better to understand factory pattern first as abstract factory is enhanced version of factory pattern. To simplify things, I will say we will be having one super factory class which will be responsible for creating factory object among several factory classes.

In abstract factory design pattern , we have multiple factory classes which are responsible for object creation. This factory classes are segregated based on the object categorization. Like in our case we have divided the object based on operating system of mobile device and then created factory classes for same. So there are two factory classes in our case Android Device Factory and Apple Device Factory, both of them are created from Abstract Factory class.so we will be using this abstract factory class only for creating objects.

Object which we will be creating is a Mobile Device , so the base interface will be MobileDevice using which the concret class will be created for actual mobile devices , In our case we have created devices like Android Device1,Apple Device 1 etc..

Let's cut the theory out and make some practical :)

MobileDevice.java

 package com.design.pattern.abstract_factory;

public interface MobileDevice {
	public void printDeviceType();
}

AndroidDevice1.java

 package com.design.pattern.abstract_factory;

public class AndroidDevice1 implements MobileDevice {

	@Override
	public void printDeviceType() {
		System.out.println("This is Android Device 1");
	}

}

AndroidDevice2.java

 package com.design.pattern.abstract_factory;

public class AndroidDevice2 implements MobileDevice {

	@Override
	public void printDeviceType() {
		System.out.println("This is Android Device 2");
	}

}

AppleDevice1.java

 package com.design.pattern.abstract_factory;

public class AppleDevice1 implements MobileDevice {

	@Override
	public void printDeviceType() {
		System.out.println("This is Apple Device 1");
	}

}

AppleDevice2.java

 package com.design.pattern.abstract_factory;

public class AppleDevice2 implements MobileDevice {

	@Override
	public void printDeviceType() {
		System.out.println("This is Apple Device 2");
	}

}

AbstractFactory.java

 package com.design.pattern.abstract_factory;

public abstract class AbstractFactory {
	abstract MobileDevice getMobileDevice(String deviceName) ;
}

AndroidDeviceFactory.java

 package com.design.pattern.abstract_factory;

public class AndroidDeviceFactory extends AbstractFactory {
	

	@Override
	MobileDevice getMobileDevice(String deviceName) {
		if(deviceName.equals("AndroidDevice1")) {
			return new AndroidDevice1();
		}else if(deviceName.equals("AndroidDevice2")) {
			return new AndroidDevice2();
		}else {
			return null;
		}
	}

}

AppleDeviceFactory.java

 package com.design.pattern.abstract_factory;

public class AppleDeviceFactory  extends AbstractFactory {

	@Override
	MobileDevice getMobileDevice(String deviceName) {
		if(deviceName.equals("AppleDevice1")) {
			return new AppleDevice1();
		}else if(deviceName.equals("AppleDevice2")) {
			return new AppleDevice2();
		}else {
			return null;
		}
	}

}

DeviceFactoryProducer.java

 package com.design.pattern.abstract_factory;

public class DeviceFactoryProducer {
	public static AbstractFactory getDeviceFactory(boolean needOpenSourceOS) {
		if (needOpenSourceOS) {
			return new AndroidDeviceFactory();
		} else {
			return new AppleDeviceFactory();
		}
	}
}

AbstractFactoryDemo.java

 package com.design.pattern.abstract_factory;

public class AbstractFactoryDemo {
	public static void main(String[] args) {
		AbstractFactory openSourceDeviceFactory=DeviceFactoryProducer.getDeviceFactory(true);
		
		MobileDevice android1=openSourceDeviceFactory.getMobileDevice("AndroidDevice1");
		android1.printDeviceType();
		
		MobileDevice android2=openSourceDeviceFactory.getMobileDevice("AndroidDevice1");
		android2.printDeviceType();
		
		AbstractFactory closedSourceDeviceFactory=DeviceFactoryProducer.getDeviceFactory(false);
		
		MobileDevice apple1=closedSourceDeviceFactory.getMobileDevice("AppleDevice1");
		apple1.printDeviceType();

		MobileDevice apple2=closedSourceDeviceFactory.getMobileDevice("AppleDevice2");
		apple2.printDeviceType();
	}
}

Factory design pattern is creational design pattern , this design pattern is used whenever developer does not need to expose the object creational logic. In factory pattern objects are created using the factory class. This class will be having the object creation part. So when ever the new object is needed of specific type we can create the object using the factory class. Below diagram shows the basic structure of factory pattern.

One of the example where we can use this design pattern is in a case some predefined parameters are there for creating object. So what we can do in our case is , we can pass the type of object in getShape() method and based on the type with default attributes we can return the required object.

Let's see some practical example of it.

Shape Class

package com.design.pattern.factory;

public interface Shape {
	void draw();
}

Circle Class

package com.design.pattern.factory;

public class Circle implements Shape {

	@Override
	public void draw() {
		System.out.println("Drawing circle...");
	}

}

Triangle Class

package com.design.pattern.factory;

public class Triangle implements Shape {

	@Override
	public void draw() {
		System.out.println("Drawing triangle...");
	}

}

Rectangle Class

package com.design.pattern.factory;

public class Rectangle  implements Shape {

	@Override
	public void draw() {
		System.out.println("Drawing rectangle...");
	}

}

ShapeFactory Class

package com.design.pattern.factory;

public class ShapeFactory {

	public static final String CIRCLE="circle";
	public static final String TRIANGLE="triangle";
	public static final String RECTANGLE="rectangle";
	
	public Shape getShape(String shapeType) {
		if (shapeType == null) {
			return null;
		}
		if (shapeType.equalsIgnoreCase(CIRCLE)) {
			return new Circle();

		} else if (shapeType.equalsIgnoreCase(RECTANGLE)) {
			return new Rectangle();

		} else if (shapeType.equalsIgnoreCase(TRIANGLE)) {
			return new Triangle();
		}

		return null;
	}

}

FactoryDemo Class

package com.design.pattern.factory;

public class FactoryDemo {

	public static void main(String[] args) {

		ShapeFactory shapeFactory = new ShapeFactory();

		Shape shape1 = shapeFactory.getShape(ShapeFactory.CIRCLE);
		Shape shape2 = shapeFactory.getShape(ShapeFactory.TRIANGLE);
		Shape shape3 = shapeFactory.getShape(ShapeFactory.RECTANGLE);
		shape1.draw();
		shape2.draw();
		shape3.draw();

	}
}

Prototype design pattern is creational design pattern. Developers needs to use this design pattern when the object creation is costlier in terms o resources or by some other means. One of the example of costly object creation is that , while creating object we need to make some database calls or may be some webservice calls. In this case we should use prototype design pattern. This pattern is doing nothing special , what it does is it is cashing the new created object and whenever required it will clone this cashed object and will return that copy to the client.

We will be having one abstract class in which we will be implementing the cloneable interface. The reason behind implementing the cloneable interface is that, that every subsequent class which will be extending this base abstract class will be using this base method implementation from abstract class. In above mentioned class we have the abstract class named as Shape so we will implement clone method in that class.

One more things which needs to be done is, we need to create each type of object and cache it in some kind of store like HashMap or HashTable and whenever required we can return the required type of object by cloning the cached copy. Let's see some practical example now.

Abstract Shape Class

public abstract class Shape implements Cloneable {

	public  Object clone() {
		Object clone = null;
		try {
			clone = super.clone();
		} catch (CloneNotSupportedException e) {
			e.printStackTrace();
		}
		return clone;
	}

	abstract String displayType();

}

Circle Class

package com.design.pattern.prototype;

public class Circle extends Shape {

	private String type;

	public Circle() {
		this.type = "Circle";
	}
	
	public String displayType() {
		return "This is "+type+" of object";
	}
}

Rectangle Class

package com.design.pattern.prototype;

public class Rectangle extends Shape {
	private String type;

 	public Rectangle() {
		this.type = "rectangle";
	}
	
	public String displayType() {
		return "This is "+type+" of object";
	}
}

Triangle Class

package com.design.pattern.prototype;

public class Triangle extends Shape{
	 	private String type;
	 	
	 	public Triangle() {
			this.type = "Triangle";
		}
	 	
		public String displayType() {
			return "This is "+type+" of object";
		}
}

ShapeStore Class

package com.design.pattern.prototype;

import java.util.HashMap;
import java.util.Map;

public class ShapeStore {
	
	public static Map cachedObjects=new HashMap<>();
	
	public static void loadObjects() {
		cachedObjects.put("circle", new Circle());
		cachedObjects.put("triangle", new Triangle());
		cachedObjects.put("rectangle", new Rectangle());
	}
	
	public static Shape getShape(String shape) {
		return (Shape) cachedObjects.get(shape).clone();
	}
	
}

PrototypeDemo Class

package com.design.pattern.prototype;

public class PrototypeDemo {

	public static void main(String[] args) {

		ShapeStore.loadObjects();
		Shape circle=ShapeStore.getShape("circle");
		Shape triangle=ShapeStore.getShape("triangle");
		Shape rectangle=ShapeStore.getShape("rectangle");
		
		System.out.println(circle.displayType());
		System.out.println(triangle.displayType());
		System.out.println(rectangle.displayType());
	}
}

Subscribe to: Posts ( Atom )

Navigation

Krutik Jayswal

Things to know when you are dealing with Apache Spark

Builder Design Patterns

Abstract Factory Design Pattern

Factory Design Patterns

Prototype Design Pattern

Instagram