Spring Batch with Spring Boot

In this post I'm going to setup Spring Batch project with Spring Boot and this will use mysql data base with Spring Data as persistence API.
In this demonstration I will read CSV file and save the data in that file in to Database using Spring Batch.

Prerequisites 

1. You should have install java 1.8 or above.
2. You should have Eclipse installed in your PC.
3. Your PC should setup Maven installed and configured.

Lest Create the project


First of all open Eclipse IDE and go to File - > New -> Maven Project. Click on create simple project and press next. Then provide your project details. In my example im giving group Id as "com.nirmal.springbatch" and artifact Id as "SpringBatch". Then click finish.

First of all make sure to change run time JRE in to JDK. You can do it by right click on the project and go to Build path -> Configure Build path... then go to libraries and select JDK instead of JRE.

Spring Boot configurations

First of all lets add the dependencies in to POM file. Since Spring Boot remove complexity of project setup we just need to add Spring Boot references dependencies in to pom and do the required configurations based on application.properties file.

<project xmlns="http://maven.apache.org/POM/4.0.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>com.nirmal.springbatch</groupId>

<artifactId>SpringBatch</artifactId>

<version>0.0.1-SNAPSHOT</version>

<parent>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-parent</artifactId>

<version>1.3.2.RELEASE</version>

</parent>

<dependencies>

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-batch</artifactId>

</dependency>

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-web</artifactId>

</dependency>

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-data-jpa</artifactId>

</dependency>

<dependency>

<groupId>javax.servlet</groupId>

<artifactId>jstl</artifactId>

</dependency>

<dependency>

<groupId>mysql</groupId>

<artifactId>mysql-connector-java</artifactId>

<version>5.1.6</version>

</dependency>

<dependency>

    <groupId>com.h2database</groupId>

            <artifactId>h2</artifactId>

            <scope>runtime</scope>

        </dependency>

</dependencies>

<build>

<plugins>

<plugin>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

</plugin>

</plugins>

</build>

</project>

Then lets add required configurations based on application.properties file. Right click on src/main/resources add go to New -> Other select file and provide file name as "application.properties".

#==== connect to mysql ======#

spring.jpa.hibernate.ddl-auto=update

spring.datasource.url=jdbc:mysql://localhost:3306/sample?useSSL=false

spring.datasource.username=root

spring.datasource.password=

spring.datasource.driver-class-name=com.mysql.jdbc.Driver

spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

spring.jpa.properties.hibernate.show_sql=false

spring.jpa.properties.hibernate.use_sql_comments=false

spring.jpa.properties.hibernate.format_sql=true

#==== Logging configurations ======#

logging.level.root=WARN,INFO,ERROR

logging.level.com.baeldung=TRACE


Then lets add our Spring Boot initializer class in to the project. Right click on project and go to New -> Class and provide class name as "AppInitializer" and package I would give as "com.nirmal.springbatch".

I will add following content in to out initializer class.

package com.nirmal.springbatch;

import javax.sql.DataSource;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;

/**
 * 
 * Spring Boot initialization class of the SpringBatch project
 * 
 * @author Nirmal Balasooriya
 * 
 * 
 */

@ComponentScan({ "com.nirmal.springbatch" })
@EnableJpaRepositories("com.nirmal.springbatch")
@SpringBootApplication(scanBasePackages = { "com.nirmal.springbatch" })
public class AppInitializer extends SpringBootServletInitializer {

@Autowired
DataSource dataSource;

@Override
protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
return application.sources(AppInitializer.class);
}

public static void main(String[] args) {
SpringApplication.run(AppInitializer.class, args);
}

}

Setup the Data Models

Lets create out Book table using following SQL.

CREATE TABLE `sample`.`BOOK` ( `ID` INT NOT NULL AUTO_INCREMENT , `ISBMNUMBER` VARCHAR(250) NOT NULL , `NAME` VARCHAR(500) NOT NULL , `DESCRIPTION` VARCHAR(1024) NULL , `AUTHER` VARCHAR(250) NULL , PRIMARY KEY (`ID`)) ENGINE = MyISAM;

Then lets create model class for this Book. Right click on project and add new class with name as "Book" and package as "com.nirmal.springbatch.model".

package com.nirmal.springbatch.model;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;

/**
 * 
 * Model class for map the BOOK table using Hibernate
 * @author Nirmal Balasooriya
 * 
 */
@Entity
@Table(name = "book")
public class Book {
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
private int id;
@Column(name = "ISBMNUMBER")
private String isbmNumber;
private String name;
private String description;
private String auther;

/**
* 
* This is auto generated id from database
* @return the id
* 
*/
public int getId() {
return id;
}

/**
* 
* @param id the id to set
* 
*/
public void setId(int id) {
this.id = id;
}

/**
* 
* @return the isbmNumber
* 
*/
public String getIsbmNumber() {
return isbmNumber;
}

/**
* 
* @param isbmNumber the isbmNumber to set
* 
*/
public void setIsbmNumber(String isbmNumber) {
this.isbmNumber = isbmNumber;
}

/**
* 
* @return the name
* 
*/
public String getName() {
return name;
}

/**
* 
* @param name the name to set
* 
*/
public void setName(String name) {
this.name = name;
}

/**
* 
* @return the description
* 
*/
public String getDescription() {
return description;
}

/**
* 
* @param description the description to set
* 
*/
public void setDescription(String description) {
this.description = description;
}

/**
* 
* @return the auther
* 
*/
public String getAuther() {
return auther;
}

/**
* 
* @param auther the auther to set
* 
*/
public void setAuther(String auther) {
this.auther = auther;
}

/**
* 
* @param isbmNumber
* @param name
* 
*/
public Book(String isbmNumber, String name) {
super();
this.isbmNumber = isbmNumber;
this.name = name;
}

/**
* 
* @param isbmNumber
* 
* @param name
* @param description
* @param auther
* 
*/
public Book(String isbmNumber, String name, String description, String auther) {
super();
this.isbmNumber = isbmNumber;
this.name = name;
this.description = description;
this.auther = auther;
}

/**

* 

*/
public Book() {
super();
}

/*
* (non-Javadoc)
* 
* @see java.lang.Object#hashCode()
* 
*/
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = prime * result + ((isbmNumber == null) ? 0 : isbmNumber.hashCode());
return result;
}

/*
* (non-Javadoc)
* 
* @see java.lang.Object#equals(java.lang.Object)
* 
*/
@Override
public boolean equals(Object obj) {
if (this == obj)
return true;
if (obj == null)
return false;
if (getClass() != obj.getClass())
return false;
Book other = (Book) obj;
if (isbmNumber == null) {
if (other.isbmNumber != null)
return false;
} else if (!isbmNumber.equals(other.isbmNumber))
return false;
return true;
}

}

Creating  Spring Batch configurations

First of all lets create Spring processor class for our project. I will name the class as "BookProcessor" inside "com.nirmal.springbatch.processor" package. In this processor I will set Book name and Arther upper case.

package com.nirmal.springbatch.processor;


import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.nirmal.springbatch.model.Book;

/**
 * Intermediate processor class for the process books
 * @author User
 *
 */

public class BookProcessor implements org.springframework.batch.item.ItemProcessor<Book, Book > {

    private static final Logger log = LoggerFactory.getLogger(BookProcessor.class);


@Override
public Book process(Book book) throws Exception {
final String name = book.getName().toUpperCase();
        final String auther = book.getAuther().toUpperCase();

        final Book changedBook = new Book(book.getIsbmNumber(),name, book.getDescription(), auther);

        log.info("Converting (" + book + ") into (" + changedBook + ")");
        return changedBook;
}

}

Then lets create the Batch Configuration in the project.I will name the class as "BatchConfiguration" inside "com.nirmal.springbatch.config" package.
package com.nirmal.springbatch.config;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider;
import org.springframework.batch.item.database.JdbcBatchItemWriter;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;

import com.nirmal.springbatch.model.Book;
import com.nirmal.springbatch.processor.BookProcessor;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

@Autowired
    private JobBuilderFactory jobBuilderFactory;
     
    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Autowired
    private org.apache.tomcat.jdbc.pool.DataSource datasource; 
    
    @Bean
    public Job readCSVFilesJob() {
        return jobBuilderFactory
                .get("readCSVFilesJob")
                .incrementer(new RunIdIncrementer())
                .start(step1())
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1").<Book, Book>chunk(5)
                .reader(reader())
                .processor(new BookProcessor())
                .writer(writer())
                .build();
    }

    @SuppressWarnings({ "rawtypes", "unchecked" })
    @Bean
    public FlatFileItemReader<Book> reader()
    {
        FlatFileItemReader<Book> flatFileItemReader = new FlatFileItemReader<Book>();
         
        //Read input file from location
        flatFileItemReader.setResource(new FileSystemResource("input/inputData.csv"));
         
        //Set number of lines to skips. Use it if file has header rows.
        //reader.setLinesToSkip(1);  
         
        //Configure values using mapper
        flatFileItemReader.setLineMapper(new DefaultLineMapper() {
            {
                //3 read each columns from csv
                setLineTokenizer(new DelimitedLineTokenizer() {
                    {
                        setNames(new String[] { "isbmNumber", "name", "description", "auther" });
                    }
                });
                //Set values in Book class
                setFieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {
                    {
                        setTargetType(Book.class);
                    }
                });
            }
        });
        return flatFileItemReader;
    }
     
    /**
     * JDBCBatch Item writer
     * @return
     */
    @Bean
    public JdbcBatchItemWriter<Book> writer() {
        JdbcBatchItemWriter<Book> itemWriter = new JdbcBatchItemWriter<Book>();
        itemWriter.setDataSource(datasource);
        itemWriter.setSql("INSERT INTO BOOK (ISBMNUMBER, NAME, DESCRIPTION, AUTHER) VALUES (:isbmNumber, :name, :description, :auther)");
        itemWriter.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Book>());
        return itemWriter;
    }


}

reader()
This is the method used to read the input file from our class path and it contains the row mapper of the each line which convert each columns in to Book object parameters.

writer()
This method contains the insertion query which used to write the each Book object in to the database.

step1()
This is the method we define the our step which read file from input file and then process it and write back to database.

Now we are completed the batch application. In order to run the application we have to perform maven install command. (Write click on project and go to Run As -> Maven Install) Then lets create our input folder inside the target folder and add the inputData.csv with following content.


11111,Aaaa,DDDa,nirmal
22222,Bbbb,DescriptionB,balasooriya

Then you will be able to see the generated jar file on target folder. Go to that folder and execute the jar file using following command.
java -jar SpringBatch-0.0.1-SNAPSHOT.jar



Then you will be able to see similar output on command line as below image.

After run the jar file you will be able to see similar output as this

And also if you see in your database you will be able to see data has inserted in to the table. If you see the Name and Arther columns they have been saved as upper case which we did in our processor.

Data in CSV file uploaded in to 

You can access the source code on following GITHub location.

AOP with AspectJ in Spring Boot

In my previous blog post I have created Spring Boot REST API for simple CRUD operation and integrated junit unit tests for it. In this post I'm demonstrating how to use AspectJ for implement AOP in the REST API.

Prerequisites 

1. You should have install java 1.8 or above.
2. You should have Eclipse installed in your PC.
3. Your PC should setup Maven installed and configured.
4. You should have REST API project to implement AOP on top of it. I'm using REST api which I developed on previous post. (You can checkout previous code from here)

What is Aspect Orient Programing 

Aspect Orient Programming or AOP is a programming technique that aims to increase modularity by allowing the separation of cross-cutting concerns such as transaction management, logging etc. It does so by adding additional behavior to existing code without modification of the code itself. Instead, we declare separately which code is to modify.

AspectJ is currently the most complete implementation of an AOP language for Java. In practical terms, it is an extension to Java that treats AOP concepts as first-class elements of the language. It allows aspects to be implemented as part of a Java based application using familiar Eclipse based tools.

AOP concepts 

Before we go in to programming on AOP we have to have idea about terms which we use on AOP.

1. Aspect: An aspect is a class that implements enterprise application concerns that cut across multiple classes, such as transaction management, logging etc. Aspects is a normal class configured through Spring XML configuration or we can use Spring AspectJ integration to define a class as Aspect using @Aspect annotation.
2. Join Point: A join point is the specific point in the application such as method execution, exception handling, changing object variable values etc. In Spring AOP a join points is always the execution of a method.
3. Advice: Advices are actions taken for a particular join point. In terms of programming, they are methods that gets executed when a certain join point with matching pointcut is reached in the application. You can think of Advices as Struts2 interceptors or Servlet Filters.
4. Pointcut: Pointcut are expressions that is matched with join points to determine whether advice needs to be executed or not. Pointcut uses different kinds of expressions that are matched with the join points and Spring framework uses the AspectJ pointcut expression language.
5. Target Object: They are the object on which advices are applied. Spring AOP is implemented using runtime proxies so this object is always a proxied object. What is means is that a subclass is created at runtime where the target method is overridden and advices are included based on their configuration.
6. AOP proxy: Spring AOP implementation uses JDK dynamic proxy to create the Proxy classes with target classes and advice invocations, these are called AOP proxy classes. We can also use CGLIB proxy by adding it as the dependency in the Spring AOP project.
7. Weaving: It is the process of linking aspects with other objects to create the advised proxy objects. This can be done at compile time, load time or at runtime. Spring AOP performs weaving at the runtime.

Spring Boot dependency 

I have checkout my previous project from GIT repository and then add the dependency in to POM file related to the AOP programming. All required dependencies comes under the spring-boot-starter-test. Just need to add following part on the existing pom file.

        <dependency>

            <groupId>org.springframework.boot</groupId>

            <artifactId>spring-boot-starter-aop</artifactId>

        </dependency>

Lets define our Aspect 

In AspectJ we can define the aspect as a java class with @Aspect annotation. I have created new class "MyAspect" in "com.nirmal.springbootrest.aop" package. I will demonstrate following features in AspectJ in this tutorial. 

• @Before
• @After
• @AfterReturning
• @Around

I will here add the whole implementation of the MyAspect class and latter I will describe each method in more.

package com.nirmal.springbootrest.aop;

import org.slf4j.Logger;

import org.aspectj.lang.JoinPoint;

import org.aspectj.lang.ProceedingJoinPoint;

import org.aspectj.lang.annotation.After;

import org.aspectj.lang.annotation.AfterReturning;

import org.aspectj.lang.annotation.Around;

import org.aspectj.lang.annotation.Aspect;

import org.aspectj.lang.annotation.Before;

import org.slf4j.LoggerFactory;

import org.springframework.context.annotation.Configuration;

@Aspect

@Configuration

public class MyAspect {

private Logger logger = LoggerFactory.getLogger(this.getClass());

/**

* Before execution on

* 

* @param joinPoint

*/

@Before("execution(* com.nirmal.springbootrest.service.*.*(..))")

public void before(JoinPoint joinPoint) {

logger.info("MyAspect Before : execution for {}", joinPoint);

}

@AfterReturning(value = "execution(* com.nirmal.springbootrest.service.*.saveOrUpdateBook(..))", returning = "result")

public void afterReturning(JoinPoint joinPoint, Object result) {

logger.info("MyAspect AfterReturning : returned with value {}", joinPoint, result);

}

@After(value = "execution(* com.nirmal.springbootrest.service.*.deleteBook(..))")

public void after(JoinPoint joinPoint) {

logger.info("MyAspect After : execution for {}", joinPoint);

}

@Around("@annotation(com.nirmal.springbootrest.aop.TimeTrack)")

    public void around(ProceedingJoinPoint joinPoint) throws Throwable {

        long startTime = System.currentTimeMillis();

        joinPoint.proceed();

        long timeTaken = System.currentTimeMillis() - startTime;

        logger.info("MyAspect Around : execution for {}", joinPoint, timeTaken);

    }

}

@Before 

I have created method before with @Before  annotation in my aspect class. in there I mention execution point as "com.nirmal.springbootrest.service.*.*(..))". So this before method will execute every method under the package  "com.nirmal.springbootrest.service".

First of all lets start our application then we can test it by calling our get method by using following command in command line.

curl -X GET http://localhost:8080/findBook/9999

Then you can see the following output on your console.

@AfterReturning  

I have created method AfterReturning with @AfterReturning annotation in my aspect class. in there I mention execution point as "* com.nirmal.springbootrest.service.*.saveOrUpdateBook(..))". So this method will execute every method saveOrUpdateBook under the package  "com.nirmal.springbootrest.service" if method successfully executed.

we can test it by calling our get method by using following command in command line.

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{ \"isbmNumber\":\"9999\", \"name\":\"How to develop API\", \"description\":\"sample book\", \"auther\":\"Nirmal Balasooriya \" }" http://localhost:8080/saveOrUpdate

Then you can see similar output for following 

Actually in here two joint points found in our Aspect one in for before and other one is for AfterReturning.

@After  

I have created method After with @After annotation in my aspect class. in there I mention execution point as  "* com.nirmal.springbootrest.service.*.deleteBook(..))". So this after method will execute every method deleteBook under the package  "com.nirmal.springbootrest.service".

we can test it by calling our get method by using following command in command line.

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{ \"isbm\":\"9999\" }" http://localhost:8080/deleteBook

Then you can see similar output for following (Please note that since I have used same deleteBook method to simulate around annotation, you can see that method also executed)

@Around

For this we create Time Track annotation class in our project with following contents

package com.nirmal.springbootrest.aop;

import java.lang.annotation.ElementType;

import java.lang.annotation.Retention;

import java.lang.annotation.RetentionPolicy;

import java.lang.annotation.Target;

@Target(ElementType.METHOD)

@Retention(RetentionPolicy.RUNTIME)

public @interface TimeTrack {

}

And then I have added @TimeTrack annotation on the deleteBook class. Then I have created method around with @Around annotation in my aspect class. in there I mention execution point as  @annotation(com.nirmal.springbootrest.aop.TimeTrack). So this around method will execute every method which has TimeTrack annotation.

we can test it by calling our get method by using following command in command line.

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{ \"isbm\":\"9999\" }" http://localhost:8080/deleteBook

Then you can see similar output for following same as previous 

This will cover basic demonstration about AspectJ AOP with spring boot. You will be able to access the source code on following github location.