taekyung - Tumblr blog

[taekyung]

MySQL gap lock, next key lock by example

MySQL gap lock, next key lock by example

MySQL’s gap lock or next key lock is a specific mechanism to guarantee it’s isolation level.

Gap lock is set on before or after values of the accessed record. Next key lock is the accessed row lock + gap lock. Therefore they have similar features.

Gap lock is activated in the following conditions.

Isolation level is Repeatable read (default value)

Storage engine is innodb

The query is executed in…

[taekyung]

The difference of Isolation level - Read Committed vs Repeatable Read

The difference of Isolation level – Read Committed vs Repeatable Read

MySQL’s default isolation level is “Repeatable Read“. But Oracle’s default isolation level is “Read Committed“. In this post, I will show you the difference by example.

Test environment : MySQL 5.7

Test table

create table lock_test( col1 varchar(10), col2 varchar(20), primary key(col1)); insert into lock_test values('1', 'A'); insert into lock_test values('2', 'B'); insert into lock_test…

[taekyung]

Kafka tip - keep the order of delivery

Kafka tip – keep the order of delivery

Kafka guarantees the order of delivery within a partition. That is, the order is not kept among partitions in a topic. There are some tips to keep the order of delivery in a topic.

1. One partition for a Topic

This is what Kafka basically supports.

<Pros>

Simple solution

<Cons>

A partition is applied to only one consumer, which has limitation in scale out

2. Custom partitioner

Partition number…

[taekyung]

Spring Batch tip - Partitioning

Spring Batch tip – Partitioning

Basically, Spring Batch Job runs in single thread. To increase the throughput, we need to parallelize the job by partitioning. The core architecture of Job partitioning is as follows.

Partition Step : The wrapper of common Step with a partitioner and grid size (or partition size)

Single Step : Common Step with a Reader, a Processor and a Writer

Partitioner : The core part of partitioning.…

[taekyung]

Spring Batch - Read once, Write multi example

Spring Batch – Read once, Write multi example

Sometimes, we need to read once, write multi ETL with Spring Batch. Spring Batch supports it with CompositeItemWriter (org.springframework.batch.item.support.CompositeItemWriter).

Sample table schema

CREATE TABLE "TB_SOURCE" ( "COL1" VARCHAR2(10 BYTE), "COL2" VARCHAR2(20 BYTE), "COL3" VARCHAR2(20 BYTE) ); CREATE TABLE "TB_TARGET1" ( "NEW_COL1" VARCHAR2(10 BYTE), "NEW_COL2" VARCHAR2(20…

[taekyung]

More on Spring Batch Writer

More on Spring Batch Writer

Spring Batch Writer is the implementation of org.springframework.batch.item.ItemWriter. You can write a custom writer, but Spring Batch already has some useful implementations, such as

org.springframework.batch.item.amqp.AmqpItemWriter for AMQP brokers such as RabbitMQ

org.springframework.batch.item.file.FlatFileItemWriter for a file

org.springframework.batch.item.database.JpaItemWriterfor…

[taekyung]

More on Spring batch Reader

More on Spring batch Reader

Spring Batch Job is composed of Reader, Processor and Writer. This post shows some more details on Reader.

Reader is the implementation of org.springframework.batch.item.Reader. I recommend to use the following implementations.

AmqpItemReader (org.springframework.batch.item.amqp.AmqpItemReader) : to interact with queue such as RabbitMQ

FlatFileItemReader(org.springframework.batch.item.file.FlatFi…

[taekyung]

Simple ETL with Spring Batch

Simple ETL with Spring Batch

Spring Batch is literally a batch framework based on Spring Framework. I usually use it to develop a simple ETL(Extraction, Transaformation and Loading) program.

In this post, I’ll show you how to write a simple ETL program. (This sample is tested on Spring Batch 3.0.10)

Prerequisites

Database (MySQL or Oracle)

Spring batch context database

Spring batch libraries (spring-batch-core, spring…

[taekyung]

A tip on using user transaction on Mybatis

A tip on using user transaction on Mybatis

When using transation on Mybatis, there are 3 ways.

Container managed transaction – JEE engine manages transaction

User managed transaction – transaction is controlled by user source code

Spring managed transaction – transaction is controlled by Spring Transaction policy

I usually use Spring managed or container managed transaction. But sometimes, I need to control transaction in source code. For…

[taekyung]

Some useful commands to manage Kafka

Some useful commands to manage Kafka

This post shows some useful commands to manage Kafka. (based on version 2.12)

Detailed information is on Kafka official documentation

Notice that some commands use zookeeper ip/port and others use kafka ip/port

Topic management commands List all topics

${KAFKA_HOME}/bin/kafka-topics.sh –list –zookeeper zookeeper_ip:port

Create a topic

${KAFKA_HOME}/bin/kafka-topics.sh –create –zookeeper zookeeper…

[taekyung]

At which point does Kafka consumer start to read

At which point does Kafka consumer start to read

At which point does  Kafka consumer start to read? It depends on the following scenarios.

Basic concept

Kafka can have several topics and each topic can have several partitions

Each partition keeps offset for each message

Offset starts from 0 and increases by 1 when a new message is published

Kafka server keeps the next offset to read for each consumer (or consumer group)

Kafka doesn’t have any…

[taekyung]

Managing Kafka log

Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer’s ack.

But it is anyhow a file, so it has storage limitation. Now I’m showing how to manage Kafka log. (The following contents are tested on version 2.12.)

Test environment

For test, I created a topic…

[taekyung]

Building a robust Zookeeper client : connection management

Building a robust Zookeeper client : connection management

To support high available Zookeeper service, we need to configure Zookeeper ensemble. But this is not enough. we need to understand how client manages connection to Zookeeper server.

For test, I set up a Zookeeper ensemble. (port : 2181, 2182, 2183) After then, write a test client with 2 important parameters – 1) connect string 2) session timeout

Case 1) connect string is targeting one zookeeper…

[taekyung]

Mockito thenReturn() vs thenAnswer()

Mockito thenReturn() vs thenAnswer()

Mockito is a very powerful test library for Java. In most cases, setting some mock action with thenReturn() is enough. But in some rare cases, we need to use thenAnswer().

thenReturn() example

First, let’s review the basic use of thenReturn()

public class TestTargetClass { public String getResultWithParam(int param) { return null; } public String getResultWithoutParam() { return null; } }

[taekyung]

How Kafka consumer works?

How Kafka consumer works?

Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture.

A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group)

Each group and client is identified by group id and client id. (which can be…

[taekyung]

How Kafka Topic failover works?

How Kafka Topic failover works?

After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication.

The conclusion in advance is that if a Topic’s replication factor is more than 2,

Kafka supports automatic leader failover

Data rebalance is supported only in manual operation

Test environment

Kafka 2.12

3 Kakfa brokers (Id : 0, 1, 2) on a PC

To create a Topic

${KAFKA_HOME}/bin/kafka-…

[taekyung]

Building Kafka cluster

Building Kafka cluster is crucial for the production system.

Kafka cluster gives the following advantages.

Support for failover in case of a node down

Queue replication

Support for consumer scale out (standalone Kafka also supports consumer scale out)

Step 1 – Build Zookeeper ensemble

Kafka depends on Zookeeper for it’s configuration management. Therefore, Zookeeper needs to run in cluster. (Ref…