[Issue]
Apache Storm과 Apache Kafka를 연동하기 위해, KafkaSpout 라이브러리를 사용하였다.
이때, Storm 토폴로지는 정상적으로 제출(submit)되었지만 메시지가 Kafka에서 Storm으로 넘어오지 않는 문제가 발생하였다.
Spout의 로그를 확인해보니 아래와 같은 에러메시지가 출력되어 있었다.
에러는 Log4j 클래스를 찾지 못한다는 내용이다.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | 2017-10-27 18:18:37.310 o.s.h.Util Thread-18-kafka-spout-executor[15, 15] [ERROR] SLF4J: Detected both log4j-over-slf4j.jar AND bound slf4j-log4j12.jar on the class path, preempting StackOverflowError. 2017-10-27 18:18:37.310 o.s.h.Util Thread-18-kafka-spout-executor[15, 15] [ERROR] SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details. 2017-10-27 18:18:37.311 o.a.s.u.Utils Thread-18-kafka-spout-executor[15, 15] [ERROR] Async loop died! java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory at org.apache.log4j.Logger.getLogger(Logger.java:39) ~[log4j-over-slf4j-1.6.6.jar:1.6.6] at kafka.utils.Logging$class.logger(Logging.scala:24) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.utils.Logging$class.info(Logging.scala:77) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:97) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:86) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:152) ~[stormjar.jar:?] at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:81) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:81) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:71) ~[stormjar.jar:?] at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:135) ~[stormjar.jar:?] at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:108) ~[stormjar.jar:?] at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:130) ~[stormjar.jar:?] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:154) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at org.apache.storm.utils.Utils$6.run(Utils.java:2207) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] 2017-10-27 18:18:37.315 o.a.s.e.e.ReportError Thread-18-kafka-spout-executor[15, 15] [ERROR] Error java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory at org.apache.storm.utils.Utils$6.run(Utils.java:2217) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory at org.apache.log4j.Logger.getLogger(Logger.java:39) ~[log4j-over-slf4j-1.6.6.jar:1.6.6] at kafka.utils.Logging$class.logger(Logging.scala:24) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.utils.Logging$class.info(Logging.scala:77) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:38) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:97) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:86) ~[stormjar.jar:?] at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:152) ~[stormjar.jar:?] at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:81) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:81) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:71) ~[stormjar.jar:?] at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:135) ~[stormjar.jar:?] at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:108) ~[stormjar.jar:?] at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[stormjar.jar:?] at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:130) ~[stormjar.jar:?] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:154) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at org.apache.storm.utils.Utils$6.run(Utils.java:2207) ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] ... 1 more 2017-10-27 18:18:37.376 o.a.s.u.Utils Thread-18-kafka-spout-executor[15, 15] [ERROR] Halting process: Worker died java.lang.RuntimeException: Halting process: Worker died at org.apache.storm.utils.Utils.exitProcess(Utils.java:1819) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at org.apache.storm.utils.Utils$4.run(Utils.java:1827) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at org.apache.storm.executor.error.ReportErrorAndDie.uncaughtException(ReportErrorAndDie.java:46) [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT] at java.lang.Thread.dispatchUncaughtException(Thread.java:1956) [?:1.8.0_102] | cs |
[Solved]
에러는 Kafka 라이브러리의 Log4j와 Storm의 Log4j가 충돌이 나기 때문이다.
따라서 Kafka 라이브러리에서 Log4j를 제외시켜, Storm은 Storm의 Log4j만 사용하도록 하자.
기존에 pom.xml에 등록한 Kafka 라이브러리는 다음과 같다.
1 2 3 4 5 6 7 | <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11 --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>0.11.0.1</version> </dependency> | cs |
이를 다음과 같이 exclusions을 추가하였다. (6번 라인~ 19번 라인)
즉, Kafka 라이브러리에서 Storm과 충돌이 일어나는 라이브러리는 제외시켰다.
그 결과 Storm이 Kafka로부터 정상적으로 메시지를 수신하였다.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.11 --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>0.11.0.1</version> <exclusions> <exclusion> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> | cs |
[참고]
Storm 1.0 on HDP 2.5: java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory, https://community.hortonworks.com/questions/56204/storm-10-on-hdp-25-javalangnoclassdeffounderror-co.html
'빅데이터 > Storm' 카테고리의 다른 글
| Apache Storm maven build (0) | 2019.05.21 |
|---|---|
| How to run storm-perf (Storm benchmark tool) (0) | 2019.05.21 |
| Apache Storm 데몬 중지 명령어 (kill storm) (0) | 2017.10.19 |