栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

sqoop从mysql导数据到hdfs使用lzop压缩格式,报:NullPointerException

sqoop从mysql导数据到hdfs使用lzop压缩格式,报:NullPointerException

具体报错如下:

Error: java.lang.NullPointerException
        at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:63)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2021-12-21 11:14:28,253 INFO mapreduce.Job: Task Id : attempt_1639967851440_0006_m_000000_1, Status : FAILED
Error: java.lang.NullPointerException
        at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:63)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2021-12-21 11:14:31,265 INFO mapreduce.Job: Task Id : attempt_1639967851440_0006_m_000000_2, Status : FAILED
Error: java.lang.NullPointerException
        at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:63)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2021-12-21 11:14:36,283 INFO mapreduce.Job:  map 100% reduce 0%
2021-12-21 11:14:37,292 INFO mapreduce.Job: Job job_1639967851440_0006 failed with state FAILED due to: Task failed task_1639967851440_0006_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2021-12-21 11:14:37,344 INFO mapreduce.Job: Counters: 9
        Job Counters
                Failed map tasks=4
                Launched map tasks=4
                Other local map tasks=3
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=11344
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=5672
                Total vcore-milliseconds taken by all map tasks=5672
                Total megabyte-milliseconds taken by all map tasks=5808128
2021-12-21 11:14:37,345 ERROR lzo.DistributedLzoIndexer: DistributedIndexer job job_1639967851440_0006 failed.
解决方法:

在core-site.xml增加配置支持LZO压缩配置即可。


    
        io.compression.codecs
        
            org.apache.hadoop.io.compress.GzipCodec,
            org.apache.hadoop.io.compress.DefaultCodec,
            org.apache.hadoop.io.compress.BZip2Codec,
            org.apache.hadoop.io.compress.SnappyCodec,
            com.hadoop.compression.lzo.LzoCodec,
            com.hadoop.compression.lzo.LzopCodec
        
    

    
        io.compression.codec.lzo.class
        com.hadoop.compression.lzo.LzoCodec
    

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/673941.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号