栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Java

common-pool剖析

Java 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

common-pool剖析

common-pool是什么

apache commons-pool是apache基金会的一个开源对象池组件,常用的数据库连接池dpcp和redis的java客户端jedis都是基于commons-pool实现的。

什么是对象池

单例模式 singleton:全局只有一个对象,供所有线程使用。

原型模式 propotype:每次使用对象时都新建,用完之后释放。

对象池模式 pool:一个池子,里面 维护 多个对象。调用方从对象池 借取 对象,使用之后 归还 给对象池。

使用对象池的利与弊

弊:

  1. 增加复杂度。正常对象的生命周期由使用者和JVM维护: 使用者创建(JVM分配内存) → 销毁 → GC。对象池对象的生命周期更复杂,需要引入多个状态。 IDLE(空闲)、ALLOCATED(分配)、EVICTION(驱逐)、VALIDATION(验证)、INVALID(无效)、RETURNING(返回中);
  2. 多线程资源竞争。common-pool内部实现大量使用了synchronized以及其他juc原子类,可能会造成锁竞争,影响程序性能;
  3. 参数配置失误可能会带来严重问题。

利:

  1. 复用池中对象,消除创建对象、回收对象 所产生的内存开销、cpu开销以及(若跨网络)产生的网络开销common-pool参数配置详解

适用场景
  1.  受限的, 不需要可伸缩性的环境(cpu内存等物理资源有限): cpu性能不够强劲, 内存比较紧张, 垃圾收集, 内存抖动会造成比较大的影响, 需要提高内存管理效率, 响应性比吞吐量更为重要;
  2. 数量受限的资源, 比如数据库连接;
  3. 创建成本高昂的对象。

common-pool核心类

1. PooledObjectFactory【对象工厂】,用于产生一个新对象

public interface PooledObjectFactory {

    

    PooledObject makeObject() throws Exception;

  

    

    void destroyObject(PooledObject p) throws Exception;

  

    

    boolean validateObject(PooledObject p);

  

    

    void activateObject(PooledObject p) throws Exception;

  

    

    void passivateObject(PooledObject p) throws Exception;

}

2. ObjectPool【链接池】,用于存放链接对象的一个池子,提供接口用于对象的借取和归还。

2.1 GenericObjectPool【ObjectPool的具体实现类】,这就是包里用来作为默认的链接池对象的实现类

GenericObjectPool中主要的方法包括borrowObject,returnObject,create, clear,close,destroy,ensureIdle,evict等。

borrowObject: 用于从对象池中取出一个对象;

returnObject:把取出来的对象归还给对象池;

create:调用PooledObjectFactory的makeObject方法新建对象

clear:清空对象池中的对象;

close:停止驱逐任务运行,清空对象池对象,注销对象池的Mbean,中断当前等待从池中获取对象的线程;

destroy:空闲对象队列中remove指定对象,并且全部对象队列中remove该对象,销毁对象数量+1,新建对象-1。PooledObjectFactory的实现类destroy具体的对象;

ensureIdle:确保有充足的空闲对象;

evict:驱逐无效的对象。

2.2 BaseObjectPoolConfig【链接池主要配置】,提供了链接池的主要配置,例如进出策略,最大链接,空闲链接,超时时间,回收策略等等一系列丰富的对于链接池的管理。

详见common-pool参数配置详解

2.3 AbandonedConfig【丢弃策略】,提供了针对链接池内链接的丢弃策略,主要防止内存泄漏

3. PooledObject【链接对象】,链接池内的对象。

common-pool参数配置详解

GenericObjectPoolConfig

参数名

类型

描述

默认值

maxTotal
int最大对象数8
maxIdle
int最大空闲对象数8
minIdle
int最小空闲对象数0
lifo
boolean双端队列中空闲对象是否后进先出true
fairness
boolean队列锁使用的是否公平锁false
maxWaitDuration
Duration最大等待时间 -1L
minEvictableIdleDuration
Duration最小空闲驱逐时间 
1000L * 60L * 30L
evictorShutdownTimeoutDuration
Duration驱逐停止超时时间 
10L * 1000L
softMinEvictableIdleDuration
Duration最小安全驱逐空闲时间 -1L
numTestsPerEvictionRun
int每次驱逐运行的测试数量3
evictionPolicy
EvictionPolicy驱逐策略
DefaultEvictionPolicy
evictionPolicyClassName
String驱逐策略名称org.apache.commons.pool2.impl.DefaultEvictionPolicy
testOnCreate
boolean创建池对象时检测池对象是否有效,无效则剔除,并尝试继续获取新池对象false
testOnBorrow
boolean取出池对象时检测池对象是否有效,无效则剔除,并尝试继续获取新池对象false
testOnReturn
boolean归还对象时检测对象是否有效,无效则不归还false
testWhileIdle
boolean检测是否空闲,空闲则调用evict方法驱逐对象false
durationBetweenEvictionRuns
Duration空闲对象驱除任务间隔时间 -1L
blockWhenExhausted
boolean值【true】maxWaitMillis生效,【false】对象池池没资源时,立马抛异常true
jmxEnabled
boolean是否开启Java管理扩展功能true
jmxNamePrefix
Stringjmx名称前缀”pool“
jmxNameBase
Stringjmx名称basenull

common-pool实现SFTP连接池demo

SFTPProperties 配置类

public class SFTPProperties {

    private String username;

    private String password;

    private String privateKey;

    private String host;

    private Integer port;

    private Integer maxTotal;

    private Integer maxIdle;

    private Integer minIdle;

    ...

}

ChannelSftpFactory SFTP连接池工厂

channelSftp.getSession().disconnect() 如果不关闭连接将一直占用

public class ChannelSftpFactory extends BasePooledObjectFactory {

    private SFTPProperties properties;

    public ChannelSftpFactory(SFTPProperties properties) {

        this.properties = properties;

    }

    public ChannelSftp create() {

        return SFTPOperationHelper.login(this.properties);

    }

    public PooledObject wrap(ChannelSftp channelSftp) {

        return new DefaultPooledObject(channelSftp);

    }

    public void destroyObject(PooledObject p) {

        ChannelSftp channelSftp = (ChannelSftp)p.getObject();

        channelSftp.disconnect();

        try {

            channelSftp.getSession().disconnect();

        } catch (JSchException e) {

            e.printStackTrace();

        }

    }

    ...

}

ChannelSftpPool SFTP连接池

public class ChannelSftpPool extends GenericObjectPool {

    private String name;

    public ChannelSftpPool(PooledObjectFactory factory) {

        super(factory);

    }

    public ChannelSftpPool(PooledObjectFactory factory, GenericObjectPoolConfig config) {

        super(factory, config);

    }

    ...

}

ChannelSftpPoolHelper SFTP连接池操作辅助类

public class ChannelSftpPoolHelper {

    private static Map customerSFTPPools = new ConcurrentHashMap();

    private static final Integer DEFAULT_MAX_TOTAL = 5;

    private static final Integer DEFAULT_MAX_IDLE = 4;

    private static final Integer DEFAULT_MIN_IDLE = 1;

    private static final Long IDLE_EVICT_TIME = 20000L;

    private static final Integer MAX_WAIT_TIME = 3000;

    public ChannelSftpPoolHelper() {

    }

    

    public static void addPool(String userName, ChannelSftpPool sftpPool) {

        customerSFTPPools.put(userName, sftpPool);

    }

    

    public static ChannelSftpPool getPool(String userName) {

        return (ChannelSftpPool)customerSFTPPools.get(userName);

    }

    

    public static ChannelSftpPool getAndCreatePool(SFTPProperties properties) {

        String userName = properties.getUsername();

        ChannelSftpPool pool = getPool(userName);

        if (pool == null) {

            pool = createPool(properties);

            if (pool != null) {

                customerSFTPPools.put(userName, pool);

            }

        }

        return pool;

    }

    

    public static ChannelSftp borrowObject(ChannelSftpPool pool) {

        try {

            if (pool != null) {

                ChannelSftp channelSftp = (ChannelSftp)pool.borrowObject();

                if (channelSftp.isClosed()) {

                    pool.invalidateObject(channelSftp);

                    channelSftp = (ChannelSftp)pool.borrowObject();

                }

                return channelSftp;

            }

        } catch (Exception var2) {

            System.err.println("连接池获取异常");

        }

        return null;

    }

    

    public static ChannelSftp borrowObject(String userName) {

        ChannelSftpPool pool = getPool(userName);

        return borrowObject(pool);

    }

    

    public static ChannelSftp borrowObject(SFTPProperties properties) {

        ChannelSftpPool pool = getAndCreatePool(properties);

        return borrowObject(pool);

    }

    

    public static void returnObject(String userName, ChannelSftp channelSftp) {

        if (channelSftp != null) {

            ChannelSftpPool pool = getPool(userName);

            pool.returnObject(channelSftp);

        }

    }

    

    public static ChannelSftpPool createPool(SFTPProperties properties) {

        ChannelSftpFactory factory = new ChannelSftpFactory(properties);

        return createPool(properties, factory);

    }

    

    public static ChannelSftpPool createPool(SFTPProperties sftpProperties, ChannelSftpFactory channelSftpFactory) {

        GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();

        Integer maxIdle = (Integer)Optional.ofNullable(sftpProperties.getMaxIdle()).orElse(DEFAULT_MAX_IDLE);

        poolConfig.setMaxIdle(maxIdle);

        Integer minIdle = (Integer)Optional.ofNullable(sftpProperties.getMinIdle()).orElse(DEFAULT_MIN_IDLE);

        poolConfig.setMinIdle(minIdle);

        Integer maxTotal = (Integer)Optional.ofNullable(sftpProperties.getMaxTotal()).orElse(DEFAULT_MAX_TOTAL);

        poolConfig.setMaxTotal(maxTotal);

        poolConfig.setBlockWhenExhausted(true);

        poolConfig.setTestOnCreate(true);

        poolConfig.setTestOnBorrow(false);

        poolConfig.setTestOnReturn(true);

        poolConfig.setTestWhileIdle(true);

        poolConfig.setTimeBetweenEvictionRuns(Duration.ofMillis(5000L));

        poolConfig.setMinEvictableIdleTime(Duration.ofMillis(1000L));

        poolConfig.setMaxWait(Duration.ofMillis((long)MAX_WAIT_TIME));

        poolConfig.setJmxEnabled(false);

        ChannelSftpPool pool = new ChannelSftpPool(channelSftpFactory, poolConfig);

        pool.setName(sftpProperties.getUsername());

        addPool(sftpProperties.getUsername(), pool);

        return pool;

    }

    

    public static Map getPools() {

        return customerSFTPPools;

    }

}

SFTPOperationHelper SFTP操作辅助父类  

public class SFTPOperationHelper {

    public SFTPOperationHelper() {

    }

    

    public static ChannelSftp login(SFTPProperties properties) {

        try {

            JSch jsch = new JSch();

            if (StringUtils.hasText(properties.getPrivateKey())) {

                jsch.addIdentity("idname", properties.getPrivateKey().getBytes(), (byte[])null, (byte[])null);

            }

            Session sshSession = jsch.getSession(properties.getUsername(), properties.getHost(), properties.getPort());

            if (StringUtils.hasText(properties.getPassword())) {

                sshSession.setPassword(properties.getPassword());

            }

            Properties sshConfig = new Properties();

            sshConfig.put("StrictHostKeyChecking", "no");

            sshConfig.put("PreferredAuthentications", "publickey,keyboard-interactive,password");

            sshSession.setConfig(sshConfig);

            sshSession.connect();

            ChannelSftp channel = (ChannelSftp)sshSession.openChannel("sftp");

            channel.connect();

            return channel;

        } catch (JSchException var5) {

            System.err.println("连接SFTP服务器={}异常");

            return null;

        }

    }

    

    public static InputStream downloadToStream(String dir, String name, ChannelSftp sftp) {

        try {

            sftp.cd(dir);

            return sftp.get(name);

        } catch (SftpException var4) {

            System.err.println("SFTP下载文件={}{}异常");

            return null;

        }

    }

    

    public static byte[] downloadToBytes(String dir, String name, ChannelSftp sftp) {

        try {

            sftp.cd(dir);

            InputStream inputStream = sftp.get(name);

            return IOUtils.toByteArray(inputStream);

        } catch (IOException | SftpException var4) {

            System.err.println("SFTP下载文件={}{}异常");

            return null;

        }

    }

    

    public static void upload(String dir, String name, InputStream in, ChannelSftp sftp) throws SftpException {

        mkdirs(dir, sftp);

        sftp.cd(dir);

        sftp.put(in, name);

    }

    

    public static void delete(String dir, String name, ChannelSftp sftp) throws SftpException {

        sftp.cd(dir);

        sftp.rm(name);

    }

    

    public static void mkdirs(String dir, ChannelSftp sftp) throws SftpException {

        String[] folders = dir.split("/");

        sftp.cd("/");

        String[] var3 = folders;

        int var4 = folders.length;

        for(int var5 = 0; var5 < var4; ++var5) {

            String folder = var3[var5];

            if (folder.length() > 0) {

                try {

                    sftp.cd(folder);

                } catch (Exception var8) {

                    sftp.mkdir(folder);

                    sftp.cd(folder);

                }

            }

        }

    }

}

SFTPPoolOperationHelper SFTP操作辅助子类 

public class SFTPPoolOperationHelper extends SFTPOperationHelper {

    public SFTPPoolOperationHelper() {

    }

    

    public static byte[] downloadToBytes(String dir, String name, ChannelSftpPool pool) throws Exception {

        ChannelSftp sftp = ChannelSftpPoolHelper.borrowObject(pool);

        byte[] var4;

        try {

            var4 = downloadToBytes(dir, name, sftp);

        } finally {

            pool.returnObject(sftp);

        }

        return var4;

    }

    

    public static void upload(String dir, String name, InputStream in, ChannelSftpPool pool) throws Exception {

        ChannelSftp sftp = (ChannelSftp)pool.borrowObject();

        try {

            upload(dir, name, in, sftp);

        } finally {

            pool.returnObject(sftp);

        }

    }

    

    public static void delete(String dir, String name, ChannelSftpPool pool) throws Exception {

        ChannelSftp sftp = (ChannelSftp)pool.borrowObject();

        try {

            delete(dir, name, sftp);

        } finally {

            pool.returnObject(sftp);

        }

    }

    

    private static void mkdirs(String dir, ChannelSftpPool pool) throws Exception {

        ChannelSftp sftp = (ChannelSftp)pool.borrowObject();

        try {

            mkdirs(dir, sftp);

        } finally {

            pool.returnObject(sftp);

        }

    }

}

DmpCrowdFtpUtils SFTP util类 

public class DmpCrowdFtpUtils {

    private final static String FTP_USER_NAME = "squidxu";

    private final static String FTP_PASSWORD = "squidyu.1";

    private final static String FTP_HOST = "10.1.18.75";

    private final static Integer FTP_PORT = 22;

    

    private static final String SEPARATOR = "/";

    

    public static String getFtpTargetPath(Integer taskId) {

        return SEPARATOR + "download" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + taskId + SEPARATOR;

//        return SEPARATOR + "in" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + date + SEPARATOR + taskId + SEPARATOR;

    }

    

    public static String uploadFile(File file, Integer taskId, String fileName) throws Exception {

        String ftpTargetPath = getFtpTargetPath(taskId);

        InputStream content = new FileInputStream(file);

        ChannelSftpPool channelSftpPool = getChannelSftpPool();

        SFTPPoolOperationHelper.upload(ftpTargetPath, fileName, content, channelSftpPool);

        return ftpTargetPath + fileName;

    }

    public static void main(String[] args) {

        for (int i = 0; i < 10; i++) {

            File file = new File("C:\Users\squidxu\Desktop\周报模板.xlsx");

            try {

                uploadFile(file, 1000 + i, "周报模板" + i + ".xlsx");

            } catch (Exception e) {

                e.printStackTrace();

            }

        }

    }

    

    public static InputStream downloadStream(String fullPath) throws Exception {

        return new ByteArrayInputStream(downloadFile(fullPath));

    }

    

    public static void download2Local(String path, String localPath) throws Exception {

        BufferedReader buffReader = null;

        BufferedWriter buffWriter = null;

        try {

            buffReader = new BufferedReader(new InputStreamReader(downloadStream(path)));

            buffWriter = new BufferedWriter(new OutputStreamWriter(

                    new FileOutputStream(localPath,true)));

            String line = null;

            while ((line = buffReader.readLine()) != null) {

                buffWriter.write(line + "n");

            }

        } finally {

            flush(buffWriter);

            close(buffReader);

            close(buffWriter);

        }

    }

    private static void flush(BufferedWriter bufferedWriter) {

        if (bufferedWriter != null) {

            try {

                bufferedWriter.flush();

            } catch (IOException e) {

                System.err.println("close error, message={}"+ e.getMessage());

            }

        }

    }

    

    public static byte[] downloadFile(String fullPath) throws Exception {

        String ftpTargetPath = StringUtils.substringBeforeLast(fullPath, SEPARATOR) + SEPARATOR;

        String ftpFileName = StringUtils.substringAfterLast(fullPath, SEPARATOR);

        long start = System.currentTimeMillis();

        ChannelSftpPool channelSftpPool = getChannelSftpPool();

        System.out.println("channelSftpPool:" + channelSftpPool.hashCode());

        long end = System.currentTimeMillis();

        System.out.println("getChannelSftpPool" + (end - start) + "ms");

        return SFTPPoolOperationHelper.downloadToBytes(ftpTargetPath, ftpFileName, channelSftpPool);

    }

    

    private static ChannelSftpPool getChannelSftpPool() {

        SFTPProperties properties = new SFTPProperties();

        properties.setUsername(FTP_USER_NAME);

        properties.setPassword(FTP_PASSWORD);

        properties.setHost(FTP_HOST);

        properties.setPort(FTP_PORT);

        properties.setMaxTotal(5);

        properties.setMaxIdle(1);

        properties.setMinIdle(1);

        final ChannelSftpPool pool = ChannelSftpPoolHelper.getAndCreatePool(properties);

        System.out.println("object size" + pool.getNumActive());

        System.out.println("MaxIdle size" + pool.getMaxIdle());

        System.out.println("MaxIdle NumIdle" + pool.getNumIdle());

        if (pool == null) {

            throw new RuntimeException("获取pool错误");

        }

        return pool;

    }

    

    private static void close(BufferedReader reader) throws IOException {

        if (reader != null) {

            reader.close();

        }

    }

    

    private static void close(BufferedWriter writer) throws IOException {

        if (writer != null) {

            writer.close();

        }

    }

}

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/981291.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号