common-pool是什么
apache commons-pool是apache基金会的一个开源对象池组件,常用的数据库连接池dpcp和redis的java客户端jedis都是基于commons-pool实现的。
什么是对象池
单例模式 singleton:全局只有一个对象,供所有线程使用。
原型模式 propotype:每次使用对象时都新建,用完之后释放。
对象池模式 pool:一个池子,里面 维护 多个对象。调用方从对象池 借取 对象,使用之后 归还 给对象池。
使用对象池的利与弊
弊:
- 增加复杂度。正常对象的生命周期由使用者和JVM维护: 使用者创建(JVM分配内存) → 销毁 → GC。对象池对象的生命周期更复杂,需要引入多个状态。 IDLE(空闲)、ALLOCATED(分配)、EVICTION(驱逐)、VALIDATION(验证)、INVALID(无效)、RETURNING(返回中);
- 多线程资源竞争。common-pool内部实现大量使用了synchronized以及其他juc原子类,可能会造成锁竞争,影响程序性能;
- 参数配置失误可能会带来严重问题。
利:
- 复用池中对象,消除创建对象、回收对象 所产生的内存开销、cpu开销以及(若跨网络)产生的网络开销common-pool参数配置详解
适用场景
- 受限的, 不需要可伸缩性的环境(cpu内存等物理资源有限): cpu性能不够强劲, 内存比较紧张, 垃圾收集, 内存抖动会造成比较大的影响, 需要提高内存管理效率, 响应性比吞吐量更为重要;
- 数量受限的资源, 比如数据库连接;
- 创建成本高昂的对象。
common-pool核心类
1. PooledObjectFactory【对象工厂】,用于产生一个新对象
1. PooledObjectFactory【对象工厂】,用于产生一个新对象
| public interface PooledObjectFactory
PooledObject
void destroyObject(PooledObject
boolean validateObject(PooledObject
void activateObject(PooledObject
void passivateObject(PooledObject } |
2. ObjectPool【链接池】,用于存放链接对象的一个池子,提供接口用于对象的借取和归还。
2.1 GenericObjectPool【ObjectPool的具体实现类】,这就是包里用来作为默认的链接池对象的实现类
GenericObjectPool中主要的方法包括borrowObject,returnObject,create, clear,close,destroy,ensureIdle,evict等。
borrowObject: 用于从对象池中取出一个对象;
returnObject:把取出来的对象归还给对象池;
create:调用PooledObjectFactory的makeObject方法新建对象
clear:清空对象池中的对象;
close:停止驱逐任务运行,清空对象池对象,注销对象池的Mbean,中断当前等待从池中获取对象的线程;
destroy:空闲对象队列中remove指定对象,并且全部对象队列中remove该对象,销毁对象数量+1,新建对象-1。PooledObjectFactory的实现类destroy具体的对象;
ensureIdle:确保有充足的空闲对象;
evict:驱逐无效的对象。
2.2 BaseObjectPoolConfig【链接池主要配置】,提供了链接池的主要配置,例如进出策略,最大链接,空闲链接,超时时间,回收策略等等一系列丰富的对于链接池的管理。
详见common-pool参数配置详解
2.3 AbandonedConfig【丢弃策略】,提供了针对链接池内链接的丢弃策略,主要防止内存泄漏
3. PooledObject【链接对象】,链接池内的对象。
common-pool参数配置详解
GenericObjectPoolConfig
| 参数名 | 类型 | 描述 | 默认值 |
|---|---|---|---|
maxTotal | int | 最大对象数 | 8 |
maxIdle | int | 最大空闲对象数 | 8 |
minIdle | int | 最小空闲对象数 | 0 |
lifo | boolean | 双端队列中空闲对象是否后进先出 | true |
fairness | boolean | 队列锁使用的是否公平锁 | false |
maxWaitDuration | Duration | 最大等待时间 -1L | |
minEvictableIdleDuration | Duration | 最小空闲驱逐时间 1000L * 60L * 30L | |
evictorShutdownTimeoutDuration | Duration | 驱逐停止超时时间 10L * 1000L | |
softMinEvictableIdleDuration | Duration | 最小安全驱逐空闲时间 -1L | |
numTestsPerEvictionRun | int | 每次驱逐运行的测试数量 | 3 |
evictionPolicy | EvictionPolicy | 驱逐策略 | DefaultEvictionPolicy |
evictionPolicyClassName | String | 驱逐策略名称 | org.apache.commons.pool2.impl.DefaultEvictionPolicy |
testOnCreate | boolean | 创建池对象时检测池对象是否有效,无效则剔除,并尝试继续获取新池对象 | false |
testOnBorrow | boolean | 取出池对象时检测池对象是否有效,无效则剔除,并尝试继续获取新池对象 | false |
testOnReturn | boolean | 归还对象时检测对象是否有效,无效则不归还 | false |
testWhileIdle | boolean | 检测是否空闲,空闲则调用evict方法驱逐对象 | false |
durationBetweenEvictionRuns | Duration | 空闲对象驱除任务间隔时间 -1L | |
blockWhenExhausted | boolean | 值【true】maxWaitMillis生效,【false】对象池池没资源时,立马抛异常 | true |
jmxEnabled | boolean | 是否开启Java管理扩展功能 | true |
jmxNamePrefix | String | jmx名称前缀 | ”pool“ |
jmxNameBase | String | jmx名称base | null |
common-pool实现SFTP连接池demo
SFTPProperties 配置类
| public class SFTPProperties { private String username; private String password; private String privateKey; private String host; private Integer port; private Integer maxTotal; private Integer maxIdle; private Integer minIdle; ... } |
ChannelSftpFactory SFTP连接池工厂
channelSftp.getSession().disconnect() 如果不关闭连接将一直占用
| public class ChannelSftpFactory extends BasePooledObjectFactory private SFTPProperties properties; public ChannelSftpFactory(SFTPProperties properties) { this.properties = properties; } public ChannelSftp create() { return SFTPOperationHelper.login(this.properties); } public PooledObject return new DefaultPooledObject(channelSftp); } public void destroyObject(PooledObject ChannelSftp channelSftp = (ChannelSftp)p.getObject(); channelSftp.disconnect(); try { channelSftp.getSession().disconnect(); } catch (JSchException e) { e.printStackTrace(); } } ... } |
ChannelSftpPool SFTP连接池
| public class ChannelSftpPool extends GenericObjectPool private String name; public ChannelSftpPool(PooledObjectFactory super(factory); } public ChannelSftpPool(PooledObjectFactory super(factory, config); } ... } |
ChannelSftpPoolHelper SFTP连接池操作辅助类
| public class ChannelSftpPoolHelper { private static Map private static final Integer DEFAULT_MAX_TOTAL = 5; private static final Integer DEFAULT_MAX_IDLE = 4; private static final Integer DEFAULT_MIN_IDLE = 1; private static final Long IDLE_EVICT_TIME = 20000L; private static final Integer MAX_WAIT_TIME = 3000; public ChannelSftpPoolHelper() { }
public static void addPool(String userName, ChannelSftpPool sftpPool) { customerSFTPPools.put(userName, sftpPool); }
public static ChannelSftpPool getPool(String userName) { return (ChannelSftpPool)customerSFTPPools.get(userName); }
public static ChannelSftpPool getAndCreatePool(SFTPProperties properties) { String userName = properties.getUsername(); ChannelSftpPool pool = getPool(userName); if (pool == null) { pool = createPool(properties); if (pool != null) { customerSFTPPools.put(userName, pool); } } return pool; }
public static ChannelSftp borrowObject(ChannelSftpPool pool) { try { if (pool != null) { ChannelSftp channelSftp = (ChannelSftp)pool.borrowObject(); if (channelSftp.isClosed()) { pool.invalidateObject(channelSftp); channelSftp = (ChannelSftp)pool.borrowObject(); } return channelSftp; } } catch (Exception var2) { System.err.println("连接池获取异常"); } return null; }
public static ChannelSftp borrowObject(String userName) { ChannelSftpPool pool = getPool(userName); return borrowObject(pool); }
public static ChannelSftp borrowObject(SFTPProperties properties) { ChannelSftpPool pool = getAndCreatePool(properties); return borrowObject(pool); }
public static void returnObject(String userName, ChannelSftp channelSftp) { if (channelSftp != null) { ChannelSftpPool pool = getPool(userName); pool.returnObject(channelSftp); } }
public static ChannelSftpPool createPool(SFTPProperties properties) { ChannelSftpFactory factory = new ChannelSftpFactory(properties); return createPool(properties, factory); }
public static ChannelSftpPool createPool(SFTPProperties sftpProperties, ChannelSftpFactory channelSftpFactory) { GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig(); Integer maxIdle = (Integer)Optional.ofNullable(sftpProperties.getMaxIdle()).orElse(DEFAULT_MAX_IDLE); poolConfig.setMaxIdle(maxIdle); Integer minIdle = (Integer)Optional.ofNullable(sftpProperties.getMinIdle()).orElse(DEFAULT_MIN_IDLE); poolConfig.setMinIdle(minIdle); Integer maxTotal = (Integer)Optional.ofNullable(sftpProperties.getMaxTotal()).orElse(DEFAULT_MAX_TOTAL); poolConfig.setMaxTotal(maxTotal); poolConfig.setBlockWhenExhausted(true); poolConfig.setTestOnCreate(true); poolConfig.setTestOnBorrow(false); poolConfig.setTestOnReturn(true); poolConfig.setTestWhileIdle(true); poolConfig.setTimeBetweenEvictionRuns(Duration.ofMillis(5000L)); poolConfig.setMinEvictableIdleTime(Duration.ofMillis(1000L)); poolConfig.setMaxWait(Duration.ofMillis((long)MAX_WAIT_TIME)); poolConfig.setJmxEnabled(false); ChannelSftpPool pool = new ChannelSftpPool(channelSftpFactory, poolConfig); pool.setName(sftpProperties.getUsername()); addPool(sftpProperties.getUsername(), pool); return pool; }
public static Map return customerSFTPPools; } } |
SFTPOperationHelper SFTP操作辅助父类
| public class SFTPOperationHelper { public SFTPOperationHelper() { }
public static ChannelSftp login(SFTPProperties properties) { try { JSch jsch = new JSch(); if (StringUtils.hasText(properties.getPrivateKey())) { jsch.addIdentity("idname", properties.getPrivateKey().getBytes(), (byte[])null, (byte[])null); } Session sshSession = jsch.getSession(properties.getUsername(), properties.getHost(), properties.getPort()); if (StringUtils.hasText(properties.getPassword())) { sshSession.setPassword(properties.getPassword()); } Properties sshConfig = new Properties(); sshConfig.put("StrictHostKeyChecking", "no"); sshConfig.put("PreferredAuthentications", "publickey,keyboard-interactive,password"); sshSession.setConfig(sshConfig); sshSession.connect(); ChannelSftp channel = (ChannelSftp)sshSession.openChannel("sftp"); channel.connect(); return channel; } catch (JSchException var5) { System.err.println("连接SFTP服务器={}异常"); return null; } }
public static InputStream downloadToStream(String dir, String name, ChannelSftp sftp) { try { sftp.cd(dir); return sftp.get(name); } catch (SftpException var4) { System.err.println("SFTP下载文件={}{}异常"); return null; } }
public static byte[] downloadToBytes(String dir, String name, ChannelSftp sftp) { try { sftp.cd(dir); InputStream inputStream = sftp.get(name); return IOUtils.toByteArray(inputStream); } catch (IOException | SftpException var4) { System.err.println("SFTP下载文件={}{}异常"); return null; } }
public static void upload(String dir, String name, InputStream in, ChannelSftp sftp) throws SftpException { mkdirs(dir, sftp); sftp.cd(dir); sftp.put(in, name); }
public static void delete(String dir, String name, ChannelSftp sftp) throws SftpException { sftp.cd(dir); sftp.rm(name); }
public static void mkdirs(String dir, ChannelSftp sftp) throws SftpException { String[] folders = dir.split("/"); sftp.cd("/"); String[] var3 = folders; int var4 = folders.length; for(int var5 = 0; var5 < var4; ++var5) { String folder = var3[var5]; if (folder.length() > 0) { try { sftp.cd(folder); } catch (Exception var8) { sftp.mkdir(folder); sftp.cd(folder); } } } } } |
SFTPPoolOperationHelper SFTP操作辅助子类
| public class SFTPPoolOperationHelper extends SFTPOperationHelper { public SFTPPoolOperationHelper() { }
public static byte[] downloadToBytes(String dir, String name, ChannelSftpPool pool) throws Exception { ChannelSftp sftp = ChannelSftpPoolHelper.borrowObject(pool); byte[] var4; try { var4 = downloadToBytes(dir, name, sftp); } finally { pool.returnObject(sftp); } return var4; }
public static void upload(String dir, String name, InputStream in, ChannelSftpPool pool) throws Exception { ChannelSftp sftp = (ChannelSftp)pool.borrowObject(); try { upload(dir, name, in, sftp); } finally { pool.returnObject(sftp); } }
public static void delete(String dir, String name, ChannelSftpPool pool) throws Exception { ChannelSftp sftp = (ChannelSftp)pool.borrowObject(); try { delete(dir, name, sftp); } finally { pool.returnObject(sftp); } }
private static void mkdirs(String dir, ChannelSftpPool pool) throws Exception { ChannelSftp sftp = (ChannelSftp)pool.borrowObject(); try { mkdirs(dir, sftp); } finally { pool.returnObject(sftp); } } } |
DmpCrowdFtpUtils SFTP util类
| public class DmpCrowdFtpUtils { private final static String FTP_USER_NAME = "squidxu"; private final static String FTP_PASSWORD = "squidyu.1"; private final static String FTP_HOST = "10.1.18.75"; private final static Integer FTP_PORT = 22;
private static final String SEPARATOR = "/";
public static String getFtpTargetPath(Integer taskId) { return SEPARATOR + "download" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + taskId + SEPARATOR; // return SEPARATOR + "in" + SEPARATOR + "dmp" + SEPARATOR + "channel_ad_marketing_server" + SEPARATOR + date + SEPARATOR + taskId + SEPARATOR; }
public static String uploadFile(File file, Integer taskId, String fileName) throws Exception { String ftpTargetPath = getFtpTargetPath(taskId); InputStream content = new FileInputStream(file); ChannelSftpPool channelSftpPool = getChannelSftpPool(); SFTPPoolOperationHelper.upload(ftpTargetPath, fileName, content, channelSftpPool); return ftpTargetPath + fileName; } public static void main(String[] args) { for (int i = 0; i < 10; i++) { File file = new File("C:\Users\squidxu\Desktop\周报模板.xlsx"); try { uploadFile(file, 1000 + i, "周报模板" + i + ".xlsx"); } catch (Exception e) { e.printStackTrace(); } } }
public static InputStream downloadStream(String fullPath) throws Exception { return new ByteArrayInputStream(downloadFile(fullPath)); }
public static void download2Local(String path, String localPath) throws Exception { BufferedReader buffReader = null; BufferedWriter buffWriter = null; try { buffReader = new BufferedReader(new InputStreamReader(downloadStream(path))); buffWriter = new BufferedWriter(new OutputStreamWriter( new FileOutputStream(localPath,true))); String line = null; while ((line = buffReader.readLine()) != null) { buffWriter.write(line + "n"); } } finally { flush(buffWriter); close(buffReader); close(buffWriter); } } private static void flush(BufferedWriter bufferedWriter) { if (bufferedWriter != null) { try { bufferedWriter.flush(); } catch (IOException e) { System.err.println("close error, message={}"+ e.getMessage()); } } }
public static byte[] downloadFile(String fullPath) throws Exception { String ftpTargetPath = StringUtils.substringBeforeLast(fullPath, SEPARATOR) + SEPARATOR; String ftpFileName = StringUtils.substringAfterLast(fullPath, SEPARATOR); long start = System.currentTimeMillis(); ChannelSftpPool channelSftpPool = getChannelSftpPool(); System.out.println("channelSftpPool:" + channelSftpPool.hashCode()); long end = System.currentTimeMillis(); System.out.println("getChannelSftpPool" + (end - start) + "ms"); return SFTPPoolOperationHelper.downloadToBytes(ftpTargetPath, ftpFileName, channelSftpPool); }
private static ChannelSftpPool getChannelSftpPool() { SFTPProperties properties = new SFTPProperties(); properties.setUsername(FTP_USER_NAME); properties.setPassword(FTP_PASSWORD); properties.setHost(FTP_HOST); properties.setPort(FTP_PORT); properties.setMaxTotal(5); properties.setMaxIdle(1); properties.setMinIdle(1); final ChannelSftpPool pool = ChannelSftpPoolHelper.getAndCreatePool(properties); System.out.println("object size" + pool.getNumActive()); System.out.println("MaxIdle size" + pool.getMaxIdle()); System.out.println("MaxIdle NumIdle" + pool.getNumIdle()); if (pool == null) { throw new RuntimeException("获取pool错误"); } return pool; }
private static void close(BufferedReader reader) throws IOException { if (reader != null) { reader.close(); } }
private static void close(BufferedWriter writer) throws IOException { if (writer != null) { writer.close(); } } } |



