项目中使用到了Spring data jpa技术,调用 JpaRepository.saveAll()/deleteAll()方法对list中的数据进行插入/删除时,发现速度特别慢,数据量5k+的数据大概需要话费20s+,导致请求时间过长,前端报错请求超时。
saveAll底层源码:
@Transactional publicListsaveAll(Iterableentities) { Assert.notNull(entities, "The given Iterable of entities not be null!"); Listresult = new ArrayList(); for (S entity : entities) { result.add(save(entity)); } return result; }
save方法源码:
@Transactional publicS save(S entity) { if (entityInformation.isNew(entity)) { em.persist(entity); return entity; } else { return em.merge(entity); } }
deleteAll()源码:
@Transactional
public void deleteAll() {
for (T element : findAll()) {
delete(element);
}
}
原因分析:
查看Spring data jpa 底层==savaAll()==方法,发现,每次保存都是一条条执行,先查询出来,判断是否存在,存在做编辑操作,不存在则新增,导致耗时很久。==deleteAll()==方法也是先查询出所有结果,然后一条条执行删除操作。
解决方案: 1.批量更新1.之前看网上有人说在ymal配置文件中配置可以解决,但是测试下来发现并没有用,配置文件如下:
spring:
jpa:
properties:
hibernate:
jdbc:
batch_size: 500
batch_versioned_data: true
order_inserts: true
order_updates: true
此方法并没有解决我的问题
2.使用EntityManager执行自定义sql
1.引入对象
@PersistenceContext private EntityManager entityManager;
2.重写saveAll实现批量插入
entityManager.createNativeQuery(sb.toString()+";").executeUpdate()
@Override public ListsaveAll(List list) { StringBuilder sb = into(); int i=0; for (DistributeSuccessPlaybill dis : list) { i++; if(i>1){ sb.append(","); } sb.append(String.format(" ('%s','%s','%s','%s','%s','%s'," + "'%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s','%s')", dis.getLayoutId(), dis.getOutChannelId(), dis.getContentId(), dis.getAssetId(), dis.getType(), dis.getPlayUrl(), dis.getName(), dis.getDisplayName(), dis.getPlayType(), dis.getStartTime(), dis.getEndTime(), dis.getDuration(), dis.getInsertMode(), dis.getDistributeStatus(), dis.getCreatedBy(), dis.getCreatedTime(), dis.getUpdatedBy(), dis.getUpdatedTime())); //一千条执行一次 if(i >= 1000){ log.info(sb.toString()); entityManager.createNativeQuery(sb.toString()+";").executeUpdate(); i = 0; sb = into(); } } if(i > 0){ entityManager.createNativeQuery(sb.toString()+";").executeUpdate(); } return list; } //sql前半段方法提取 private StringBuilder into(){ StringBuilder sb = new StringBuilder(); sb.append("insert into lp_distribute_success_playbill(" + "layoutId," + "outChannelId," + "contentId," + "asset_id," + "type," + "playUrl," + "name," + "displayName," + "playType," + "startTime," + "endTime," + "duration," + "insertMode," + "distributeStatus," + "createdBy," + "createdTime," + "updatedBy," + "updatedTime) values"); return sb; }
此方法再重新执行批量插入,5k+大约消耗3s,效率极大提高,完美解决。
2.批量删除在Repository层使用原生自定义sql,根据需要删除的条件组装sql语句,因为项目的事务加在impl层,所以此处不加事务注解,但是需要加==@Modifying==来标识。
@Modifying @Query(value = "delete from lp_distribute_success_playbill where outChannelId=?1 and startTime>=?2",nativeQuery = true) void deleteBatch(Long outChannelId,LocalDateTime startTime);
消耗时间过长问题完美解决~~~!!!!



