第三课 Spring Cloud分布式微服务实战-开发文件服务

tags:

Java
慕课网

categories:

MinIO
内容安全

文章目录

第三课 Spring Cloud分布式微服务实战-开发文件服务
- 第一节文件服务器
- - 1.1 文件服务器选择
  - 1.2 部署MinIO
- 第二节 MinIO文件上传
- - 2.1 创建文件服务Module
  - 2.2 整合MinIO的service实现
  - 2.3 整合MinIO的Controller实现
  - 2.4 整合MinIO的Controller实现
- 第三节云存储阿里OSS
- - 3.1 阿里OSS优势
  - 3.2 阿里云接入

第一节文件服务器 1.1 文件服务器选择

文件存储的网络服务选择有很多，比如阿里云OSS、七牛云、腾讯云等等, 有收费。
fastDFS安装部署(运维)复杂度高，没有官方文档，维护比较麻烦。
MinIO文档: http://docs.minio.org.cn/docs/ 开箱即用，UI页面，性能高。MinIO号称是世界上速度最快的对象存储服务器。在标准硬件上，对象存储的读/写速度最高可以达到183 GB/s和171 GB/s。文档齐全社区活跃。这里我们就选它(开箱即用)
ceph 学习成本高运维部署复杂。
一些分布式储存对比：https://www.cxyzjd.com/article/qq_32811865/107926890

1.2 部署MinIO

访问 web 管理页面: http://192.168.44.128:6090/login
创建bucket, 设置权限为public, 上传图片测试，访问路径这里注意是9000端口：http://192.168.44.128:9000/imooc/aa.png
如果权限为private,最多只有七天的分享访问时间。

sudo docker run -d -p 9000:9000 -p 6090:9090 --name minio1 
  -e "MINIO_ROOT_USER=admin" 
  -e "MINIO_ROOT_PASSWORD=123456789" 
  -v /mnt/data:/data 
  --restart=always 
  minio/minio server /data --console-address ':9090'

第二节 MinIO文件上传 2.1 创建文件服务Module

创建Module, 名称为imooc-news-dev-service-files。和user一样引入api模块。


    imooc-news-dev-service-files

    
        
            com.imooc
            >imooc-news-dev-service-api
            1.0-SNAPSHOT

先把配置文件和日志配置拿过来改一改。加上上传文件大小的限制。

  servlet:
    multipart:
      max-file-size: 512000 # 请求文件大小限制为500k
      max-request-size: 512000

把启动类和HelloWorld控制器拿过来，启动运行测试看看是否成功。发现报错

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

因为我们把yaml文件中数据源的相关信息删除了，files文件服务中用不到数据源的东西。但是我们在common服务中配置了数据源的驱动, 所以它也会引用到。解决办法：自动装配时候排除某数据源的自动装配。

@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

common中引入MinIO的依赖包。

        
        
            io.minio
            minio
            8.3.3
        

        
            org.projectlombok
            >lombok

2.2 整合MinIO的service实现

创建com.imooc.files.service.UploaderService接口。

package com.imooc.files.service;

import org.springframework.web.multipart.MultipartFile;

import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;


public interface UploaderService {
    public String uploadMinIO(MultipartFile file) throws IOException, NoSuchAlgorithmException, InvalidKeyException;
}

配置application.yaml将minio的配置写到其中加载到容器中。

minio:
  endpoint: http://192.168.44.128:9000
  accessKey: MinIOAdmin
  secretKey: MinIOAdmin
  bucketName: "imooc"

在中com.imooc.files.utils.MinioProp写一个MinioProp的bean。把配置放到容器中。

package com.imooc.files.utils;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Data
@Component
@ConfigurationProperties(prefix = "minio")
public class MinioProp {
    private String endpoint;
    private String accesskey;
    private String secretKey;
    private String bucketName;
}

写一个实现类com.imooc.files.service.impl.UploaderServiceImpl。@Service把实现类注入到容器中。

package com.imooc.files.service.impl;

import com.imooc.exception.GraceException;
import com.imooc.files.service.UploaderService;
import com.imooc.files.utils.MinioProp;
import com.imooc.grace.result.ResponseStatusEnum;
import io.minio.*;
import io.minio.errors.*;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import java.util.UUID;
import java.io.IOException;
import java.io.InputStream;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

@Service
public class UploaderServiceImpl implements UploaderService {

    @Autowired
    private MinioProp minioProp;

    @Override
    public String uploadMinIO(MultipartFile file) throws IOException, NoSuchAlgorithmException, InvalidKeyException {

        MinioClient minioClient = MinioClient.builder()
                .endpoint(minioProp.getEndpoint())
                .credentials(minioProp.getSecretKey(), minioProp.getAccesskey())
                .build();

        try {
            boolean found =
                    minioClient.bucketExists(BucketExistsArgs.builder().bucket(minioProp.getBucketName()).build());
            if (!found) {
                minioClient.makeBucket(MakeBucketArgs.builder().bucket(minioProp.getBucketName()).build());
            } else {
                System.out.println("Bucket" + minioProp.getBucketName() + "already exists.");
            }

            String filename = file.getOriginalFilename();
            String contentType = file.getContentType();
            String newName = UUID.randomUUID().toString().replaceAll("-", "")
                    + filename.substring(filename.lastIndexOf("."));

            InputStream inputStream = file.getInputStream();    //获取file的inputStream

            Long filesize = Long.valueOf(inputStream.available());
            minioClient.putObject(
                    PutObjectArgs.builder().bucket(minioProp.getBucketName()).object(newName).stream(
                            inputStream, filesize, -1).contentType(contentType)
                            .build());

            inputStream.close();
            return minioProp.getEndpoint() + "/" + minioProp.getBucketName() + "/" + newName;
        } catch (MinioException e) {
            System.out.println("Error occurred: " + e);
            System.out.println("HTTP trace: " + e.httpTrace());
            GraceException.display(ResponseStatusEnum.FILE_UPLOAD_FAILD);
            return null;
        }
    }
}

2.3 整合MinIO的Controller实现

API中创建com.imooc.api.controller.files.FileUploaderControllerApi

package com.imooc.api.controller.files;


import com.imooc.grace.result.GraceJSONResult;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;


@Api(value = "文件上传的Controller的标题", tags = {"文件上传的Controller的标题"})
@RequestMapping("fs")
public interface FileUploaderControllerApi {

    @ApiOperation(value = "上传用户头像", notes = "上传用户头像", httpMethod = "POST")
    @PostMapping("/uploadFace")
    public GraceJSONResult uploadFace(@RequestParam String userId,
                                      MultipartFile file) throws Exception;

实现上面API接口com.imooc.files.controller.FileUploaderController

package com.imooc.files.controller;


import com.imooc.api.controller.files.FileUploaderControllerApi;
import com.imooc.exception.GraceException;
import com.imooc.files.service.UploaderService;
import com.imooc.grace.result.GraceJSONResult;
import com.imooc.grace.result.ResponseStatusEnum;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

@RestController
public class FileUploaderController implements FileUploaderControllerApi {

    @Autowired
    private UploaderService uploaderService;

    final static Logger logger = LoggerFactory.getLogger(FileUploaderController.class);

    @Override
    public GraceJSONResult uploadFace(String userId,
                                      MultipartFile file) throws Exception {
        String path = "";
        if (file != null) {
            // 获取文件上传的名称
            String filename = file.getOriginalFilename();
            // 判断文件名不能为空
            if (StringUtils.isNotBlank(filename)){
                String fileNameArr[] = filename.split("\.");
                // 获得后缀
                String suffix = fileNameArr[fileNameArr.length - 1];
                // 判断后缀是否符合预定义规范 防止黑客乱传脚本攻击
                if (!suffix.equalsIgnoreCase("jpg") &&
                        !suffix.equalsIgnoreCase("png") &&
                        !suffix.equalsIgnoreCase("jpeg")){
                    return GraceJSONResult.errorCustom(ResponseStatusEnum.FILE_FORMATTER_FAILD);
                }
                // 执行上传
                path = uploaderService.uploadMinIO(file);
            }
        } else {
            GraceException.display(ResponseStatusEnum.FILE_UPLOAD_NULL_ERROR);
        }
        logger.info("path =" + path);
        return GraceJSONResult.ok(path);
    }
}

扫描api中的com.imooc之前的跨域解决。@ComponentScan(basePackages = {"com.imooc", "org.n3r.idworker"})

@SpringBootApplication(exclude = DataSourceAutoConfiguration.class)
@ComponentScan(basePackages = {"com.imooc", "org.n3r.idworker"})
public class Application {
    public static void main(String[] args) {
        ConfigurableApplicationContext run = SpringApplication.run(Application.class, args);

        MinioProp minioProp = run.getBean(MinioProp.class);
        System.out.println(minioProp.getEndpoint());
    }
}

运行user模块和file模块验证功能。

2.4 整合MinIO的Controller实现

common模块中添加图片大小控制的异常处理。com.imooc.exception.GraceExceptionHandler

    @ExceptionHandler(MaxUploadSizeExceededException.class)
    @ResponseBody
    public GraceJSONResult returnMaxUploadSizeExceededException(MaxUploadSizeExceededException e) {
        return GraceJSONResult.errorCustom(ResponseStatusEnum.FILE_MAX_SIZE_ERROR);
    }

测试验证。配置文件中设置的是500k的大小。

第三节云存储阿里OSS 3.1 阿里OSS优势

SDK使用简单
提供强大的文件处理功能
零运维成本
图形化管理控制台
CDN加速

3.2 阿里云接入

自己看java的api的接入文档就行
内容安全。阿里提供了一些自动审核，防止一些不良的信息。如果自己的oss，那肯定没有这些审核的。API的调用还是自己去看就行。
common中引入。

        
            com.aliyun.oss
            >aliyun-sdk-oss
            3.10.2
        

        
            com.aliyun
            >aliyun-java-sdk-green
            3.5.1

common编写工具类注入到容器中。com.imooc.utils.extend.AliImageReviewUtils

package com.imooc.utils.extend;

import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.aliyuncs.DefaultAcsClient;
import com.aliyuncs.IAcsClient;
import com.aliyuncs.green.model.v20180509.ImageSyncScanRequest;
import com.aliyuncs.http.FormatType;
import com.aliyuncs.http.HttpResponse;
import com.aliyuncs.http.MethodType;
import com.aliyuncs.http.ProtocolType;
import com.aliyuncs.profile.DefaultProfile;
import com.aliyuncs.profile.IClientProfile;
import com.imooc.enums.ArticleReviewLevel;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.Arrays;
import java.util.Date;
import java.util.UUID;

@Component
public class AliImageReviewUtils {

//    文档地址：https://help.aliyun.com/document_detail/70292.html?spm=a2c4g.11186623.2.49.6f9c75fdjaW30p#reference-fzy-ztm-v2b

    @Autowired
    private AliyunResource aliyunResource;

    public boolean reviewImage(String imgUrl) throws Exception {
        IClientProfile profile = DefaultProfile
                .getProfile("cn-shanghai", "", "");
        DefaultProfile
                .addEndpoint("cn-shanghai", "cn-shanghai", "Green", "green.cn-shanghai.aliyuncs.com");
        IAcsClient client = new DefaultAcsClient(profile);

        ImageSyncScanRequest imageSyncScanRequest = new ImageSyncScanRequest();
        // 指定api返回格式
        imageSyncScanRequest.setAcceptFormat(FormatType.JSON);
        // 指定请求方法
        imageSyncScanRequest.setMethod(MethodType.POST);
        imageSyncScanRequest.setEncoding("utf-8");
        //支持http和https
        imageSyncScanRequest.setProtocol(ProtocolType.HTTP);


        JSONObject httpBody = new JSONObject();
        
        httpBody.put("scenes", Arrays.asList("terrorism"));

        
        JSONObject task = new JSONObject();
        task.put("dataId", UUID.randomUUID().toString());

        //设置图片链接
        task.put("url", imgUrl);
        task.put("time", new Date());
        httpBody.put("tasks", Arrays.asList(task));

        imageSyncScanRequest.setHttpContent(org.apache.commons.codec.binary.StringUtils.getBytesUtf8(httpBody.toJSONString()),
                "UTF-8", FormatType.JSON);

        
        imageSyncScanRequest.setConnectTimeout(3000);
        imageSyncScanRequest.setReadTimeout(10000);
        HttpResponse httpResponse = null;
        try {
            httpResponse = client.doAction(imageSyncScanRequest);
        } catch (Exception e) {
            e.printStackTrace();
        }

        //服务端接收到请求，并完成处理返回的结果
        if (httpResponse != null && httpResponse.isSuccess()) {
            JSONObject scrResponse = JSON.parseObject(org.apache.commons.codec.binary.StringUtils.newStringUtf8(httpResponse.getHttpContent()));
            System.out.println(JSON.toJSONString(scrResponse, true));
            int requestCode = scrResponse.getIntValue("code");
            //每一张图片的检测结果
            JSONArray taskResults = scrResponse.getJSONArray("data");
            if (200 == requestCode) {
                for (Object taskResult : taskResults) {
                    //单张图片的处理结果
                    int taskCode = ((JSONObject) taskResult).getIntValue("code");
                    //图片要检测的场景的处理结果, 如果是多个场景，则会有每个场景的结果
                    JSONArray sceneResults = ((JSONObject) taskResult).getJSONArray("results");
                    if (200 == taskCode) {
                        Object sceneResult = sceneResults.get(0);
//                        for (Object sceneResult : sceneResults) {
                        String scene = ((JSONObject) sceneResult).getString("scene");
                        String suggestion = ((JSONObject) sceneResult).getString("suggestion");
                        //根据scene和suggetion做相关处理
                        //do something
                        System.out.println("scene = [" + scene + "]");
                        System.out.println("suggestion = [" + suggestion + "]");

                        return suggestion.equalsIgnoreCase(ArticleReviewLevel.PASS.type) ? true : false;
//                        }
                    } else {
                        //单张图片处理失败, 原因视具体的情况详细分析
                        System.out.println("task process fail. task response:" + JSON.toJSONString(taskResult));
                        return false;
                    }
                }
            } else {
                
                System.out.println("the whole image scan request failed. response:" + JSON.toJSONString(scrResponse));
                return false;
            }
        }
        return false;
    }

}

调用工具审核图片。根据策略审核，一张图片一个过滤条件为一次计费。

    @Autowired
    private AliImageReviewUtils aliImageReviewUtils;

    public static final String FAILED_IMAGE_URL = "https://imooc-news.oss-cn-shanghai.aliyuncs.com/image/faild.jpeg";
    private String doAliImageReview(String pendingImageUrl) {

        
        boolean result = false;
        try {
            result = aliImageReviewUtils.reviewImage(pendingImageUrl);
        } catch (Exception e) {
            e.printStackTrace();
        }

        if (!result) {
            return FAILED_IMAGE_URL;
        }

        return pendingImageUrl;
    }
    
// return GraceJSONResult.ok(doAliImageReview(finalPath)); 加到返回路径中

第三课 Spring Cloud分布式微服务实战-开发文件服务

Linux相关栏目本月热门文章