中文多模态 图文对比学习模型CLIP学习

参考：https://huggingface.co/IDEA-CCNL/Taiyi-CLIP-Roberta-102M-Chinese
https://github.com/OFA-Sys/Chinese-CLIP

主要用的这个，他还是用的官方openai 的CLIP模型结构，只单独把文本塔换成中文，图片塔还是用训练好的vit模型，训练的时候也把图像模型冻结；所以图像还是用openai 的vit模型，文本模型用这里的中文模型Taiyi-CLIP-Roberta-102M-Chinese

***官方案例

from PIL import Image
import requests
import clip
import torch
from transformers import BertForSequenceClassification, BertConfig, BertTokenizer
import numpy as np

# 加载Taiyi 中文 text encoder
text_tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-102M-Chinese")
text_encoder = BertForSequenceClassification.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-102M-Chinese").eval()
text = text_tokenizer(["一只猫", "一只狗",'两只猫', '两只老虎','一只老虎'], return_tensors='pt', padding=True)['input_ids']

# 加载CLIP的image encoder
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
clip_model, preprocess = clip.load("ViT-B/32", device='cpu')
image = preprocess(Image.open(requests.get(url, stream=True).raw)).unsqueeze(0)

with torch.no_grad():
    image_features = clip_model.encode_image(image)
    text_features = text_encoder(text).logits
    # 归一化
    image_features = image_features / image_features.norm(dim=1, keepdim=True)
    text_features = text_features / text_features.norm(dim=1, keepdim=True)
    # 计算余弦相似度 logit_scale是尺度系数
    logit_scale = clip_model.logit_scale.exp()
    logits_per_image = logit_scale * image_features @ text_features.t()
    logits_per_text = logits_per_image.t()
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()
    print(np.around(probs, 3))

*** 应用案例（文字query搜索图片）

参考：https://blog.csdn.net/weixin_42357472/article/details/115482718

还是用的上面参考文章里的代码结构，只是更改了文本加载和使用模块
*** transformers 转pt参考：https://blog.csdn.net/weixin_42357472/article/details/125749246

# 加载Taiyi 中文 text encoder
text_tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-102M-Chinese")
# text_encoder = BertForSequenceClassification.from_pretrained("IDEA-CCNL/Taiyi-CLIP-Roberta-102M-Chinese").eval()
text_encoder = torch.load(r'D*****text_encoder.pt')

##使用
text1 = text_tokenizer([query], return_tensors='pt', padding=True)['input_ids'] 

text_features = text_encoder(text1).logits
text_features /= text_features.norm(dim=-1, keepdim=True)

##相似图像召回（这里不用logit_scale效果感受差不多）
#logit_scale = model.logit_scale.exp()
 #similarities = (logit_scale * image @ text_features.T).squeeze(1)
similarities = ( image @ text_features.T).squeeze(1)
best_photo_idx = np.argsort(similarities[:, 0].numpy())[::-1]
print(best_photo_idx)
rank_results = [aidss[i] for i in best_photo_idx[:50]]

搜索：两只小狗

搜索：消防车