[LLaVA系列]CLIP/LLaVA/LLaVA1.5/VILA筆記

• Text Encoder: 對(duì)于每個(gè)句子，將其編碼成一個(gè)隱向量，維度(1,512)；N個(gè)句子，因此有，即[N, 512]

• Image Encoder: 對(duì)于每張img，將其編碼成一個(gè)隱向量，維度(1,512)；

  N張圖，因此有-，即[N, 512]

# image_encoder - ResNet or Vision Transformer# text_encoder - CBOW or Text Transformer# I[n, h, w, c] - minibatch of aligned images# T[n, l] - minibatch of aligned texts# W_i[d_i, d_e] - learned proj of image to embed# W_t[d_t, d_e] - learned proj of text to embed# t - learned temperature parameter# extract feature representations of each modalityI_f = image_encoder(I) #[n, d_i]T_f = text_encoder(T) #[n, d_t]# joint multimodal embedding [n, d_e]I_e = l2_normalize(np.dot(I_f, W_i), axis=1)T_e = l2_normalize(np.dot(T_f, W_t), axis=1)# scaled pairwise cosine similarities [n, n]logits = np.dot(I_e, T_e.T) * np.exp(t)# symmetric loss functionlabels = np.arange(n)loss_i = cross_entropy_loss(logits, labels, axis=0)loss_t = cross_entropy_loss(logits, labels, axis=1)loss = (loss_i + loss_t)/2

$ conda install --yes -c pytorch pytorch torchvision cudatoolkit
$ pip install ftfy regex tqdm
$ pip install git+https://github.com/openai/CLIP.git

import torchimport clipfrom PIL import Imagedevice = "cuda" if torch.cuda.is_available() else "cpu"model, preprocess = clip.load("ViT-B/32", device=device)image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)with torch.no_grad():
    image_features = model.encode_image(image)
    print("image_features shape:", image_features.shape) # [1, 512]
    text_features = model.encode_text(text)
    print("text_features shape:", text_features.shape) # [3, 512]
    
    logits_per_image, logits_per_text = model(image, text)
    print("logits_per_image shape:", logits_per_image.shape) # [1, 3]
    print("logits_per_text shape:", logits_per_text.shape) # [3, 1]
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()print("Label probs:", probs)  # prints: [[0.9927937  0.00421068 0.00299572]]print("      Label: {}".format(["a diagram", "a dog", "a cat"]))

• 階段一：特征對(duì)齊預(yù)訓(xùn)練。由于從CLIP提取的特征與word embedding不在同一個(gè)語(yǔ)義表達(dá)空間，因此，需要通過預(yù)訓(xùn)練，將image token embedding對(duì)齊到text word embedding的語(yǔ)義表達(dá)空間。這個(gè)階段凍結(jié)Vision Encoder和LLM模型的權(quán)重參數(shù)，只訓(xùn)練插值層Projection W的權(quán)重。

• 階段二：端到端訓(xùn)練。這個(gè)階段，依然凍結(jié)Vision Encoder的權(quán)重，訓(xùn)練過程中同時(shí)更新插值層Projection W和LLM語(yǔ)言模型的權(quán)重，訓(xùn)練考慮Multimodal Chatbot和Science QA兩種典型的任務(wù)。

• 實(shí)驗(yàn)結(jié)論：實(shí)驗(yàn)結(jié)果表明LLaVA在對(duì)話、細(xì)節(jié)描述和復(fù)雜推理等任務(wù)上均優(yōu)于BLIP-2。

• Vision Encoder分辨率：支持更大的分辨率，包括672x672, 336x1344, 1344x336 幾種分辨率的輸入，并且支持通過圖片裁切，編碼，合并來實(shí)現(xiàn)。

• LLM模型參數(shù)量大升級(jí)：由LLaVA 1.5的13B參數(shù)，增加到最多34B參數(shù)。

• OCR能力提升：更好的推理和OCR能力：通過修改指令數(shù)據(jù)集實(shí)現(xiàn)

• 更好的視覺對(duì)話：在一些場(chǎng)景下，擁有更好的世界知識(shí)