GPT-4終結(jié)人工標(biāo)注！AI標(biāo)注比人類標(biāo)注效率高100倍，成本僅1/7

---

  新智元報(bào)道  

Autolabel安裝

pip install 'refuel-autolabel[openai]'

from autolabel import get_dataget_data('civil_comments')

Downloading seed example dataset to "seed.csv"...100% [..............................................................................] 65757 / 65757Downloading test dataset to "test.csv"...100% [............................................................................] 610663 / 610663

   

    

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    
    
config = {    "task_name": "ToxicCommentClassification",    "task_type": "classification", # classification task    "dataset": {        "label_column": "label",    },    "model": {        "provider": "openai",        "name": "gpt-3.5-turbo" # the model we want to use    },    "prompt": {        # very simple instructions for the LLM        "task_guidelines": "Does the provided comment contain 'toxic' language? Say toxic or not toxic.",        "labels": [ # list of labels to choose from            "toxic",            "not toxic"        ],        "example_template": "Input: {example}\nOutput: {label}"    }}

   

如果要創(chuàng)建自定義配置，可以使用CLI或編寫自己的配置。

   

    

     

     

     

     

     

     

    
    
from autolabel import LabelingAgent, AutolabelDataset
agent = LabelingAgent(config)ds = AutolabelDataset('test.csv', config = config)agent.plan(ds)

   
   

    輸出：
    

   

   

    

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    
    
┌──────────────────────────┬─────────┐│ Total Estimated Cost     │ $4.4442 ││ Number of Examples       │ 2000    ││ Average cost per example │ $0.0022 │└──────────────────────────┴─────────┘───────────────────────────────────────────────── Prompt Example ──────────────────────────────────────────────────Does the provided comment contain 'toxic' language? Say toxic or not toxic.
You will return the answer with just one element: "the correct label"
Now I want you to label the following example:Input: [ Integrity means that you pay your debts.]. Does this apply to President Trump too?Output:

   

最后，進(jìn)行數(shù)據(jù)標(biāo)注:

   

    

     

    
    
ds = agent.run(ds, max_items=100)
   
   

    

     

     

     

     

     

     

    
    
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓┃ support ┃ threshold ┃ accuracy ┃ completion_rate ┃┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩│ 100     │ -inf      │ 0.54     │ 1.0             │└─────────┴───────────┴──────────┴─────────────────┘

   

輸出結(jié)果為54%的準(zhǔn)確率不是很好，進(jìn)一步改進(jìn)的具體方法可以訪問以下鏈接查看：

結(jié)果