HuggingFace Transformers 4.17 : ガイド : 下流タスク用の再調整 – 要約 (翻訳/解説)

翻訳 : (株)クラスキャット セールスインフォメーション
作成日時 : 05/02/2022 (v4.17.0)

* 本ページは、HuggingFace Transformers の以下のドキュメントを翻訳した上で適宜、補足説明したものです：

• How-To Guide : Fine-Tune for Downstream Tasks : Summarization

* サンプルコードの動作確認はしておりますが、必要な場合には適宜、追加改変しています。
* ご自由にリンクを張って頂いてかまいませんが、sales-info@classcat.com までご一報いただけると嬉しいです。

HuggingFace Transformers : ガイド : 下流タスク用の再調整 – 要約

要約はドキュメントや記事のより短いバージョンを作成します、これは重要な情報の総てを捉えています。翻訳とともに、それは sequence-to-sequence タスクとして定式化できる別の例です。要約は以下のものであり得ます :

• Extractive : ドキュメントから最も関連性のある情報を抽出する。
  

• Abstractive : 最も関連性のある情報を捉えた新しいテキストを生成する。

このガイドは abstractive 要約のための BillSum データセットのカリフォルニア州 bill サブセット上で T5 を再調整する方法を示します。

Note : 関連するモデル, データセット, そしてメトリックスについての詳細は要約 タスクのページ を見てください。

BillSum データセットのロード

Datasets ライブラリから BillSum データセットをロードします :

from datasets import load_dataset

billsum = load_dataset("billsum", split="ca_test")

このデータセットを訓練とテストセットに分割します :

billsum = billsum.train_test_split(test_size=0.2)

そしてサンプルを見てみましょう :

billsum["train"][0]

{'summary': 'Existing law authorizes state agencies to enter into contracts for the acquisition of goods or services upon approval by the Department of General Services. Existing law sets forth various requirements and prohibitions for those contracts, including, but not limited to, a prohibition on entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between spouses and domestic partners or same-sex and different-sex couples in the provision of benefits. Existing law provides that a contract entered into in violation of those requirements and prohibitions is void and authorizes the state or any person acting on behalf of the state to bring a civil action seeking a determination that a contract is in violation and therefore void. Under existing law, a willful violation of those requirements and prohibitions is a misdemeanor.\nThis bill would also prohibit a state agency from entering into contracts for the acquisition of goods or services of $100,000 or more with a contractor that discriminates between employees on the basis of gender identity in the provision of benefits, as specified. By expanding the scope of a crime, this bill would impose a state-mandated local program.\nThe California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.\nThis bill would provide that no reimbursement is required by this act for a specified reason.',
 'text': 'The people of the State of California do enact as follows:\n\n\nSECTION 1.\nSection 10295.35 is added to the Public Contract Code, to read:\n10295.35.\n(a) (1) Notwithstanding any other law, a state agency shall not enter into any contract for the acquisition of goods or services in the amount of one hundred thousand dollars ($100,000) or more with a contractor that, in the provision of benefits, discriminates between employees on the basis of an employee’s or dependent’s actual or perceived gender identity, including, but not limited to, the employee’s or dependent’s identification as transgender.\n(2) For purposes of this section, “contract” includes contracts with a cumulative amount of one hundred thousand dollars ($100,000) or more per contractor in each fiscal year.\n(3) For purposes of this section, an employee health plan is discriminatory if the plan is not consistent with Section 1365.5 of the Health and Safety Code and Section 10140 of the Insurance Code.\n(4) The requirements of this section shall apply only to those portions of a contractor’s operations that occur under any of the following conditions:\n(A) Within the state.\n(B) On real property outside the state if the property is owned by the state or if the state has a right to occupy the property, and if the contractor’s presence at that location is connected to a contract with the state.\n(C) Elsewhere in the United States where work related to a state contract is being performed.\n(b) Contractors shall treat as confidential, to the maximum extent allowed by law or by the requirement of the contractor’s insurance provider, any request by an employee or applicant for employment benefits or any documentation of eligibility for benefits submitted by an employee or applicant for employment.\n(c) After taking all reasonable measures to find a contractor that complies with this section, as determined by the state agency, the requirements of this section may be waived under any of the following circumstances:\n(1) There is only one prospective contractor willing to enter into a specific contract with the state agency.\n(2) The contract is necessary to respond to an emergency, as determined by the state agency, that endangers the public health, welfare, or safety, or the contract is necessary for the provision of essential services, and no entity that complies with the requirements of this section capable of responding to the emergency is immediately available.\n(3) The requirements of this section violate, or are inconsistent with, the terms or conditions of a grant, subvention, or agreement, if the agency has made a good faith attempt to change the terms or conditions of any grant, subvention, or agreement to authorize application of this section.\n(4) The contractor is providing wholesale or bulk water, power, or natural gas, the conveyance or transmission of the same, or ancillary services, as required for ensuring reliable services in accordance with good utility practice, if the purchase of the same cannot practically be accomplished through the standard competitive bidding procedures and the contractor is not providing direct retail services to end users.\n(d) (1) A contractor shall not be deemed to discriminate in the provision of benefits if the contractor, in providing the benefits, pays the actual costs incurred in obtaining the benefit.\n(2) If a contractor is unable to provide a certain benefit, despite taking reasonable measures to do so, the contractor shall not be deemed to discriminate in the provision of benefits.\n(e) (1) Every contract subject to this chapter shall contain a statement by which the contractor certifies that the contractor is in compliance with this section.\n(2) The department or other contracting agency shall enforce this section pursuant to its existing enforcement powers.\n(3) (A) If a contractor falsely certifies that it is in compliance with this section, the contract with that contractor shall be subject to Article 9 (commencing with Section 10420), unless, within a time period specified by the department or other contracting agency, the contractor provides to the department or agency proof that it has complied, or is in the process of complying, with this section.\n(B) The application of the remedies or penalties contained in Article 9 (commencing with Section 10420) to a contract subject to this chapter shall not preclude the application of any existing remedies otherwise available to the department or other contracting agency under its existing enforcement powers.\n(f) Nothing in this section is intended to regulate the contracting practices of any local jurisdiction.\n(g) This section shall be construed so as not to conflict with applicable federal laws, rules, or regulations. In the event that a court or agency of competent jurisdiction holds that federal law, rule, or regulation invalidates any clause, sentence, paragraph, or section of this code or the application thereof to any person or circumstances, it is the intent of the state that the court or agency sever that clause, sentence, paragraph, or section so that the remainder of this section shall remain in effect.\nSEC. 2.\nSection 10295.35 of the Public Contract Code shall not be construed to create any new enforcement authority or responsibility in the Department of General Services or any other contracting agency.\nSEC. 3.\nNo reimbursement is required by this act pursuant to Section 6 of Article XIII\u2009B of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIII\u2009B of the California Constitution.',
 'title': 'An act to add Section 10295.35 to the Public Contract Code, relating to public contracts.'}

text フィールドが入力で summary フィールドがターゲットです。

前処理

text と summary を処理するために T5 トークナイザーをロードします :

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("t5-small")

前処理は以下を必要とします :

1. T5 がこれが要約タスクであることを知るように、入力をプロンプトで prefix します。複数の NLP タスクが可能な幾つかのモデルは特定のタスクに対してプロンプトを必要とします。
   

2. 入力とラベルのトークン化を並列化するために as_target_tokenizer() 関数でコンテキストマネージャを使用します。
   

3. max_length パラメータで設定された最大長よりも長くならないようにシークエンスを切り詰めます。

prefix = "summarize: "


def preprocess_function(examples):
    inputs = [prefix + doc for doc in examples["text"]]
    model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["summary"], max_length=128, truncation=True)

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

データセット全体に対して前処理関数を適用するために Datasets map 関数を使用します。データセットの複数の要素を一度に処理する batched=True を設定することにより map 関数を高速化できます :

tokenized_billsum = billsum.map(preprocess_function, batched=True)

サンプルのバッチを作成するために DataCollatorForSeq2Seq を使用します。それはまたバッチ内の最長要素の長さにテキストとラベルを動的にパディングしますので、それらは均一な長さです。padding=True を設定することでトークナイザーの関数でテキストをパディングすることも可能ですが、動的パディングはより効率的です。

from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model, return_tensors="tf")

Trainer で再調整

AutoModelForSeq2SeqLM で T5 をロードします :

from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")

この時点で、3 つのステップだけが残っています :

1. Seq2SeqTrainingArguments で訓練ハイパーパラメータを定義します。
   

2. モデル, データセット, トークナイザー, そしてデータ collator と共に訓練引数を Seq2SeqTrainer に渡します。
   

3. モデルを再調整するために train() を呼び出します。

training_args = Seq2SeqTrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=1,
    fp16=True,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_billsum["train"],
    eval_dataset=tokenized_billsum["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

TensorFlow による再調整

TensorFlow でモデルを再調整することは、幾つかの違いはありますが、同様に簡単です。

データセットを to_tf_dataset で tf.data.Dataset 形式に変換します。columns で入力とラベルを、データセット順序をシャッフルするか否か、バッチサイズ、そしてデータ collator を指定します :

tf_train_set = tokenized_billsum["train"].to_tf_dataset(
    columns=["attention_mask", "input_ids", "labels"],
    shuffle=True,
    batch_size=16,
    collate_fn=data_collator,
)

tf_test_set = tokenized_billsum["test"].to_tf_dataset(
    columns=["attention_mask", "input_ids", "labels"],
    shuffle=False,
    batch_size=16,
    collate_fn=data_collator,
)

optimizer 関数, 学習率スケジュール, そして幾つかの訓練ハイパーパラメータをセットアップします :

from transformers import create_optimizer, AdamWeightDecay

optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)

TFAutoModelForSeq2SeqLM で T5 をロードします :

from transformers import TFAutoModelForSeq2SeqLM

model = TFAutoModelForSeq2SeqLM.from_pretrained("t5-small")

compile で訓練のためにモデルを configure します :

model.compile(optimizer=optimizer)

モデルを再調整するために fit を呼び出します :

model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3)

Note : 要約のためにモデルを再調整する方法の詳細なサンプルについては、対応する PyTorch ノートブック か TensorFlow ノートブック を見てください。

以上