Skip to content

车牌识别场景下 FunASR 微调标注规范及 AutoModelVLLM 逆文本正则化 (ITN) 传参问题 #3611

Description

@CurtainRight

Question

在针对挪车场景进行 FunASR 语音识别模型微调时,遇到以下几个关于标注规则与效果的问题:

1、标注范围选择:在微调数据标注时,是只截取并标注录音中说出车牌号的具体片段(如 3-5 秒),还是需要保留并标注整段完整的挪车客服/人机对话录音?

2、重复与确认语音的标注策略:对于用户在说话过程中出现的重复、修正或确认语气(例如:“您报车牌号鄂A鄂A六六七T二五F二五F啊。六七二五。”),标注文本应该按实际语音所说的完整字词(如:鄂A鄂A六六七T二五F二五F啊六七二五)标注,还是按语义归一化后的最终车牌(如:鄂A6T725)进行标注?

3、微调效果评估:在针对此类高频重复、口语化重叠的垂直场景(挪车场景)下,直接使用语音识别(ASR)模型微调来解决车牌提取问题效果如何?是否有推荐的最佳实践(例如:ASR 负责逐字识别 + 后处理/LLM 负责规整提取)?

4、中文数字转阿拉伯数字参数:FunASR 在推理阶段是否有现成的配置参数或内置逆文本正则化(ITN / Text Normalization)工具,可以将识别出的中文数字(如“六六七”)直接输出为阿拉伯数字(如“667”)?

Code or command

from funasr.auto.auto_model_vllm import AutoModelVLLM

def main():
    model = AutoModelVLLM(
        model="model/FunAudioLLM--Fun-ASR-Nano-2512/snapshots/master",
        model_conf={"lora_path": "output_lora_model/model.pt.best"},
        hub="ms",
        tensor_parallel_size=1,
        gpu_memory_utilization=0.3,
        itn=True,            # 在模型级别开启默认 ITN
    )

    results = model.generate(
        ["/workspace/processed_16k_wavs/134.200.46.9_5090-1787544499.850172_16k.wav", "/workspace/processed_16k_wavs/134.200.46.9_5090-1787546051.851120_16k.wav"],
        language="zh",
        hotwords=["鄂"],
        # itn=True,            # 在模型级别开启默认 ITN
    )
    for r in results:
        print(f"[{r['key']}] {r['text']}")
    print("热词为鄂A")
    results = model.generate(
        ["/workspace/processed_16k_wavs/134.200.46.9_5090-1787544499.850172_16k.wav", "/workspace/processed_16k_wavs/134.200.46.9_5090-1787546051.851120_16k.wav"],
        language="zg",
        hotwords=["鄂A"],
        # itn=True,            # 在模型级别开启默认 ITN
    )
    for r in results:
        print(f"[{r['key']}] {r['text']}")

if __name__ == '__main__':
    main()

`
#!/usr/bin/env bash

设置显卡

export CUDA_VISIBLE_DEVICES="0"

启动单卡微调

torchrun --nproc_per_node=1
-m funasr.bin.train_ds
++model="model/FunAudioLLM--Fun-ASR-Nano-2512/snapshots/master"
++trust_remote_code=true
++train_data_set_list="/workspace/train/data/train/train.jsonl"
++valid_data_set_list="/workspace/train/data/train/train.jsonl"
++dataset="AudioDataset"
++dataset_conf.index_ds="IndexDSJsonl"
++dataset_conf.data_split_num=1
++dataset_conf.batch_sampler="BatchSampler"
++dataset_conf.batch_size=2000
++dataset_conf.sort_size=1024
++dataset_conf.batch_type="token"
++dataset_conf.num_workers=4
++train_conf.max_epoch=20
++train_conf.log_interval=10
++train_conf.resume=true
++train_conf.validate_interval=500
++train_conf.save_checkpoint_interval=500
++train_conf.keep_nbest_models=5
++train_conf.avg_nbest_model=3
++train_conf.use_deepspeed=false
++optim_conf.lr=0.0001
++use_lora=true
++lora_rank=8
++lora_alpha=16
++lora_target_modules="['qkv', 'out']"
++audio_encoder_conf.freeze=true
++audio_adaptor_conf.freeze=true
++llm_conf.freeze=false
++output_dir="./output_lora_model"
`

What have you tried?

查阅了 FunASR 文档中关于 ITN(Inverse Text Normalization)及后处理的相关说明。

尝试通过修改训练集文本进行初步测试,但在应对用户反复确认、纠错的复述口语时,模型容易出现漏字或多字的情况。

Environment

OS: Linux (Ubuntu 22.04)

Python version: 3.10

FunASR version: 1.4.8

ModelScope version: 最新

PyTorch / torchaudio version: 2.13.0

Install method (pip, source, Docker): pip

Device (cuda, cpu, mps): cuda

GPU model: NVIDIA RTX 4090

CUDA/cuDNN version: 13.2

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageNeeds maintainer triage and routingquestionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions