stanfordnlp使用报错“index

注：

标题所述报错信息为Windows+pytorch (cpu)所得

RuntimeError: index_select(): Expected dtype int32 or int64 for index

Ubuntu+pytorch (cuda)中对应报错信息为

RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float'

安装stanfordnlp + 调用：

pip install stanfordnlp

import stanfordnlp
stanfordnlp.download('en')

nlp = stanfordnlp.Pipeline()
doc = nlp("Barack Obama was born in Hawaii.  He was elected president in 2008.")
doc.sentences[0].print_dependencies()

环境 Windows + pytorch(cpu)为例：

Requirement already satisfied: stanfordnlp in c:userswanaconda3libsite-packages (0.2.0)
Requirement already satisfied: torch>=1.0.0 in c:userswanaconda3libsite-packages (from stanfordnlp) (1.8.1)

报错 Windows + pytorch(cpu)：

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 in 
      1 get_ipython().run_line_magic('pdb', 'on')
----> 2 doc = nlp("Barack Obama was born in Hawaii.  He was elected president in 2008.")

~anaconda3libsite-packagesstanfordnlppipelinecore.py in __call__(self, doc)
    174         if isinstance(doc, str) or isinstance(doc, list):
    175             doc = Document(doc)
--> 176         self.process(doc)
    177         return doc

~anaconda3libsite-packagesstanfordnlppipelinecore.py in process(self, doc)
    168         for processor_name in self.processor_names:
    169             if self.processors[processor_name] is not None:
--> 170                 self.processors[processor_name].process(doc)
    171         doc.load_annotations()
    172 

~anaconda3libsite-packagesstanfordnlppipelinelemma_processor.py in process(self, doc)
     64             edits = []
     65             for i, b in enumerate(seq2seq_batch):
---> 66                 ps, es = self.trainer.predict(b, self.config['beam_size'])
     67                 preds += ps
     68                 if es is not None:

~anaconda3libsite-packagesstanfordnlpmodelslemmatrainer.py in predict(self, batch, beam_size)
     86         self.model.eval()
     87         batch_size = src.size(0)
---> 88         preds, edit_logits = self.model.predict(src, src_mask, pos=pos, beam_size=beam_size)
     89         pred_seqs = [self.vocab['char'].unmap(ids) for ids in preds] # unmap to tokens
     90         pred_seqs = utils.prune_decoded_seqs(pred_seqs)

~anaconda3libsite-packagesstanfordnlpmodelscommonseq2seq_model.py in predict(self, src, src_mask, pos, beam_size)
    208                     done += [b]
    209                 # update beam state
--> 210                 update_state((hn, cn), b, beam[b].get_current_origin(), beam_size)
    211 
    212             if len(done) == batch_size:

~anaconda3libsite-packagesstanfordnlpmodelscommonseq2seq_model.py in update_state(states, idx, positions, beam_size)
    191                 br, d = e.size()
    192                 s = e.contiguous().view(beam_size, br // beam_size, d)[:,idx]
--> 193                 s.data.copy_(s.data.index_select(0, positions))
    194 
    195         # (3) main loop

RuntimeError: index_select(): Expected dtype int32 or int64 for index

报错 Ubuntu+ pytorch(cuda)：

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [15], in ()
      1 nlp = stanfordnlp.Pipeline()
----> 2 doc = nlp("Barack Obama was born in Hawaii.  He was elected president in 2008.")
      3 doc.sentences[0].print_dependencies()

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/pipeline/core.py:176, in Pipeline.__call__(self, doc)
    174 if isinstance(doc, str) or isinstance(doc, list):
    175     doc = Document(doc)
--> 176 self.process(doc)
    177 return doc

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/pipeline/core.py:170, in Pipeline.process(self, doc)
    168 for processor_name in self.processor_names:
    169     if self.processors[processor_name] is not None:
--> 170         self.processors[processor_name].process(doc)
    171 doc.load_annotations()

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/pipeline/lemma_processor.py:66, in LemmaProcessor.process(self, doc)
     64 edits = []
     65 for i, b in enumerate(seq2seq_batch):
---> 66     ps, es = self.trainer.predict(b, self.config['beam_size'])
     67     preds += ps
     68     if es is not None:

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/models/lemma/trainer.py:88, in Trainer.predict(self, batch, beam_size)
     86 self.model.eval()
     87 batch_size = src.size(0)
---> 88 preds, edit_logits = self.model.predict(src, src_mask, pos=pos, beam_size=beam_size)
     89 pred_seqs = [self.vocab['char'].unmap(ids) for ids in preds] # unmap to tokens
     90 pred_seqs = utils.prune_decoded_seqs(pred_seqs)

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/models/common/seq2seq_model.py:210, in Seq2SeqModel.predict(self, src, src_mask, pos, beam_size)
    208         done += [b]
    209     # update beam state
--> 210     update_state((hn, cn), b, beam[b].get_current_origin(), beam_size)
    212 if len(done) == batch_size:
    213     break

File ~/anaconda3/lib/python3.9/site-packages/stanfordnlp/models/common/seq2seq_model.py:193, in Seq2SeqModel.predict..update_state(states, idx, positions, beam_size)
    191 br, d = e.size()
    192 s = e.contiguous().view(beam_size, br // beam_size, d)[:,idx]
--> 193 s.data.copy_(s.data.index_select(0, positions))

RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float'

Bug出处 Windows + pytorch(cpu)为例：

C:UsersWanaconda3Libsite-packagesstanfordnlpmodelscommonbeam.py
line 84:

        prevK = bestScoresId / numWords

这里是想计算[beam_size * seq_len]长度的序列中第c个属于[beam_size, seq_len]矩阵中的第几列
即做整除、忽略余数，输出int64结果
但是当前pytorch-1.8.1里，这里会输出float32的结果，导致后续用此作为index时报错

解决方法：

将上述行改为：

		prevK = (bestScoresId / numWords).long()

希望对大家有帮助，不过看了下官网，这个工具应该是废弃了，新的替代品为Stanzahttps://github.com/stanfordnlp/stanza/