栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 面试经验 > 面试问答

Python NLTK pos_tag未返回正确的词性标签

面试问答 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

Python NLTK pos_tag未返回正确的词性标签

简而言之

NLTK并不完美。实际上,没有任何模型是完美的。

注意:

从NLTK版本3.1开始,默认

pos_tag
功能不再是旧的MaxEnt English pickle。

现在它是 感知恶搞 从@
Honnibal的实现
,见

nltk.tag.pos_tag

>>> import inspect>>> print inspect.getsource(pos_tag)def pos_tag(tokens, tagset=None):    tagger = PerceptronTagger()    return _pos_tag(tokens, tagset, tagger)

还是更好,但并不完美:

>>> from nltk import pos_tag>>> pos_tag("The quick brown fox jumps over the lazy dog".split())[('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

在某些时候,如果有人想要

TL;DR
解决方案,请参见https://github.com/alvations/nltk_cli


总而言之

尝试使用其他标记器(请参阅
https://github.com/nltk/nltk/tree/develop/nltk/tag
),例如

  • Hanbao
    Stanford POS
    Senna leaf

使用NLTK的默认MaxEnt POS标记器,即

nltk.pos_tag

>>> from nltk import word_tokenize, pos_tag>>> text = "The quick brown fox jumps over the lazy dog">>> pos_tag(word_tokenize(text))[('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')]

使用斯坦福POS标记器

$ cd ~$ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip$ unzip stanford-postagger-2015-04-20.zip$ mv stanford-postagger-2015-04-20 stanford-postagger$ python>>> from os.path import expanduser>>> home = expanduser("~")>>> from nltk.tag.stanford import POSTagger>>> _path_to_model = home + '/stanford-postagger/models/english-bidirectional-distsim.tagger'>>> _path_to_jar = home + '/stanford-postagger/stanford-postagger.jar'>>> st = POSTagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar)>>> text = "The quick brown fox jumps over the lazy dog">>> st.tag(text.split())[(u'The', u'DT'), (u'quick', u'JJ'), (u'brown', u'JJ'), (u'fox', u'NN'), (u'jumps', u'VBZ'), (u'over', u'IN'), (u'the', u'DT'), (u'lazy', u'JJ'), (u'dog', u'NN')]

使用HunPOS (注意:默认编码为ISO-8859-1而不是UTF8):

$ cd ~$ wget https://hunpos.googlepre.com/files/hunpos-1.0-linux.tgz$ tar zxvf hunpos-1.0-linux.tgz$ wget https://hunpos.googlepre.com/files/en_wsj.model.gz$ gzip -d en_wsj.model.gz $ mv en_wsj.model hunpos-1.0-linux/$ python>>> from os.path import expanduser>>> home = expanduser("~")>>> from nltk.tag.hunpos import HunposTagger>>> _path_to_bin = home + '/hunpos-1.0-linux/hunpos-tag'>>> _path_to_model = home + '/hunpos-1.0-linux/en_wsj.model'>>> ht = HunposTagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin)>>> text = "The quick brown fox jumps over the lazy dog">>> ht.tag(text.split())[('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')]

使用Senna (请确保您拥有最新版本的NLTK,并对API进行了一些更改):

$ cd ~$ wget http://ronan.collobert.com/senna/senna-v3.0.tgz$ tar zxvf senna-v3.0.tgz$ python>>> from os.path import expanduser>>> home = expanduser("~")>>> from nltk.tag.senna import SennaTagger>>> st = SennaTagger(home+'/senna')>>> text = "The quick brown fox jumps over the lazy dog">>> st.tag(text.split())[('The', u'DT'), ('quick', u'JJ'), ('brown', u'JJ'), ('fox', u'NN'), ('jumps', u'VBZ'), ('over', u'IN'), ('the', u'DT'), ('lazy', u'JJ'), ('dog', u'NN')]

或者尝试构建更好的POS标记器

  • Ngram Tagger:http://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
  • Affix / Regex Tagger:http://streamhacker.com/2008/11/10/part-of-speech-tagging-with-nltk-part-2/
  • 构建自己的Brill(阅读代码,这是一个非常有趣的标记器,http: //www.nltk.org/_modules/nltk/tag/brill.html ),请参阅http://streamhacker.com/2008/12/03/使用nltk-part-3进行语音标记的部分/
  • Perceptron Tagger:https://honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
  • LDA Tagger:http://scm.io/blog/hack/2015/02/lda-intentions/


转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/652148.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号