当
iterparse遍历整个文件时,将构建一棵树,并且不会释放任何元素。这样做的好处是元素可以记住其父元素是谁,并且您可以形成引用祖先元素的XPath。缺点是它会消耗大量内存。
为了在解析时释放一些内存,请使用Liza Daly的
fast_iter:
def fast_iter(context, func, *args, **kwargs): """ http://lxml.de/parsing.html#modifying-the-tree based on Liza Daly's fast_iter http://www.ibm.com/developerworks/xml/library/x-hiperfparse/ See also http://effbot.org/zone/element-iterparse.htm """ for event, elem in context: func(elem, *args, **kwargs) # It's safe to call clear() here because no descendants will be # accessed elem.clear() # Also eliminate now-empty references from the root node to elem for ancestor in elem.xpath('ancestor-or-self::*'): while ancestor.getprevious() is not None: del ancestor.getparent()[0] del context然后可以这样使用:
def process_element(elem): print "why does this consume all my memory?"context = lxml.etree.iterparse('really-big-file.xml', tag='schedule', events = ('end', ))fast_iter(context, process_element)我强烈推荐上述内容所依据的文章
fast_iter;如果您要处理大型XML文件,这对您来说应该特别有趣。
在
fast_iter上面介绍的文章中所示的一个略加修改的版本。这对于删除以前的祖先更具攻击性,从而节省了更多内存。



