python去掉csv文件中重复的行

with open('test.csv','r') as in_file, open('test1.csv','w') as out_file:

    s = set() # set for fast O(1) amortized lookup

    for line in in_file:

        if line in s: continue # skip duplicate

        s.add(line)

        out_file.write(line)

这个代码有个bug，当重复的列是整齐排列的，那么最后一个重复的元素不会去掉所有重复元素，它会剩下两个.(偶尔会出现这个bug，调了一晚上最后好了，手动狗头)