Scraping the Teknoids Mailman PiperMail Archive

Putting this here in case anyone finds themselves in need of something to scrape a Pipermail web archive of a Mailman mailing list. This bit of Python 3 is based on a a bit of Python 2 I found at Scraping GNU Mailman Pipermail Email List Archives. The only changes I made from the original are to update somethings to work in Python 3. It works well for my purposes, generating a single text file of the teknoids list archive from 2005 to today.

#!/usr/bin/env python

import requests
from lxml import html
import gzip
from io import BytesIO

listname = 'teknoids'
url = 'https://lists.teknoids.net/pipermail/' + listname + '/'

response = requests.get(url)
tree = html.fromstring(response.text)

filenames = tree.xpath('//table/tr/td[3]/a/@href')

def emails_from_filename(filename):
print (filename)
response = requests.get(url + filename)
if filename[-3:] == '.gz':
contents = gzip.GzipFile(fileobj=BytesIO(response.content)).read()
else:
contents = response.content
return contents

contents = [emails_from_filename(filename) for filename in filenames]
contents.reverse()

contents = b"\n\n\n\n".join(contents)

with open(listname + '.txt', 'wb') as filehandle:
filehandle.write(contents)

Cambridge University Press to Preserve Cambridge Books Online with Portico

137 Years of Popular Science Now Available for Search and Reading

We’ve partnered with Google to offer our entire 137-year archive for free browsing. Each issue appears just as it did at its original time of publication, complete with period advertisements. It’s an amazing resource that beautifully encapsulates our ongoing fascination with the future, and science and technology’s incredible potential to improve our lives. We hope you enjoy it as much as we do.

Popular Science Archive

This is a pretty cool archive and it’s nice to see a magazine decide to make its archives open for search rather than throwing the whole thing behind a pay wall.