or, The Hitchhiker’s Guide to Fear and Loathing at a Public Library Reference Desk

The Resurrection of Newspaper Obituaries

   September 7th, 2010 Brian Herzog

Yahoo PipesLast week I started talking about newspaper obituaries. Today's post details how we're improving access to the obituaries we do have in our newspaper microfilm records, using an online index created with Yahoo Pipes.

Our microfilm records of the local papers go back to 1940. But microfilm is primarily an archival format, rather than an accessible format, so it can be cumbersome to use. Our biggest impediment was that we didn't know what was there - when a patron contacted the Reference Desk asking for someone's obituary, it was very time-consuming for us to search the microfilm for an obituary, which may or may not have even appeared in the paper - we wouldn't even know until we checked.

So we created an online searchable index to the newspaper's obituaries - not the text of the obituaries, just a name/date/page index. Patrons and staff can use this to know whether someone's obituary appeared in our newspaper, instead of having to check the microfilm every time.

Here's how we did it: first, for about the past 10 months, volunteers have been going through every microfilm reel we have, page by page, and building an Excel spreadsheet with the following information:

Newspaper Year Month Day Page FirstName MiddleInitial LastName Maiden-Jr-Sr

The first column is necessary because we have records for both the Chelmsford Newsweekly (1940-1993) and the Chelmsford Independent (1986-present). The middle columns are reference and retrieval information. In the last column, we included extra information, like maiden name, whether a person was a "Jr." or "Sr." etc., and anything else that was random and didn't fit into another column.

The spreadsheet itself is useful, but I wanted to put this online so anyone could search it. The tool I chose was Yahoo Pipes, which has both pros and cons:


  • It's easy to play with and learn (like most Web 2.0 tools), but is also very powerful so we can grow into it
  • It can use a csv file for the data, which is easy to create with Excel
  • Beyond a simple search, it also provides fancy features like RSS feeds and tie-ins with other social media tools
  • Using Yahoo Pipes is covered in Chapter 7 of Library Mashups, written by Nicole Engard
  • The data is easy to update as the file continues to grow
  • It worked


  • Searching a database is not what Pipes is intended to do, so it's probably not the best tool out there (I wanted to use DabbleDB, but they're in transition right now)
  • The csv file must be ftp'ed to the webserver, which will be increasingly problematic - right now the file is 17,000+ lines and over 1MB. It will only get bigger, and the entire thing needs to be uploaded each time it's updated
  • Pipes has funny rules that you don't know about until something breaks. For instance, field names must be single words (hence "FirstName" and "Maiden-Jr-Sr"), you can't use certain characters in the data (like /), the search doesn't let you combine keywords (so far - I'm sure there must be some kind of fancy loop setup that will allow it, but right now people can only search either by first name or last name or year)
  • There isn't an easy way to embed the search box back into our website (there are Badge options, but only for search output) - you have to use the Pipe interface to search
  • There doesn't seem to be a wildcard for search
  • The results can't not link to something - I wanted the names and dates just to be displayed, but the way Pipes works requires the results to link to something

The last point was initially a pain, but it forced me to be creative, and I think the solution is actually more helpful for patrons than what I originally wanted. Now, when a patron finds the obituary listing they'd like to read, they click the link, and it automatically fills the obituary information into an email contact form on our website. That request gets sent to Reference staff, who then have an easy time of retrieving the obituary from the microfilm. Unfortunately, our microfilm machine isn't connected to a computer, so we'll just print and mail or fax the obituary to the patron. When possible we'll type them in and email them, and of course that will go into the searchable database too.

To make the connection from the Pipes listing to our email form, I had to use some javascript (which introduced another glitch: javascript makes names like O'Conner problematic, because it stops at the ', but I'll worry about this later).

Here's what the whole Pipe's source code looks like:
Yahoo Pipe for Obituary Search

Here's what it does:

  • The "Fetch CSV" module is the path to the csv file on our webserver
  • The module to the right of that controls what the patron search input box looks like. The "Label" field is "Enter EITHER a First Name, Last Name OR Year:" and you can see where that displays on the Pipe page
  • Both of those modules feed into "Filter" module - this one takes what the patron enters into the search box and filters the data from the csv file to create a subset of just matching records. Whatever the patron enters gets searched for in all the fields listed in the "Filter" module
  • The next module is "Rename" and I'm not sure I'm using it properly - I needed to create two new fields, so I'm just taking two existing fields, copying them, and renaming them so I can work with them later. The fields that got copied still exist untouched
  • Next is the "Regex" module, which is the most complicated and powerful, and I use it to create what the patron sees for the search results. The "Title" field is one I created, and here I'm replacing the contents from when I copied it to display what the patron will see on the screen - the code for it is "${FirstName} ${MiddleInitial} ${LastName} ${Maiden-Jr-Sr} - ${Newspaper}, ${Month} ${Day}, ${Year}, Page ${Page} ${Obituary}" which also includes punctuation formatting. So, for example, the result looks like this:

    Katherine M. Polley - Chelmsford Newsweekly, December 31, 1940, Page 7

    Because this field has to be a link, I also had to define what it links to, which is what I'm doing in the "Link" field. The value for that field is being written as


    which carries the data over to the library's website and some javascript pulls the data from the url and puts it in an email form. The patron can fill in their name and contact info into the form and submit it to us as an email message

  • The "Sort" module is self-explanatory, and I chose to list them with most recent first

This feels far more complicated than it should be, and I'm sharing it here to both save someone else from having to figure it all out again on their own, and to hopefully get suggestions on how to simplify/improve it.

Although, speaking of improving it, I do have one idea for future development: the local Cemetery Department has spreadsheet online listing complete burial locations - it would be neat to mashup up that data, so the obituary is linked to the cemetery plot location.

That's down the road a bit, so in the meantime I just keep adding whatever new obituaries appear in the paper to the csv data file - I had planned to do that weekly, but lately there have been many weeks without any obituaries in the paper (see my previous post). Anyway, we'll see how this works - it only went live last week, but already patrons have been using it, and it certainly does save a lot of staff time.

Tags: , , , , , , , , , , , , , ,

The Death of Newspaper Obituaries

   September 2nd, 2010 Brian Herzog

Newspaper MONEY SectionOne function of public libraries is to be a repository for community history. The extent to which a library can do this will vary, but at the very least, the library has holdings of the local newspaper, which patrons can use to look up obituaries of local residents.

But the reality of this is changing. As newspapers struggle to stay alive, they're exploring new revenue streams - our local paper recently started charging families to list obituaries, instead of providing that service for free. The paper is only published once a week for a town of 32,000 residents, but you can still see the effect below:

Year #/Obits
2000 444
2001 527
2002 523
2003 566
2004 556
2005 479
2006 500
2007 220
2008 215
2009 80
2010 26 (as of Aug.)

And of the 80 obituaries in 2009, only 12 were from June-December. With dramatically fewer obituaries appearing in the paper, the long-term research value of a library's newspaper holdings is diminished. There must be other factors at play too, but hopefully newspaper revenues will stabilize and this downward obituary trend will be reversed. Regardless, there will always at least be a gap for anyone doing genealogical research or just looking up a friend of family member.

And this doesn't seem to be just a local thing. A Slashdot post describes the same thing on a bigger scale. There's also a Boing Boing post that looks into Legacy.com, the company many newspapers are using to outsource obituary listings. The bottom line in both posts is that obituaries and death notices are turning into a cash cow business - and as it becomes more and more expensive to run an obituary, there are going to be fewer and fewer of them.

So, all of that is sad news - doubly so since it's out of the control of libraries (unless we start publishing family-written obituaries on our own websites for free). But at my library, we have been working to improve access to what we do have. Tune in next week for Part Two of this post, detailing how we created an online index to the obituaries in our newspaper microfilm records, to make then easier for patrons and staff to locate.

Tags: , , , , , , , ,

Reference Question of the Week – 10/11/09

   October 17th, 2009 Brian Herzog

Instead of a reference question this week, here's a good example of another question-of-the-week service:

The Seattle Public Library has a regular feature on the website of a local paper, the Seattle Post-Intelligencer. It's titled Shelf Talk, and in addition to general library content, it also features interesting reference questions (and their answers).

I think it's a great idea for libraries to have regular columns in the paper, in addition to events listings. It's not only entertaining and informative, but also promotes the library's reference service, subtly reminding people they can get help with tough questions at the library. And not surprisingly, Seattle is doing an excellent job of it.

via LISNews

Tags: , , , , , , , , ,

New York Times x2

   December 11th, 2008 Brian Herzog

New York Times front page from July 29th, 1974Yesterday must have been National New York Times day - I learned two news ways to access articles from the NYT, completely coincidentally.

The first way was the arrival of a book I ordered for our Reference Collection, The New York Times: The Complete Front Pages: 1851-2008. The oversized book itself is 300+ front pages from significant days since 1851, and it also comes with every front page contained on a set of DVDs.

Of course, the first thing we did was look up our birthdays, and so far no one was born on a significant day. But we found them in the very easy-to-use, PDF-based, DVDs. No special programs need to be installed, everything worked first time, etc.

Which makes me more comfortable having this book in the Reference Collection - people don't need to take it home if the DVDs work flawlessly on our computers, and the PDFs are ready-formatted to print on 8.5"x11" paper. Reading them is easier electronically when you can zoom in, but the book also comes complete with magnifying lens.

The second way was through Google News search. Not that I was surprised, but I had just never noticed before that Google News added "Archives" links to the left side of the search results page. Clicking into the 1800's, the matches were for-pay links into the Washington Post, but also free full-text links into the NYT. I knew the NYT had made their archive available, but having their articles show up in a context search like this is very useful. Plus, when you click through into the story, there is a link for a PDF version of the original newsprint, which I think qualifies as a primary source.

So, a good day for historical research using the New York Times.

Tags: , , , , , , , , , , , , , ,

Free Online Historical Newspapers

   January 10th, 2008 Brian Herzog

Access NewspaperARCHIVE logoA few months ago I got an email about a website called Access NewspaperARCHIVE, saying that libraries could signup for free access to historical newspapers, dating back to the 1700s.

Sweet. I'm always looking for good primary source resources, especially online ones (and especially-especially free ones), so I thought I'd check this out. The signup process was a bit odd, having to download and then fax in their signup form [pdf, 418 kb]. I didn't hear anything back from them for months, so one day I just tried their url again (from within the library) and it IP-authenticated me.

So, I took that as us being signed up, and I started playing. The database is neat, as all the newspapers in there are saved as PDF files (see the 7/29/1895 Sandusky Register). And some are older than I could find in our other available resources, so those are two great things in its favor. However, I did see some drawbacks:

  • In-Library use only. And right on the authenticated homepage (the one patrons would see by logging in at the library) is a link to "Sign up for a home account." Which isn't expensive, but it's not free. It's just a little bit underhanded to give libraries a free account and then use that as a vehicle to sell to our patrons. So, I bypass this page and go right to the Browse page
  • No keyword searching. You can only browse by location, date, or newspaper title. Which will be fine for the "what happened on my birthday" questions, or if you were just looking up anything old in your area, but eliminates searching for a topic. And, the browse tool and the results listing are kind of clunky
  • No Massachusetts Newspapers. Which is a pain, since I mainly serve Massachusetts patrons. So, I guess no local historical information for me
  • Front pages only? For the papers I viewed, it wasn't the entire paper but just the front page. That's a pain
  • Not high-quality scans. The newspapers are legible, both on screen and printed, but they are just a little bit too bitmapped. And they are images, rather than text-based, which means no copy/pasting

So, my overall verdict is this: it's an amazing resource for primary source newspapers, and it's free, so it's better than nothing. There are some drawbacks, but I am rarely completely satisfied anyway.

Something else I did like was they had a "Questions? Ask a Librarian" link. This is an email link to whatever email address you supplied on the signup form. Which is good, since my patrons using this will be able to write to me, instead of this company.

Anyway, this is available, so I'm going to give it a try. If anyone has experience with this company or database, please comment below and let us know what you think. Thanks.

access, database, databases, historic, historical, libraries, library, newspaper, newspaperarchive, newspapers, primary, public, source, sources

Tags: , , , , , , , , , , , , ,