or, The Hitchhiker’s Guide to Fear and Loathing at a Public Library Reference Desk


The Resurrection of Newspaper Obituaries

   September 7th, 2010 Brian Herzog

Yahoo PipesLast week I started talking about newspaper obituaries. Today's post details how we're improving access to the obituaries we do have in our newspaper microfilm records, using an online index created with Yahoo Pipes.

Our microfilm records of the local papers go back to 1940. But microfilm is primarily an archival format, rather than an accessible format, so it can be cumbersome to use. Our biggest impediment was that we didn't know what was there - when a patron contacted the Reference Desk asking for someone's obituary, it was very time-consuming for us to search the microfilm for an obituary, which may or may not have even appeared in the paper - we wouldn't even know until we checked.

So we created an online searchable index to the newspaper's obituaries - not the text of the obituaries, just a name/date/page index. Patrons and staff can use this to know whether someone's obituary appeared in our newspaper, instead of having to check the microfilm every time.

Here's how we did it: first, for about the past 10 months, volunteers have been going through every microfilm reel we have, page by page, and building an Excel spreadsheet with the following information:

Newspaper Year Month Day Page FirstName MiddleInitial LastName Maiden-Jr-Sr

The first column is necessary because we have records for both the Chelmsford Newsweekly (1940-1993) and the Chelmsford Independent (1986-present). The middle columns are reference and retrieval information. In the last column, we included extra information, like maiden name, whether a person was a "Jr." or "Sr." etc., and anything else that was random and didn't fit into another column.

The spreadsheet itself is useful, but I wanted to put this online so anyone could search it. The tool I chose was Yahoo Pipes, which has both pros and cons:

Pros:

  • It's easy to play with and learn (like most Web 2.0 tools), but is also very powerful so we can grow into it
  • It can use a csv file for the data, which is easy to create with Excel
  • Beyond a simple search, it also provides fancy features like RSS feeds and tie-ins with other social media tools
  • Using Yahoo Pipes is covered in Chapter 7 of Library Mashups, written by Nicole Engard
  • The data is easy to update as the file continues to grow
  • It worked

Cons:

  • Searching a database is not what Pipes is intended to do, so it's probably not the best tool out there (I wanted to use DabbleDB, but they're in transition right now)
  • The csv file must be ftp'ed to the webserver, which will be increasingly problematic - right now the file is 17,000+ lines and over 1MB. It will only get bigger, and the entire thing needs to be uploaded each time it's updated
  • Pipes has funny rules that you don't know about until something breaks. For instance, field names must be single words (hence "FirstName" and "Maiden-Jr-Sr"), you can't use certain characters in the data (like /), the search doesn't let you combine keywords (so far - I'm sure there must be some kind of fancy loop setup that will allow it, but right now people can only search either by first name or last name or year)
  • There isn't an easy way to embed the search box back into our website (there are Badge options, but only for search output) - you have to use the Pipe interface to search
  • There doesn't seem to be a wildcard for search
  • The results can't not link to something - I wanted the names and dates just to be displayed, but the way Pipes works requires the results to link to something

The last point was initially a pain, but it forced me to be creative, and I think the solution is actually more helpful for patrons than what I originally wanted. Now, when a patron finds the obituary listing they'd like to read, they click the link, and it automatically fills the obituary information into an email contact form on our website. That request gets sent to Reference staff, who then have an easy time of retrieving the obituary from the microfilm. Unfortunately, our microfilm machine isn't connected to a computer, so we'll just print and mail or fax the obituary to the patron. When possible we'll type them in and email them, and of course that will go into the searchable database too.

To make the connection from the Pipes listing to our email form, I had to use some javascript (which introduced another glitch: javascript makes names like O'Conner problematic, because it stops at the ', but I'll worry about this later).

Here's what the whole Pipe's source code looks like:
Yahoo Pipe for Obituary Search

Here's what it does:

  • The "Fetch CSV" module is the path to the csv file on our webserver
  • The module to the right of that controls what the patron search input box looks like. The "Label" field is "Enter EITHER a First Name, Last Name OR Year:" and you can see where that displays on the Pipe page
  • Both of those modules feed into "Filter" module - this one takes what the patron enters into the search box and filters the data from the csv file to create a subset of just matching records. Whatever the patron enters gets searched for in all the fields listed in the "Filter" module
  • The next module is "Rename" and I'm not sure I'm using it properly - I needed to create two new fields, so I'm just taking two existing fields, copying them, and renaming them so I can work with them later. The fields that got copied still exist untouched
  • Next is the "Regex" module, which is the most complicated and powerful, and I use it to create what the patron sees for the search results. The "Title" field is one I created, and here I'm replacing the contents from when I copied it to display what the patron will see on the screen - the code for it is "${FirstName} ${MiddleInitial} ${LastName} ${Maiden-Jr-Sr} - ${Newspaper}, ${Month} ${Day}, ${Year}, Page ${Page} ${Obituary}" which also includes punctuation formatting. So, for example, the result looks like this:

    Katherine M. Polley - Chelmsford Newsweekly, December 31, 1940, Page 7

    Because this field has to be a link, I also had to define what it links to, which is what I'm doing in the "Link" field. The value for that field is being written as

    http://www.chelmsfordlibrary.org/reference/ask_us-obits.html?obit=${FirstName}+${MiddleInitial}+${LastName},+${Newspaper},+${Month}+${Day},+${Year},+Page+${Page}

    which carries the data over to the library's website and some javascript pulls the data from the url and puts it in an email form. The patron can fill in their name and contact info into the form and submit it to us as an email message

  • The "Sort" module is self-explanatory, and I chose to list them with most recent first

This feels far more complicated than it should be, and I'm sharing it here to both save someone else from having to figure it all out again on their own, and to hopefully get suggestions on how to simplify/improve it.

Although, speaking of improving it, I do have one idea for future development: the local Cemetery Department has spreadsheet online listing complete burial locations - it would be neat to mashup up that data, so the obituary is linked to the cemetery plot location.

That's down the road a bit, so in the meantime I just keep adding whatever new obituaries appear in the paper to the csv data file - I had planned to do that weekly, but lately there have been many weeks without any obituaries in the paper (see my previous post). Anyway, we'll see how this works - it only went live last week, but already patrons have been using it, and it certainly does save a lot of staff time.



Tags: , , , , , , , , , , , , , , , ,



The Death of Newspaper Obituaries

   September 2nd, 2010 Brian Herzog

Newspaper MONEY SectionOne function of public libraries is to be a repository for community history. The extent to which a library can do this will vary, but at the very least, the library has holdings of the local newspaper, which patrons can use to look up obituaries of local residents.

But the reality of this is changing. As newspapers struggle to stay alive, they're exploring new revenue streams - our local paper recently started charging families to list obituaries, instead of providing that service for free. The paper is only published once a week for a town of 32,000 residents, but you can still see the effect below:

Year #/Obits
2000 444
2001 527
2002 523
2003 566
2004 556
2005 479
2006 500
2007 220
2008 215
2009 80
2010 26 (as of Aug.)

And of the 80 obituaries in 2009, only 12 were from June-December. With dramatically fewer obituaries appearing in the paper, the long-term research value of a library's newspaper holdings is diminished. There must be other factors at play too, but hopefully newspaper revenues will stabilize and this downward obituary trend will be reversed. Regardless, there will always at least be a gap for anyone doing genealogical research or just looking up a friend of family member.

And this doesn't seem to be just a local thing. A Slashdot post describes the same thing on a bigger scale. There's also a Boing Boing post that looks into Legacy.com, the company many newspapers are using to outsource obituary listings. The bottom line in both posts is that obituaries and death notices are turning into a cash cow business - and as it becomes more and more expensive to run an obituary, there are going to be fewer and fewer of them.

So, all of that is sad news - doubly so since it's out of the control of libraries (unless we start publishing family-written obituaries on our own websites for free). But at my library, we have been working to improve access to what we do have. Tune in next week for Part Two of this post, detailing how we created an online index to the obituaries in our newspaper microfilm records, to make then easier for patrons and staff to locate.



Tags: , , , , , , , , ,



Free Online Historical Newspapers

   January 10th, 2008 Brian Herzog

Access NewspaperARCHIVE logoA few months ago I got an email about a website called Access NewspaperARCHIVE, saying that libraries could signup for free access to historical newspapers, dating back to the 1700s.

Sweet. I'm always looking for good primary source resources, especially online ones (and especially-especially free ones), so I thought I'd check this out. The signup process was a bit odd, having to download and then fax in their signup form [pdf, 418 kb]. I didn't hear anything back from them for months, so one day I just tried their url again (from within the library) and it IP-authenticated me.

So, I took that as us being signed up, and I started playing. The database is neat, as all the newspapers in there are saved as PDF files (see the 7/29/1895 Sandusky Register). And some are older than I could find in our other available resources, so those are two great things in its favor. However, I did see some drawbacks:

  • In-Library use only. And right on the authenticated homepage (the one patrons would see by logging in at the library) is a link to "Sign up for a home account." Which isn't expensive, but it's not free. It's just a little bit underhanded to give libraries a free account and then use that as a vehicle to sell to our patrons. So, I bypass this page and go right to the Browse page
  • No keyword searching. You can only browse by location, date, or newspaper title. Which will be fine for the "what happened on my birthday" questions, or if you were just looking up anything old in your area, but eliminates searching for a topic. And, the browse tool and the results listing are kind of clunky
  • No Massachusetts Newspapers. Which is a pain, since I mainly serve Massachusetts patrons. So, I guess no local historical information for me
  • Front pages only? For the papers I viewed, it wasn't the entire paper but just the front page. That's a pain
  • Not high-quality scans. The newspapers are legible, both on screen and printed, but they are just a little bit too bitmapped. And they are images, rather than text-based, which means no copy/pasting

So, my overall verdict is this: it's an amazing resource for primary source newspapers, and it's free, so it's better than nothing. There are some drawbacks, but I am rarely completely satisfied anyway.

Something else I did like was they had a "Questions? Ask a Librarian" link. This is an email link to whatever email address you supplied on the signup form. Which is good, since my patrons using this will be able to write to me, instead of this company.

Anyway, this is available, so I'm going to give it a try. If anyone has experience with this company or database, please comment below and let us know what you think. Thanks.

access, database, databases, historic, historical, libraries, library, newspaper, newspaperarchive, newspapers, primary, public, source, sources



Tags: , , , , , , , , , , , , , ,