Anyway, the first version of the gem (which is slightly different from the code first posted prior to C4L) can be found at the project page, here: http://rubyforge.org/projects/wcapi/

–TR

Anyway…as part of the process, I started working on some rudimentary components for ruby to make processing the OCLC provided data easier.  Essentially, creating a handful of convenience functions that would make it much easier to call and utilize that data the grid services provided – my initial component was called wcapi. 

Tomorrow (or this morning if you are in Providence), a few folks from OCLC will be putting on a preconference related to the OCLC API at Code4Lib.  There, folks from OCLC will be giving some background and insight into the development of API, as well as looking at how the API works and some sample code snippets.  I’d hoped to be there (I’m signed up), but an airline snafu has me stranded in transit until tomorrow – so by the time I get to Providence, all of, or most of the pre-conference will be over.  Too bad, I know.  Fortunately, there will be someone else from Oregon State in the audience, so I should be able to get some notes.

However, there’s no reason why I can’t participate, even while I”m jetting towards Providence – and so I submit my wcapi gem/code to the preconference folks to use however they see fit.  It doesn’t have a rakefile or docs, but once you install the gem, there is a test.rb file in the wcapi folder that demonstrates how to use all the functions.  The wrapper functions accept the same values as the API – and like the API, you need to have your OCLC developer’s key to make it work.  The wcapi gem will utilize libxml for most xml processing (if it’s on your machine, otherwise it will downgrade to rexml), save for a handful of functions that done use named namespaces within the resultset (this confuses libxml for some reason – though their is a way around it, I just didn’t have the time to add it). 

So, the gem/code can be found at: wcapi.tar.gz.  Since I’ll likely make use of this in the future, I’ll likely move this to source forge in the near future.

*******Example OpenSearch Call*******

require ‘rubygems’
require ‘wcapi’

client = WCAPI::Client.new :wskey => ‘[insert_your_own_key_here'

response = client.OpenSearch(:q=>'civil war', :format=>'atom', :start => '1', :count => '25', :cformat => 'mla')

puts "Total Results: " + response.header["totalResults"]
response.records.each {|rec|
  puts "Title: " + rec[:title] + "\n"
  puts "URL: " + rec[:link] + "\n"
  puts "OCLC #: " + rec[:id] + "\n"
  puts "Description: " + rec[:summary] + "\n"
  puts "citation: " + rec[:citation] + "\n"

}

********End Sample Call****************

And that’s it.  Pretty straightforward.  So, I hope that everyone has a great pre-conference.  I wish that I was there (and with some luck, I may catch the 2nd half) – but if not, I’ll see everyone in Providence Monday evening.

–TR

1. Update to the Harvester (this will make it a bit more fault tolerant) as well as allowing harvesting of sets (currently) and root level oai providers (not provided currently).  This change required significant changes to the search component as well that deals with the harvested materials.
2. Auto-detection of namespaces (for oai and sru — needed for libxml)
3. Removed rexml dependencies for opensearch component
4. Frozen gems for oai, sru and opensearch into vendor directories (and have added that to the environment.rb file)
5. Prep code for solr/ferret decision.  I’ll be adding support to use either ferret or solr as your backend indexer for harvesting for 0.9, but some changes are being made to make this easier.  Ferret provides an integrated rails solutions, while solr would provide a hosted index option.
6. In addition to this, some changes to the libxml module have deprecated a call being used in the oai gem (maybe the sru gem).  I’ll take a look at both this week and update appropriately [as well as keep backworks compatibility if possible]

Something else, we are starting to work with mod_rails.  This allows apache to manage the rails environment — eliminating the need to run rails through packs of mongrels (or other specialized serving mechanism).  I’ll write up something on our experiences for others that might be interested in this approach.

–TR

Anyway, back to my topic.  Now, I’m going to preface this discussion by saying that this is obviously my own opinion and one that may not be shared by many people within the library community (I really have no idea).  Even within the library open source community, where I’m sure this opinion would be more prevalent (or at least entertained), I’m pretty sure I’m still in the minority.  But as I say, I think that these conversations are important to consider — specifically as we move down a path where OCLC is very quickly positioning themselves to become the library community’s default service provider for all things library (in terms of ILL, ILS interface, cataloging, etc.).

So when I talk about breaking up OCLC, exactly what am I’m talking about?  Well, in order to follow me down the path that I am going to take you, we have to talk about OCLC as I currently see them.  Watching OCLC during the 10 years (I can’t believe it’s actually been 10 years) that I have been in libraries, I have seen a quickening evolution of OCLC from strictly a member driven organization to more of a hybrid organization.  On the one hand, there is what many would consider the membership side of OCLC, that being WorldCat, ILL and their research and development office.  On the other hand, there is OCLC’s vendor arm…a good example of this would be WorldCat Local and WorldCat Navigator.  So how do I make these distinctions — membership services are those that I would consider core services.  These are services that OCLC has developed to add value to what OCLC likes to refer to as the Library Commons (WorldCat).  OCLC’s vendor services are those tools or programs that OCLC sells on top of the Library Commons, of which, I think WorldCat Local/Navigator is a good example.  Now I think that at this point, I know that folks at OCLC (and likely in the membership) would argue that both WorldCat Local/Navigator do provide services that the OCLC membership is currently requesting.  I won’t deny that — however, I would answer that the fact that OCLC treats the Library Commons (WorldCat) as it’s own closed personal community has the unintended affect of limiting the library community’s (and I include both commercial and non-commercial entities in my definition of community) ability to develop new service models.  In effect, we become much more dependent on how OCLC envisions the future of libraries.  Let me try and tease this out a little bit more…

Philosophically, the biggest problem that I have with the current situation is the commingling of OCLC’s treatment of the Commons (WorldCat) and their current strategy of being the sole commercial entity with the ability to interact with the Commons.  I’m a firm believer that the more diverse the landscape or ecology, the more likely that innovation will take place.  We’ve seen this time and time again both inside (Evergreen and Koha certainly have shaken up the traditional ILS market) and outside (web browsers are a good example of how competition breeds innovation) the library community.  However, by isolating the Commons, OCLC is threatening this diversity of thought.  Now, I have a whole set of different issues with the current library ILS community, but in this case, I think that OCLC’s treatment of the Commons, and their ability to leverage that service unfairly skews the ability for both commercial and non-commercial entities to provide innovative services on top of those Commons (and before anyone jumps on me for non-commercial use, let me finish my thoughts here).  Commercially, I’m fairly certain that the current crop of ILS vendors would very much like to provide their own WorldCat Local/Navigator interfaces to their customers, and I’m sure, would be able to tie these interfaces closely with services already provided by the users ILS.  I could envision things like ERM (electronic resource management), simplified requesting, etc. all being possible if the likes of ExLibris or Innovative Interfaces were allowed to build tools upon the Library Commons (WorldCat).  Maybe I would like to develop my own version of WorldCat Local/Navigator that interacts with the Commons and sell it as a product (kind of the same way ezproxy was sold prior to being acquired by OCLC) or a group of researchers would like to do the same.  As a commercial entity, I’m fairly certain that this type of development model wouldn’t be kosher with OCLC unless I licensed access to WorldCat (and I’m not certain that they would given that this would compete against one of their services).  Likewise, open source folks like LibLime or Equinox may like to create an open source version of the WorldCat Local interface.  Under the current guidelines, I understand that an open source implementation of WorldCat Local can exist — but as I understand that agreement, I’m not certain that groups like LibLime or Equinox (or another entity) could not take that project and then sell support-based services around it (I’m unclear on that one though).  However, it’s very unlikely that the library world will see any of these types of developments (well, maybe the open source WorldCat Local since I have a group that could use this and a number of people interested in developing it) because OCLC has come to treat what it calls the Commons (WorldCat), as it’s own personal data store.  There’s that commingling again. 

So if it was up to me, how would I resolve this situation?  Well, I see two possible scenarios. 

1. Open up WorldCat.  OCLC likes to refer to WorldCat as the Library Commons — well, let’s treat it as such.  Remove the barriers for access and allow anyone and everyone the ability to essentially have their own copy of the Library Commons and it’s data.  Now, rather than specifying terms of transfer and telling libraries under what conditions they can and cannot make their metadata available to other groups, the membership could consider what type of Open Data license that the Commons could be made available under.  Something like the creative commons share alike license which allows for both commercial and non-commercial usage, but requires all parties to contribute all changes to the data back to the community (in essence, this is kind of what Open Library is doing with their metadata) may be appropriate.  OCLC would be free to develop their own products, but the rest of the library community (both library and vendor community) would have equal opportunity to develop new services and ways of visualizing the data found in the Commons.  Does this devalue the Commons (WorldCat)?  I don’t think so — look at Wikipedia.  It uses this model of distribution, yet I’ve never heard anyone say that this devalues it’s content.  Would there be challenges?  For sure.  Probably one of the biggest would be the way that it would change what it means to be a member of OCLC.  If each person could download their own personal copy of the Commons, would libraries stay members.  I’m certain that they would — but I’m sure that what it means to be a member would certainly change.
2. Split OCLC’s membership services from OCLC’s vendor services.  Under this example, WorldCat Local/Navigator development would be spun away from OCLC as a separate business (this happens in academia all the time).  Were this to happen, OCLC would be able to develop terms for license that could then be leverage by all members of the commercial library community removing the artificial advantage OCLC is currently able to leverage (both in terms of data and deciding who is allowed to work with the Commons).  In all likelihood, I think that this model likely represents the smallest change for the membership and would continue to allow OCLC to make the Commons more available to non-commercial development without artificially limiting other groups interested in building new services. 

One last thought.  In talking to people today, I heard a number of times that OCLC restricting access to the Commons was in fact good thing, in part, because it finally allowed the library community the ability to leverage resources not available to the vendor communities.  In some way, we could finally stick it to them.  That’s fine, I’m all for developing tools and services, but this particular type of thinking I find worrisome.  If we, as a community, feel that we are unable to develop compelling tools and services that are able to compete with other vendor offerings without an artificial advantage — well that’s just sad and says a little something about how we see ourselves as a community.  And this too is something that I’d like to see change because if you look around, you will see that there are a myriad of projects (Koha, Evergreen, VuFind, Fedora, DSpace, LibraryFind, XC Catalog, Zotero, etc.) where developers (some library developers, some not) are re-envisioning how they see many of the services within the library and putting their time and effort into realizing those visions. 

–TR

I think that part my objection to this phrase comes from my development background.  As a developer, I’m a firm believer in a very iterative approach to development.  Those that use MarcEdit can probably attest (sometimes to their dismay) that updates can come with varying frequency as the program adapts to changes within the metadata community.  Since MarcEdit doesn’t follow a point release cycle in the traditional sense, could it be a beta application?  I guess that might depend on your definition of beta — as it certainly would seem to meet the criteria that Google applies to many of it’s services.  However, there is a difference I think.  While I take an iterative approach to development, I also want to convey to users that I will support this resource into the future and as beta, that’s not the case.  My personal opinion is that services that languish in beta are doing two things:

1. telling users that these services are potentially not long-term, so they could be gone tomorrow
2. giving users a weak apology for the problems that might exist in the software (and deflecting blame for those issues, because hey, it’s beta).

So, I don’t see this as the road to nimble development.  Instead, when I heard David’s talk, I heard something else.  I believe libraries fail to innovate because as a group we are insecure that our users will leave us if we fail.  And when I hear talks like David’s, that’s what I’m hearing his organization say.  They are asking the library administrators for permission to fail, because what is technological research but a string of failures in search for a solution.  We learn through our failures, but I think that as a community, our fear of failing before our users can paralysis us.  The fear of failing before an administration that does not support this type of research and discover can paralysis innovation by smart, energetic people within an organization.  A lot of people, I think, look at Google, their labs, their beta services and say, yes, that is a model that we should emulate.  But they don’t fully understand I think that within this model, Google builds failure as an acceptable outcome into their development plan.  If libraries want to emulate anything from the Googles or the Microsofts of the world, they should emulate that and engender that type of discovery and research within their libraries. 

I am actually very fortunate at Oregon State University in that I have a director and an administration that understands that for our library to meet the needs of our users now and in the future, we cannot be standing still.  And while the path we might take might not always be the correct one, the point is that we are always moving and always learning and refining our understanding of what our users want today and tomorrow.  What I’d like to see for David’s library is that kind of environment — and I wish him luck in it.

–TR

1. oai: http://rubyforge.org/projects/oai/
   Changes:
   Prior versions of the oai component only supported the ruby libxml gem 0.3.8-.  In order to support the newer versions of the ruby-libxml (which does better memory management), I’ve updated the oai gem that will support both the older and new ruby-libxml apis.  The primary changes are related to how it handles properties/attributes and the way that the oai provider class handles dates (to support OAI 2.0 validation).
2. sru-ruby: http://rubyforge.org/projects/sru/
   Changes:
   Added libxml support.  For queries that return one or two records, the different between using REXML or Libxml as your parser will be hardly noticable.  However, when queries result in larger datasets, the difference will be measured in terms of seconds.  For example, in some tests, recordsets returning 50 records in MARCXML format could run close to 10 seconds using REXML.  On the other hand, the same search using libxml processes in 0.8 seconds.  For tools, like LibraryFind, which deal with larger datasets, this is a welcome change.

–TR

–TR

To that end, I’m writing a plug-in for MarcEdit that may enable me to mainstream the processing of web page archiving in Dspace.  At this point, the process is a bit too manual for my tastes.  Along with spidering a site (using whatever the chosen depth may be), there is this pesky manual step of flattening the site and making the urls relative.  Not a big deal (unless there are file name collisions [which there always are] when reading depths), but it takes time.  So, I spent some time this afternoon and wrote a threaded web crawler.  Seems to work well.  At this point, I just need to add the logic to flatten all paths, and come up with a naming schema to re-write all urls to provide unique file names.  Once I get that down, building the batch import package for Dspace should be fairly trivial.  Not sure how much time I’ll have to work on this over the week/weekend, but would be a pretty cool project to finish I think.  It would certainly allow the library to provide site archiving as a dspace option (at this point, its only done under very special circumstances) and should simplify the process enough to the point that it could probably become a mainstream process. 

Anyway, if I do get a chance to get this finished, I’ll certainly make it available as a plug-in (with source).  Of course, if someone has already developed a simplified process that requires no manual processing after harvest, I would love to hear it.

–TR

Changes:

MarcEdit:

• One real change.  In the MacroInterfaces.dll (the library that allows the Scripting interface and the Plug-in interface access to the MarcEditor and its functions) I’ve added two new functions: AddButton and RemoveButton.  These two functions allow users to have plug-ins place buttons on the toolbar of the application (at least, on specific windows). 

OCLC Plug-in:

• I’ve added the code to initiate a button and interact when it is clicked to the MarcEditor toolbar.
• I’ve corrected the 007 error.  See: http://oregonstate.edu/~reeset/blog/archives/480

So what will you see with the changes.  Well, the big change you will see is when you initialize the plug-in in the MarcEditor.  Once you have downloaded the update to both MarcEdit and the Plug-in (you need both), you will see the following when the plug-in is executed:

The new button added has been highlighted.  This button now acts as the new Save button when you have made your changes to the OCLC data records.  This will move the data back into the OCLC Save File.  Remember — at this point, you will want to make a backup of your Save Files before you make your changes — just in case there are other fields in OCLC’s XML format that are different than I would have expected. 

If you have downloaded the OCLC Plug-in and would like to update it.  At this point, the process isn’t as streamlined as I’d like (I’ll fix that this week while I’m taking some time off to recharge).  Essentially, you need to Uninstall the plug-in (using the plug-in manager or, delete the oclc_helper.dll from the marcedit plug-ins directory (generally, c:\program files\marcedit 5.0\plugins\).  If you uninstall with the plug-in manager, you will need to close and restart MarcEdit.  Then open the plug-in manager and download the new plug-in.  If you delete the library directly from the plug-ins directory — just open MarcEdit, select the plug-in manager and download the plug-in.

I’ve updated a new version of MarcEdit: MarcEdit51_Setup.exe.  I’ve also uploaded a new version of the plug-in (download this through plug-in manager [see above]).  Source can be downloaded from: oclc_helper.zip.

If you have a strong desire to see how this type of interaction is accomplished in C#, please see the following post and sample project file: http://oregonstate.edu/~reeset/blog/archives/481.

–TR

Because I’ve been doing a lot of work with MarcEdit and plug-ins, I thought I’d post some sample code for anyone interested in how this might work.  Essentially, the sample project includes 3 parts — a host application, a set of Interfaces and a Shared library.  Making this work requires a couple of important parts. 

First, the host application (either the form or class), need to implement the set of interfaces.  So for example, if interaction with a form in the hosted application was need, you would configure the form to implement a set of interfaces.  This would look like:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;

namespace HostApp
{
public partial class Form1 : Form, HostInterfaces.IHost
{
//...
}

This implements the IHost class (link to msdn) — a generic class that allows you to

pass objects between dynamically loaded libraries.  .NET includes a IScript interface that allows for scripting functionality as well. 

Anyway, the interfaces are simply like delegates — they define the visible functions/methods that will be accessible to a foreign assembly.  This is the simpliest file to create.  It looks something like this:

using System;
using System.Collections.Generic;
using System.Text;

}
}
Finally, the Dynamic assembly has the ability to work with any function/object within the host application that has been made public through the interface.  For this sample project, I’ve shown how to modify a label (on the host application), add a button to a toolbar and respond to click events from that button. 

The project is a simple one — but should go a long way towards showing how this works.

Cheers,

–TR

Technorati Tags: C#,plug-ins,interfaces