StructWSF Python Wrapper Tutorial

From TechWiki

This is a tutorial for using a structWSF Python wrapper called BKN_WSF which is just a single file, bkn_wsf.py. The file can be imported into a Python script or used by Javascript or PHP apps as a web service. Use as a web service is not documented (see source code for documentation). BKN_WSF was created by the Bibliographic Knowledge Network (BKN) which is a loose coalition of partners initially funded by the NSF Cyber-enabled Discovery and Innovation (CDI) Program (award 0835851). The mission is to build free tools and applications for researchers to distill, normalize, reconcile, publish, and analyze large open bibliographic datasets, and to develop a network of websites and webservices participating in this activity.

This document uses a few interchangeable terms,

OVERVIEW

bkn_wsf.py includes wrappers and examples for using structWSF, a framework for interacting with an RDF repository. Review of structwsf documentation is recommended to better understand bkn_wsf. Not all structwsf functionality is wrapped by bkn_wsf but any structwsf functionality can be called from bkn_wsf.py. For more information about structWSF usage. See, openstructs.org/structwsf.

structwsf supports a variety of data formats (RDF, XML, JSON) and converts the data to and from RDF. bkn_wsf is tailored to use JSON data with structwsf, and in particular a format structwsf calls irJSON. This irJSON format is what you typically think of when you work with JSON data. Some structwsf responses are in a JSON structure that would be more familiar to someone working with RDF. BKN projects usually refer to the irJSON format as bibJSON.

bkn_wsf.py simplifies use of structwsf and handles conversions required to use structwsf with irJSON (bibJSON). BKN projects use bkn_wsf to import large datasets into structwsf instances/repositories. bkn_wsf can also be used as a proxy web service by javascript apps (See method BKNWSF.web_proxy_services). Every structwsf instance used by BKN has a corresponding Drupal site with a module called conStruct that integrates structwsf with Drupal. The Drupal site is used for various management tasks, and can be used to search and browse data. http://people.bibkn.org is the first structwsf instance created for BKN. There are now over one million records in the repository. BKN does not make use of most Drupal/conStruct functionality which includes the ability to skin display of different types of data.

INSTALLATION

bkn_wsf.py is a single python file. The source control repository is currently at,

github.com/bkn/bkn_wsf

From there you can browse the source and download code by clicking the Download Source button at the top right which pops up the download url,

github.com/bkn/bkn_wsf/tarball/master

There may be other files but bkn_wsf.py is the only relevant file. test.py and in.json may be useful for testing.

The primary location for the code may change. All BKN projects have a link to source code from a google code project called bibkn, code.google.com/p/bibkn/w/list. A url to the primary version of bkn_wsf should always be on the corresponding wiki page, code.google.com/p/bibkn/wiki/bkn_wsf

bkn_wsf.py is not useful without a structwsf repository, and write access requires you have a user account with the structwsf repository. Ask your system administrator to create an account.

Examples in this document refer to a BKN repository at datasets.bibsoup.org. This is a real repository but it is used in this document only as an example. This document assumes you have access to your own repository. However, if you are working with bibliographic data you can request an account by emailing info@bibkn.org.

Permissions
To use a dataset created by someone else they need to give you permission. On the Drupal site, depending on your permissions role you may be able to 'join' the dataset. Go to "Datasets" page, and click on the "join" link which should go to a url like, datasets.bibsoup.org/og/. In any case, contact the dataset creator to set your permissions. See DATASET CREATE section for instructions on how to set permissions for a dataset you create.

To confirm setup you can run the autotest method with the specified repository root url. For example,

The test performs reads and writes. The test prints out lines for each test. The final step displays the content of a dataset that gets removed after the test. The last output should be a pass/fail statement. The output should show a dataset named 'dataset_test', with a record that has an '1' for the 'id' value and 'update' for the 'name' value. If the test fails after creating the dataset the test should clean up by removing the dataset.

The method Test.wsf_test() is a hodgepodge of commented out example calls. This is sometimes useful for testing specific operations, or cleaning up test datasets. The file test.py calls this method.

USAGE OVERVIEW

bkn_wsf.py is intended to make it easy to use bibjson/irjson data. There is a performance penalty for the easy of use. Many structwsf services do not return irjson data. So bkn_wsf requests 'text/xml' responses which are then sent to the structwsf service '/converter/irjson/'.

bkn_wsf wraps structwsf services but any service with any parameters can be called from bkn_wsf using the BKNWSF.structwsf_request method which is how all bkn_wsf methods interact with structwsf. For instance, a search sets parameters and executes,

structwsf authentication is handled by associating an ip address with auser account on the drupal site associated with the structwsf repository. structwsf operations that require permissions take a parameter with an ip address of the user. The BKNWSF.structwsf_request method automatically detects the user's ip address and adds the 'registered_ip=' parameter to the structwsf service call. (The ip address is detected by calling bkn_wsf.py as a web service This is necessary for cases like client-side Javascript apps because structwsf needs the externally visible ip address which may not be what the browser detects on the client side.)

Most BKN datasets have public read-only permissions. Login is typically required for write access to a dataset, and to read private datasets.

Classes

There are three primary classes: BKNWSF, Dataset, and Record.

BKNWSF - manages the structwsf instance root uri, and includes most methods for operations which are not specific to a dataset (search, browse). Among the methods is structwsf_request which is the method that handles all calls to structwsf services.

Dataset - manages the dataset uri and it's related parts (root, id), and includes methods for operations on a single dataset like create, read, update, access, list, ...

Record - manages the record uri and id, and includes methods for operations on a record like create, update, read ...

Source code documentation is available in HTML format at services.bibsoup.org/doc/bkn_wsf/

Error Handling

Most bkn_wsf.py errors are not caught. You will see the python error on stdout. Some logging functionality is implemented. More will be done in the near-future. Errors from structwsf are propagated and returned as a response to calls, and the BKNWSF.structwsf_request method catches errors. Two common errors are,

403 FORBIDDEN - Yhe user doesn't have permission to perform the operation. When a dataset is created using bkn_wsf.py, default permissions are set for creator read/write. structwsf has very rich access control settings. bkn_wsf.py just sets defaults, and currently doesn't provide any special methods to facilitate adding users to a dataset. Permissions management is currently performed using the Drupal interface. More access control functionality is on the short-list for bkn_wsf.py.

400 BAD REQUEST - This is kind of a catch all. It could be because the structwsf call is not properly formatted or a parameter value is incorrect. For instance, reading a record id that does not exist will return this error, or not setting the structwsf root.

SETUP

This tutorial is a walk through based on the Test.autotest method in bkn_wsf.py. Before attempting any operation, the structwsf instance (repository) root must be set. bkn_wsf uses the root to construct roots for structwsf service calls and for datasets. For example,

structwsf calls typically require a dataset uri and/or a record uri. bkn_wsf will use the current Dataset root to construct a uri given an id and vice versa, and handle a uri that is preceded with one or more '@' symbols (which are often the format returned by structwsf responses). In each of the cases below, the Dataset uri and id are set and the method returns the dataset uri.

Dataset.get has a few uses. By default Dataset.get() returns the uri. A key can be passed to return parts of the uri,  'id' and 'root'. There are two other special keys. The 'record_data' key is described in the "dataset record list" section below. The 'template' key is used to get a bibjson/irjson formated template. The template is used with Record operations.

A dataset template looks like,

SEARCH AND BROWSE

The search and browse methods are in the BKNWSF class because they can be called to return data from multiple datasets. So there is not necessarily a specific active dataset when a browse or search call returns. (An alternative location for these methods could be the Dataset class which is now intended for operations on a single dataset.) While search/browse can take a list of datasets, they will use the active dataset if no dataset uri is specified. To specific all datasets, use 'all' as the uri parameter. There is functionality in structwsf to filter by type and attribute. Filter functionality is not yet wrapped by bkn_wsf but you can pass additional structwsf parameters to most bkn_wsf methods.

Search and Browse take parameters for page size and page number. To page through results set whatever page size you want and increment the page number starting at 0.

Below are some example calls and distinct sections of an example responses.

The record data is in the 'recordList' object.

The response also returns metadata for each type or property. This metadata can be used for faceted displays. The 'aggregate' object contains the metadata.

The dataset section shows mapping for types and attributes.

DATASET CREATE

To create a dataset execute,

If the create succeeds the response is empty. The structwsf /dataset/create/ call gives the creator full permissions by default, and creates an access permission record for public access (0.0.0.0) but gives no public permission. The bkn_wsf Dataset.create method adds full permissions for the drupal site associated with the repository (this is currently hard-coded for BKN servers).

DATASET PERMISSIONS

structwsf implements permissions for Create Read Update Delete (CRUD) on any service. bkn_wsf sets permissions on a fixed set of services including:

And may explicitly add the following in the future,

       
To set access for a dataset use,

The 'default_access' option adds full permission for the drupal server associated with the repository. Both of the  above calls use the lower level method,

A wrapper to 'update' permissions is not yet implemented. For a Drupal repository like people.bibkn.org, permissions can also be set with a web interface. When a new dataset is imported you must first make it visible to conStruct, the module that manages structWSF transactions on a Drupal system. You need to "link" the dataset to the conStruct instance from the page,

On that page,