Chapter 7. Supporting Tools

The proximity operator @prox is a special and more restrictive version of the conjunction operator @and. Its semantics are described in section 3.7.2 (Proximity) of Z39.50 the standard itself, which can be read on-line at http://www.loc.gov/z3950/agency/markup/09.html#3.7.2

In PQF, the proximity operation is represented by a sequence of the form

       @prox exclusion distance ordered relation which-code unit-code
      

in which the meanings of the parameters are as described in the standard, and they can take the following values:

(The numeric values of the relation and well-known unit-code parameters are taken straight from the ASN.1 of the proximity structure in the standard.)

The CCL parser obeys the following grammar for the FIND argument. The syntax is annotated using lines prefixed by --.

      CCL-Find ::= CCL-Find Op Elements
                | Elements.

      Op ::= "and" | "or" | "not"
      -- The above means that Elements are separated by boolean operators.

      Elements ::= '(' CCL-Find ')'
                | Set
                | Terms
                | Qualifiers Relation Terms
                | Qualifiers Relation '(' CCL-Find ')'
                | Qualifiers '=' string '-' string
      -- Elements is either a recursive definition, a result set reference, a
      -- list of terms, qualifiers followed by terms, qualifiers followed
      -- by a recursive definition or qualifiers in a range (lower - upper).

      Set ::= 'set' = string
      -- Reference to a result set

      Terms ::= Terms Prox Term
             | Term
      -- Proximity of terms.

      Term ::= Term string
            | string
      -- This basically means that a term may include a blank

      Qualifiers ::= Qualifiers ',' string
                  | string
      -- Qualifiers is a list of strings separated by comma

      Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
      -- Relational operators. This really doesn't follow the ISO8777
      -- standard.

      Prox ::= '%' | '!'
      -- Proximity operator

     

       dylan

       "bob dylan"

       dylan or zimmerman

       set=1

       (dylan and bob) or set=1

       righttrunc?

       "notrunc?"

       singlechar#mask
      

       ti=self portrait

       au=(bob dylan and slow train coming)

       date>1980 and (ti=((self portrait)))
      

Qualifiers are used to direct the search to a particular searchable index, such as title (ti) and author indexes (au). The CCL standard itself doesn't specify a particular set of qualifiers, but it does suggest a few short-hand notations. You can customize the CCL parser to support a particular set of qualifiers to reflect the current target profile. Traditionally, a qualifier would map to a particular use-attribute within the BIB-1 attribute set. It is also possible to set other attributes, such as the structure attribute.

A CCL profile is a set of predefined CCL qualifiers that may be read from a file or set in the CCL API. The YAZ client reads its CCL qualifiers from a file named default.bib. There are four types of lines in a CCL profile: qualifier specification, qualifier alias, comments and directives.

	ti       u=4 s=1
	au       u=1 s=1
	term     s=105
	ranked   r=102
	date     u=30 r=o
       

	 ti,ranked=knuth computer
	

	 date > 1980
	

	 ti > 1980
	

All public definitions can be found in the header file ccl.h. A profile identifier is of type CCL_bibset. A profile must be created with the call to the function ccl_qual_mk which returns a profile handle of type CCL_bibset.

To read a file containing qualifier definitions the function ccl_qual_file may be convenient. This function takes an already opened FILE handle pointer as argument along with a CCL_bibset handle.

To parse a simple string with a FIND query use the function

struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
                                  int *error, int *pos);
     

which takes the CCL profile (bibset) and query (str) as input. Upon successful completion the RPN tree is returned. If an error occurs, such as a syntax error, the integer pointed to by error holds the error code and pos holds the offset inside query string in which the parsing failed.

An English representation of the error may be obtained by calling the ccl_err_msg function. The error codes are listed in ccl.h.

To convert the CCL RPN tree (type struct ccl_rpn_node *) to the Z_RPNQuery of YAZ the function ccl_rpn_query must be used. This function which is part of YAZ is implemented in yaz-ccl.c. After calling this function the CCL RPN tree is probably no longer needed. The ccl_rpn_delete destroys the CCL RPN tree.

A CCL profile may be destroyed by calling the ccl_qual_rm function.

The token names for the CCL operators may be changed by setting the globals (all type char *) ccl_token_and, ccl_token_or, ccl_token_not and ccl_token_set. An operator may have aliases, i.e. there may be more than one name for the operator. To do this, separate each alias with a space character.

A CQL parser is represented by the CQL_parser handle. Its contents should be considered YAZ internal (private).

#include <yaz/cql.h>

typedef struct cql_parser *CQL_parser;

CQL_parser cql_parser_create(void);
void cql_parser_destroy(CQL_parser cp);
      

A parser is created by cql_parser_create and is destroyed by cql_parser_destroy.

To parse a CQL query string, the following function is provided:

int cql_parser_string(CQL_parser cp, const char *str);
      

A CQL query is parsed by the cql_parser_string which takes a query str. If the query was valid (no syntax errors), then zero is returned; otherwise -1 is returned to indicate a syntax error.

int cql_parser_stream(CQL_parser cp,
                      int (*getbyte)(void *client_data),
                      void (*ungetbyte)(int b, void *client_data),
                      void *client_data);

int cql_parser_stdio(CQL_parser cp, FILE *f);
      

The functions cql_parser_stream and cql_parser_stdio parse a CQL query - just like cql_parser_string. The only difference is that the CQL query can be fed to the parser in different ways. The cql_parser_stream uses a generic byte stream as input. The cql_parser_stdio uses a FILE handle which is opened for reading.

If the query string is valid, the CQL parser generates a tree representing the structure of the CQL query.

struct cql_node *cql_parser_result(CQL_parser cp);
      

cql_parser_result returns a pointer to the root node of the resulting tree.

Each node in a CQL tree is represented by a struct cql_node. It is defined as follows:

#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
#define CQL_NODE_SORT 3
struct cql_node {
    int which;
    union {
        struct {
            char *index;
	    char *index_uri;
            char *term;
            char *relation;
	    char *relation_uri;
            struct cql_node *modifiers;
        } st;
        struct {
            char *value;
            struct cql_node *left;
            struct cql_node *right;
            struct cql_node *modifiers;
        } boolean;
        struct {
            char *index;
            struct cql_node *next;
            struct cql_node *modifiers;
            struct cql_node *search;
        } sort;
    } u;
};
      

There are three node types: search term (ST), boolean (BOOL) and sortby (SORT). A modifier is treated as a search term too.

The search term node has five members:

The boolean node represents and, or, not + proximity.

The sort node represents both the SORTBY clause.

Conversion to PQF (and Z39.50 RPN) is tricky by the fact that the resulting RPN depends on the Z39.50 target capabilities (combinations of supported attributes). In addition, the CQL and SRU operates on index prefixes (URI or strings), whereas the RPN uses Object Identifiers for attribute sets.

The CQL library of YAZ defines a cql_transform_t type. It represents a particular mapping between CQL and RPN. This handle is created and destroyed by the functions:

cql_transform_t cql_transform_open_FILE (FILE *f);
cql_transform_t cql_transform_open_fname(const char *fname);
void cql_transform_close(cql_transform_t ct);
     

The first two functions create a tranformation handle from either an already open FILE or from a filename respectively.

The handle is destroyed by cql_transform_close in which case no further reference of the handle is allowed.

When a cql_transform_t handle has been created you can convert to RPN.

int cql_transform_buf(cql_transform_t ct,
                      struct cql_node *cn, char *out, int max);
      

This function converts the CQL tree cn using handle ct. For the resulting PQF, you supply a buffer out which must be able to hold at at least max characters.

If conversion failed, cql_transform_buf returns a non-zero SRU error code; otherwise zero is returned (conversion successful). The meanings of the numeric error codes are listed in the SRU specification somewhere (no direct link anymore).

If conversion fails, more information can be obtained by calling

int cql_transform_error(cql_transform_t ct, char **addinfop);
      

This function returns the most recently returned numeric error-code and sets the string-pointer at *addinfop to point to a string containing additional information about the error that occurred: for example, if the error code is 15 ("Illegal or unsupported context set"), the additional information is the name of the requested context set that was not recognised.

The SRU error-codes may be translated into brief human-readable error messages using

const char *cql_strerror(int code);
      

If you wish to be able to produce a PQF result in a different way, there are two alternatives.

void cql_transform_pr(cql_transform_t ct,
                      struct cql_node *cn,
                      void (*pr)(const char *buf, void *client_data),
                      void *client_data);

int cql_transform_FILE(cql_transform_t ct,
                       struct cql_node *cn, FILE *f);
      

The former function produces output to a user-defined output stream. The latter writes the result to an already open FILE.

The file supplied to functions cql_transform_open_FILE, cql_transform_open_fname follows a structure found in many Unix utilities. It consists of mapping specifications - one per line. Lines starting with # are ignored (comments).

Each line is of the form

An RPN pattern is a simple attribute list. Each attribute pair takes the form:

The attribute set is optional. The type is the attribute type, value the attribute value.

The character * (asterisk) has special meaning when used in the RPN pattern. Each occurrence of * is substituted with the CQL matching name (index, relation, qualifier etc). This facility can be used to copy a CQL name verbatim to the RPN result.

The following CQL patterns are recognized:

       set.cql  = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
       set.dc   = http://www.loc.gov/zing/cql/dc-indexes/v1.0/

       index.cql.serverChoice = 1=1016
       index.dc.title         = 1=4
       index.dc.subject       = 1=21

       relation.<             = 2=1
       relation.eq            = 2=3
       relation.scr           = 2=3

       position.any           = 3=3 6=1

       structure.*            = 4=1

      

        computer
       

        @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
       

        computer^
       

        >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
       

        @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
       

       # Identifiers for prefixes used in this file. (index.*)
       set.cql  = info:srw/cql-context-set/1/cql-v1.1
       set.rpn  = http://bogus/rpn
       set      = http://bogus/rpn

       # The default index when none is specified by the query
       index.cql.serverChoice     = 1=any

       index.rpn.*                = 1=*
       relation.eq                = 2=3
       structure.*                = 4=1
       position.any               = 3=3

      

        title = a
       

        @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
       

Conversion from CQL to XCQL is trivial and does not require a mapping to be defined. There are three functions to choose from depending on the way you wish to store the resulting output (XML buffer containing XCQL).

int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
void cql_to_xml(struct cql_node *cn,
                void (*pr)(const char *buf, void *client_data),
                void *client_data);
void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
      

Function cql_to_xml_buf converts to XCQL and stores the result in a user-supplied buffer of a given max size.

cql_to_xml writes the result in a user-defined output stream. cql_to_xml_stdio writes to a a file.

Conversion from PQF to CQL is offered by the two functions shown below. The former uses a generic stream for result. The latter puts result in a WRBUF (string container).

#include <yaz/rpn2cql.h>

int cql_transform_rpn2cql_stream(cql_transform_t ct,
                                 void (*pr)(const char *buf, void *client_data),
                                 void *client_data,
                                 Z_RPNQuery *q);

int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
                                WRBUF w,
                                Z_RPNQuery *q);
      

The configuration is the same as used in CQL to PQF conversions.