The first two weeks involved:

• generating some files for translation of the Greenstone interface (Mongolian, Bhutanese) and committing changes translators had submitted (Laotian)
• fixing up the GS2 CORBA code, including bringing it up to speed with the rest of GS2’s runtime code, so that CORBA works again: it can now compile once more, and the corbaserver and corbarecptldd client program run well against each other when on the same machine. Running the server against the client in a remote situation does not yet work, but it did not work in the demo/hello-1 example of the now-updated MICO package either.
• there was still a small error in the way the PDFBox extension tests for Java when Java is version 1.7 that made the extension not work with JDK1.7. The test for the presence of Java now has to run java -version rather than just java, since the return value in Java 7 is different from that in Java 6.
• when testing the Powerpoint plugin, it was found that the OpenOffice extension needed to be corrected to make jodconverter use the same port as that which OpenOffice is run on. It was moreover discovered that users can’t already have the graphical user interface of OO running in the background, nor can they start this, during Greenstone’s processing of documents using the OO extension.

This week:

• there was some issue with Greenstone 3’s tomcat server crashing on 64 bit Linux owing to a Java segmentation fault created by an error in the JNI code. Dr Bainbridge found out that the number of bytes to store pointers to data structures shared between Java and C++ code needed to be long rather than int, so MG’s and MGPP’s JNI code was updated. The error has not returned since, but debugging code has been left in for future debugging if required.
• Dr Te Taka hoped to update the Maori translations for Greenstone’s interface using Google’s Translator Toolkit (GTT), and suggested that Greenstone’s translation process be expanded to allow this so that other translators too could benefit from the toolkit for translation if they wanted. He found out that the toolkit accepted an open-XML format called TMX, Translation Memory eXchange, and thus would need the strings that required translation to be converted into the TMX XML format (rather than into the usual spreadsheets versions of the .excel.xml format which we currently generate). Two new XSLT files have been written which Te Taka may kindly be testing for us: the first generates the TMX translation files that translators can load into Google’s Translator Toolkit. The second XSLT takes translated TMX files and converts them into an intermediary format that can be processed in the usual manner when submitting new and updated translations back into Greenstone.
• currently looking at usersDB in GS3 having the correct values on startup.

Update: did not get much further with the GS3 usersDB as there was a lot more to be done with the translation files for GTT and their processing. The process became clearer thanks to Te Taka’s explanations and his testing at each stage. TMX files will only be needed the first time a translator migrates from GS’s usual translation procedure, which makes use of excel spreadsheet files, to Google’s toolkit. The TMX file will start them up with all the up-to-date translated strings that are available so far in GS3 for the selected language. For the strings that need to be translated and updated, the translator will get a text file that contains the unicode spreadsheet data (as comma separated values, but the file will have a .txt extension instead of .csv in order to preserve the unicode). The translator will then copy the English and <Language> columns of the spreadsheet into the GTT. Once their translation work is done, they can send these same columns back by way of the same spreadsheet.

This week I have added the ability for users to register themselves for a Greenstone 3 site. To register a user must provide their username, password and email address (we may add more fields or the ability for an administrator to add custom fields) as well as match two words from an image (to make sure they’re not a bot). Users can now also modify their account settings themselves and next week we will probably look into adding the ability for Greenstone 3 to email the users (to confirm registration or to reset their passwords).

The paged-image widget I have discussed in the past also received several upgrades, such as the ability to filter pages based on their titles and it will now also show the page you are currently on within the widget. We are also planning on being able to specify number ranges to filter pages as well (e.g. typing “24-37” will show all the pages from page 24 to page 37).

I have made a list of all the things we need to complete before we can release Greenstone 3, it’s gradually getting smaller so hopefully it won’t be much longer.

With the new user-login capability of Greenstone3 I have been creating and improving various features that relate to this capability. For example, the previous administration capabilities (the ability for admin users to add/edit/remove users) were not very secure and have now had an overhaul to properly connect with the servlet security method I described in a previous post. By itself however, trying to use the current administration capabilities to manage a large-scale Greenstone 3 installation with many collections and many users would be a difficult task as each user would need to be added/modified by an administrator. To aid in this problem we will create the ability for users to register themselves and to change their own basic settings (password and details), the more powerful options such as group assignment will still be the job of an administrator.

We have also made significant progress on the RESTful URL feature. Servlets can have filters that requests are sent to before they reach the servlet itself, and this what we use to provide this functionality. The filter examines the URL before it reaches the servlet and digs out any parameters that have been written in the RESTful form (for example, it will set c=demo from http://127.0.0.1:8383/greenstone3/dev/collection/demo/page/home).

This week has had a rather exciting development that several people have been wanting for quite a long time.  The 64-bit compatible versions of MG, MGPP and GDBM have been added to the main code, meaning that Greenstone 2 and 3 can now compile successfully on 64-bit systems. The reason this has taken a long time to be done is that the 32-bit and 64-bit versions of MG and MGPP produced seemingly different files when run over the same documents, which was a concerning for us as people might want to move their 32-bit MG/MGPP collections over to a 64-bit Greenstone installation and we suspected that this might not work given the different files. This week we discovered the cause of the difference and are now reassured that files from 32-bit and 64-bit installations can be interchanged without issue.

This week has seen more upgrades to Greenstone 3 as well. One of the features we have been working on for the Pei Jones collection is the ability to zoom “screen” images by using the mouse like a magnifying glass. We have added this into the default Greenstone 3 capabilities. In order for this to work however there needs to be a “screen” (small) and “source” (usually larger) version of the same image.

In general Greenstone 3 now handles paged-images much better. They are now properly displayed at the top of their specific sections. There is also an option to change between text-only, image-only and the default text and image modes, which is available in both the paged style collections as well as normal hierarchy style collections.

Next week will most likely involve more improvements like this as we continue to prepare Greenstone 3 for release.

Our exploration into security for Greenstone 3 using the built-in security provided by the Java Servlet API has gone well. We now have the ability to allow users to restrict parts of their collections to users within certain groups by specifying constraints in the collectionConfig.xml file. We are still working on the exact format for the XML but here is an example of a set of constraints in the current format:

<security scope="documents" default_access="public">
  <documentSet name="firstSet">
    <match>HASHe08571b7f6e430e238e2dd</match>
    <match field="Title" type="regex">.* Garden</match>
  </documentSet>
  <documentSet name="secondSet">
    <match field="Title" type="regex">Egyptian .*</match>
  </documentSet>
  <exception>
    <documentSet name="firstSet"/>
    <group name="dl"/>
  </exception>
  <exception>
    <documentSet name="secondSet"/>
    <group name="administrator"/>
  </exception>
</security>


You’ll notice that in the <security> element there are two attributes. The default_access attribute can be either “public” or “private” and this specifies whether the normal (guest) user can access the collection/documents. The scope attribute can be either “collection” or “documents” and this specifies whether these rules affect the whole collection or a set of documents. An average collection will have a very simple security block like:

<security scope="collection" default_access="public"/>

which specifies that the whole collection is publicly accessible. As you can see with the first example, we also allow much more detailed control over what documents each group can access. What this example specifies is that the average user can access the majority of the documents with a few exceptions. In order to access the “firstSet” set of documents (which contains the document with the ID  HASHe08571b7f6e430e238e2dd and all documents whose titles end in “Garden”) you have to be in the “dl” group. In order to access the”secondSet” set of documents (which contains documents whose titles start with “Egyptian”) you have to be an administrator.

As well as working on security I have made various improvements to the document editor. Users can now edit documents directly on the document page, which allows users to very efficiently perform any basic (content and metadata) modifications. The previous document editor is still where more advanced edits (structure, document creation/deletion) can be performed.

Things have been fairly busy here the last few weeks so I’ve been a little slack on the blogging. We have been continuing to look into robust authentication for Greenstone 3 and as part of the we have been investigating the security features that the Java Servlet technology (that Greenstone 3 uses) has built in. We have also been devising a way to specify the security settings that you want – like in Greenstone 2 – but in a way that is more flexible. For example, we are looking into the idea of groups of users (e.g. admin, staff, students etc.) that can have access to different documents based on the groups they are in.

I’ll write more details on this next week.

This week I have been touching up on a few unfinished features. One of these was the mapping features that I can’t remember whether or not I have written about before. Basically, if your documents have coordinate information (i.e. latitude and longitude information) we now have a feature that will map those documents on a map. This feature can now be really easily enabled. We will write some documentation on this when we get the chance.

The new theme is also added which makes Greenstone 3 look a lot nicer. We’re still working on the ability to allow easy theme changing. We need to get the authentication working before we can enable this feature, as only the collection administrator should be able to change the theme.

Next week I will experimenting with trying to get more standard URLs in Greenstone 3 (e.g. http://localhost:8383/greenstone3/dev/collection/demo/document/HASHc5bce2d6d3e5b04e470ec8) rather than what we currently use.

P.S. If you’re wondering why there is no update from Anu this week, it is because she is away in India for 5 weeks on holiday.

One of the things we have been doing this week is deciding the best way to handle user authentication in Greenstone 3. We have a very basic system in place at the moment but we would like something more robust. At the moment we are investigating using the authentication system in the web-server we use for Greenstone 3 (Apache Tomcat). We need to make sure it has the flexibility we require so that collection administrators have the power to allow/prevent users access to the collection as well as (possibly) access to individual documents.

I have been continuing to assist the masters student I mentioned last week. We have been working on a way to download and replace parts of a collection via the web interface. We think that this functionality may be useful if you want to add/replace an image or run an image through your own OCR program for example.

Finally, I have been further adding to Greenstone’s CGI metadata capabilites, filling in any holes that are missing in the API. As part of this I have started developing a Javascript API which should (theoretically) make using these CGI calls a lot easier.

My time this week has mostly been spent helping out one the masters students here in our lab. I have been helping her develop the ability to tag photos and text in the Greenstone 3 collection she is working on. This has resulted in us enhancing our Greenstone 3 (and also Greenstone 2) CGI capabilities at the same time to get this working correctly. This upgrade was needed so that we could save metadata to the index, archive and import directories easily from Javascript. Some of the functionality was already there but functionality like the ability to remove metadata from the import directory (for example) was missing.

One problem we had to get around was the fact that you cannot reliably specify the position of a piece of metadata that you want to change/delete in a metadata.xml file because of the way import metadata is handled in Greenstone. We decided that a good way to get around this is to have to specify the previous value of the piece of metadata that you want to change/delete. The only problem with this approach is if you have more than one identical piece of metadata, do we delete just one? or all of them? Most likely we will add an option to specify what to do in this situation.

Next week I will most likely be working on some authentication functionality for Greenstone3.

Happy new year to all Greenstone users! We’re back at work now after a couple of weeks off over the holiday period and already we’ve got a few new things lined up.

In Greenstone we try very hard to make the modification of the look and feel of collections as easy as possible.  Unfortunately this often requires knowledge of web standards like HTML, CSS and Javascript, and in the case of Greenstone 3 it is also helpful to have knowledge of XML and XSLT. We understand that many Greenstone users will have very little knowledge of these topics, so we are looking at incorporating a very simple way of changing the appearance of a Greenstone collection.

JQuery UI has a system called ThemeRoller that allows you to create your own visual theme via an easy to use web interface. You can then download the required files to use that theme in your own website. We are currently experimenting with making Greenstone 3 compatible with these themes (which are made up of a CSS file and some images). So far it is looking promising and will hopefully prove to be a welcome addition to Greenstone 3.

It has been a short week this week so there’s not a lot to report, but next week I shall be continuing on this development as well as (most likely) starting to write some up-to-date documentation for Greenstone 3, as we have made it our goal this summer to spend a large part of it working on Greenstone’s documentation.