This week has had a rather exciting development that several people have been wanting for quite a long time.  The 64-bit compatible versions of MG, MGPP and GDBM have been added to the main code, meaning that Greenstone 2 and 3 can now compile successfully on 64-bit systems. The reason this has taken a long time to be done is that the 32-bit and 64-bit versions of MG and MGPP produced seemingly different files when run over the same documents, which was a concerning for us as people might want to move their 32-bit MG/MGPP collections over to a 64-bit Greenstone installation and we suspected that this might not work given the different files. This week we discovered the cause of the difference and are now reassured that files from 32-bit and 64-bit installations can be interchanged without issue.

This week has seen more upgrades to Greenstone 3 as well. One of the features we have been working on for the Pei Jones collection is the ability to zoom “screen” images by using the mouse like a magnifying glass. We have added this into the default Greenstone 3 capabilities. In order for this to work however there needs to be a “screen” (small) and “source” (usually larger) version of the same image.

In general Greenstone 3 now handles paged-images much better. They are now properly displayed at the top of their specific sections. There is also an option to change between text-only, image-only and the default text and image modes, which is available in both the paged style collections as well as normal hierarchy style collections.

Next week will most likely involve more improvements like this as we continue to prepare Greenstone 3 for release.

Our exploration into security for Greenstone 3 using the built-in security provided by the Java Servlet API has gone well. We now have the ability to allow users to restrict parts of their collections to users within certain groups by specifying constraints in the collectionConfig.xml file. We are still working on the exact format for the XML but here is an example of a set of constraints in the current format:

<security scope="documents" default_access="public">
  <documentSet name="firstSet">
    <match>HASHe08571b7f6e430e238e2dd</match>
    <match field="Title" type="regex">.* Garden</match>
  </documentSet>
  <documentSet name="secondSet">
    <match field="Title" type="regex">Egyptian .*</match>
  </documentSet>
  <exception>
    <documentSet name="firstSet"/>
    <group name="dl"/>
  </exception>
  <exception>
    <documentSet name="secondSet"/>
    <group name="administrator"/>
  </exception>
</security>


You’ll notice that in the <security> element there are two attributes. The default_access attribute can be either “public” or “private” and this specifies whether the normal (guest) user can access the collection/documents. The scope attribute can be either “collection” or “documents” and this specifies whether these rules affect the whole collection or a set of documents. An average collection will have a very simple security block like:

<security scope="collection" default_access="public"/>

which specifies that the whole collection is publicly accessible. As you can see with the first example, we also allow much more detailed control over what documents each group can access. What this example specifies is that the average user can access the majority of the documents with a few exceptions. In order to access the “firstSet” set of documents (which contains the document with the ID  HASHe08571b7f6e430e238e2dd and all documents whose titles end in “Garden”) you have to be in the “dl” group. In order to access the”secondSet” set of documents (which contains documents whose titles start with “Egyptian”) you have to be an administrator.

As well as working on security I have made various improvements to the document editor. Users can now edit documents directly on the document page, which allows users to very efficiently perform any basic (content and metadata) modifications. The previous document editor is still where more advanced edits (structure, document creation/deletion) can be performed.

Things have been fairly busy here the last few weeks so I’ve been a little slack on the blogging. We have been continuing to look into robust authentication for Greenstone 3 and as part of the we have been investigating the security features that the Java Servlet technology (that Greenstone 3 uses) has built in. We have also been devising a way to specify the security settings that you want – like in Greenstone 2 – but in a way that is more flexible. For example, we are looking into the idea of groups of users (e.g. admin, staff, students etc.) that can have access to different documents based on the groups they are in.

I’ll write more details on this next week.

This week I have been touching up on a few unfinished features. One of these was the mapping features that I can’t remember whether or not I have written about before. Basically, if your documents have coordinate information (i.e. latitude and longitude information) we now have a feature that will map those documents on a map. This feature can now be really easily enabled. We will write some documentation on this when we get the chance.

The new theme is also added which makes Greenstone 3 look a lot nicer. We’re still working on the ability to allow easy theme changing. We need to get the authentication working before we can enable this feature, as only the collection administrator should be able to change the theme.

Next week I will experimenting with trying to get more standard URLs in Greenstone 3 (e.g. http://localhost:8383/greenstone3/dev/collection/demo/document/HASHc5bce2d6d3e5b04e470ec8) rather than what we currently use.

P.S. If you’re wondering why there is no update from Anu this week, it is because she is away in India for 5 weeks on holiday.

One of the things we have been doing this week is deciding the best way to handle user authentication in Greenstone 3. We have a very basic system in place at the moment but we would like something more robust. At the moment we are investigating using the authentication system in the web-server we use for Greenstone 3 (Apache Tomcat). We need to make sure it has the flexibility we require so that collection administrators have the power to allow/prevent users access to the collection as well as (possibly) access to individual documents.

I have been continuing to assist the masters student I mentioned last week. We have been working on a way to download and replace parts of a collection via the web interface. We think that this functionality may be useful if you want to add/replace an image or run an image through your own OCR program for example.

Finally, I have been further adding to Greenstone’s CGI metadata capabilites, filling in any holes that are missing in the API. As part of this I have started developing a Javascript API which should (theoretically) make using these CGI calls a lot easier.

My time this week has mostly been spent helping out one the masters students here in our lab. I have been helping her develop the ability to tag photos and text in the Greenstone 3 collection she is working on. This has resulted in us enhancing our Greenstone 3 (and also Greenstone 2) CGI capabilities at the same time to get this working correctly. This upgrade was needed so that we could save metadata to the index, archive and import directories easily from Javascript. Some of the functionality was already there but functionality like the ability to remove metadata from the import directory (for example) was missing.

One problem we had to get around was the fact that you cannot reliably specify the position of a piece of metadata that you want to change/delete in a metadata.xml file because of the way import metadata is handled in Greenstone. We decided that a good way to get around this is to have to specify the previous value of the piece of metadata that you want to change/delete. The only problem with this approach is if you have more than one identical piece of metadata, do we delete just one? or all of them? Most likely we will add an option to specify what to do in this situation.

Next week I will most likely be working on some authentication functionality for Greenstone3.

Happy new year to all Greenstone users! We’re back at work now after a couple of weeks off over the holiday period and already we’ve got a few new things lined up.

In Greenstone we try very hard to make the modification of the look and feel of collections as easy as possible.  Unfortunately this often requires knowledge of web standards like HTML, CSS and Javascript, and in the case of Greenstone 3 it is also helpful to have knowledge of XML and XSLT. We understand that many Greenstone users will have very little knowledge of these topics, so we are looking at incorporating a very simple way of changing the appearance of a Greenstone collection.

JQuery UI has a system called ThemeRoller that allows you to create your own visual theme via an easy to use web interface. You can then download the required files to use that theme in your own website. We are currently experimenting with making Greenstone 3 compatible with these themes (which are made up of a CSS file and some images). So far it is looking promising and will hopefully prove to be a welcome addition to Greenstone 3.

It has been a short week this week so there’s not a lot to report, but next week I shall be continuing on this development as well as (most likely) starting to write some up-to-date documentation for Greenstone 3, as we have made it our goal this summer to spend a large part of it working on Greenstone’s documentation.

Continued on the problem that I thought had been almost resolved last week: getting the batch files in GS2 to handle not only spaces but also brackets in the Greenstone filepaths. The batch files were done, but the perl code needed some correcting too. After inspecting many files in order to see whether they needed correcting, the GS2 code seems to work well on Windows even where Greenstone is installed in a path containing brackets.

This week, I was able to finally return to the problem of jodconverter not interacting well with the LibreOffice on the Ubuntu 11 whereas the same worked perfectly against an OpenOffice on the CentOs machine. We decided that perhaps OpenOffice had different behaviour for the signals sent by jodconverter. Installing OpenOffice turned out harder than expected and I think I botched it. I ended up having to uninstall all openoffice files and libreoffice files and then reinstalled all of libreoffice. At this stage, upon trying jodconverter again, it was found to work fine each time. This seemed to confirm the suspicion that some updates to Ubuntu may have messed up some libraries or something, breaking LibreOffice a little.

However, despite things now working again, Sam wondered, very correctly, whether a user’s experience would be this convoluted or whether it would work straight away for them. He suggested trying out a VM of Ubuntu 11. Which is what I did. It was my first VM installation and after installing a Ubuntu 11.10 VM on Sam’s Windows 7 (which comes with LibreOffice), Greenstone with the open-office extension fortunately worked fine on a sequence of word documents.

On Friday, got round to Diego’s long-standing question at last: about the possibility of a single metadata.xml at the import level which defines the metadata for all files in import’s subfolders. Dr Bainbridge had already confirmed earlier that this was indeed possible, but the question was of how the metadata.xml out to specify the path to the files in the subfolders, especially if there were spaces in the path. After a series of incremental tests, it was found out to be still possible and the solution rather straightforward. Hopefully it will work for Diego also.

There was some translation work, and a few further questions on the mailing list to look at, before I finally got round to considering Michael Goodwin’s complex question on the setup.exe generated by an Export To CD-ROM operation failing on Windows 7 on 64 bit. A preliminary successful test on a Windows 7 machine turned out to be misleading: I had assumed it was a 64 bit machine but it turned out to be 32 bit after all. I will have to get back to trying this out next week. All this fine-tuning is bound to pay off in the upcoming perfected release of Greenstone 2: version 2.86.

This week I have been tidying up the new paged-image functionality so that it dynamically loads each page (rather than doing a full page reload each time) and also added the functionality that allows the user to choose from “Text view” (which only shows the OCR’d text), “Image view” (which shows the original image) and “Default view” (which shows both the text and the image). These are also switched dynamically which is nice and are remembered if you leave a document page and go to a new one.

I also fixed up an annoying problem with GLI. One of the ways you can customise collections in Greenstone 3 is by writing Javascript in the collectionConfig.xml file and those familiar with XML will know that you cannot put ‘&’, ‘<‘ or ‘>’ into text nodes (you have to replace them with &amp;, &lt; and &gt; respectively). These special characters a relatively common in Javascript so each time they are used they have to be escaped. The problem we were having with GLI was that it would read in the file and replace the characters with their usual forms (&, < and >) and when it went to save the file it wouldn’t escape these characters. So the next time this file was read in GLI would produce an error because the file was no longer valid XML. We eventually tracked this problem down and fixed it.

Next week I will continue to work on the paged-image functionality (specifically the “next page” and “previous page” buttons) as well as adding some new code to HTMLPlugin that will add any files referred to in CSS files (e.g. background-image) as associated files of the HTML page.

For the last two weeks, I was mainly learning the practical side of how to handle the Greenstone translations. Mainly how to generate the spreadsheets for translators to use, though there was also the opportunity for learning to handle translated spreadsheets. Next to that, there were some questions on the mailing list that I had a go at answering and uploaded the updates to the ACKU and AREU collections.

On the final 3 days, got round to working on getting the batch files in GS2 to handle not only spaces but also brackets in the Greenstone filepaths. There is still a final problem to resolve before the changes can be committed, but the Greenstone web server is now back to working again, despite Greenstone being installed in a path with brackets (and spaces). There’s even some allowance made in the makegs2.bat script–which is used to compile up GS2–to get apache to compile up even in those instances of there being spaces or brackets in the filepaths it works with. Fortunately, the change could be made in the makegs2.bat itself: it sets the command prompt in which Greenstone is being compiled up to be in short-filenames mode. This then is the situation that the apache compile scripts inherit also, making any space/bracket in the long pathname irrelevant.