We are working Computing with CSD to develop an integrated sign-in for Kritikos. At present, it is necessary to redirect users to the University Student Portal in order for them to sign in and perform an initial search from there. In future they will be able to sign in directly from the Kritikos site. We’re sure this will improve the overall experience for Kritikos users.
During the next couple of weeks, we’re demonstrating Kritikos to the Engineering students here in Liverpool.
Students are invited to drop by and have a play with the search engine; we can try looking for concepts and terminology encountered in the most recent lectures, etc.
If you’re a student and have feedback or comments, please leave a reply in this blog.
Thank you to students Zofia and Lucas for coming in this afternoon to help us finalise some of the site design and functionality.
Both students reacted positively to the proposed re-branding to Kritikos. There was plenty of discussion about the name, the logo design and colours, but quite a strong consent to retain what has currently been produced.
- There was a feeling that the current Learning Registry icon should be removed and placed closer to the individual activity icons.
- Suggested use of a design/colour indicator to represent the amount of LR activity for each resource
Resource Details page
- Strong preference for user activity data to be aggregated and only to show the first few items, with a “view all” option if users want to see more.
- Use of module as the subject of the activity data, rather than user
- Use coloured icons for up- and downvoting
- More consistent use of icons and other screen elements to be comply with the Search page.
- Implement second level filtering. Currently the page filters by module or by user. But a user should be able to further filter their records by module. Also, provide different sort options
- A request to introduce ‘favourite’ activity, in addition to current activity types (relevant, not-relevant, comment, recommended), without tying it to any particular module.
We need to make a final decision as to how individual students will be uniquely identifiable in our Learning Registry node. At present (01 Jan 2013), we record:
- students’ full name (given and family name)
- their degree programme (course)
- degree title (BEng, MEng, etc)
- current year of study.
In the short term, this is likely to provide ‘unique’ enough, but since we are now able to retrieve the students’ (or staff’s) University ID, we propose saving this in an encrypted form.
Having a unique identifier will make it easier to develop the code to prevent (i) users from voting for their own interactions (recommendations, comments, etc.) and (ii) users from sending duplicate paradata.
We’ve been looking at the best way to implement upvote/downvote functionality in our website. The purpose of this is to enable users to endorse or disagree with other users’ activities, thereby introducing a level of self-regulation or democratisation. It also allows users to earn credibility or trustworthiness, which can then be used in the algorithm used to rank search results, and also incentivise students to interact positively with the ENGrich service.
Although some social media sites such as Facebook only support the positive like, our initial position is that we should allow (identifiable) users to give negative votes for activities they do not agree with. We shall review this position with our student group once we have gathered and analysed sufficient activity data.
The paradata statement we propose publishing into Learning Registry will take the following form. Note that the object of the paradata statement is the URL of a page describing the first user’s activity (e.g. comment, recommendation), not the URL of the actual resource itself.
"owner": "AN Other",
"payload_schema": "LR Paradata 1.0",
"displayName": " Student Name",
"University of Liverpool",
"Aerospace Engineering with Pilot Studies",
"context": "ENGrich website",
"content": "This resource was upvoted on the ENGrich website by a 3rd year student reading Aerospace Engineering with Pilot Studies MEng (Hons) at the University of Liverpool on June28, 2012."
- prevent individual user from making more than one upvote/downvote on a particular activity
- prevent users from upvoting (or downvoting) their own activities
Today we have published two large batches of data into the JLeRN Learning Registry node:
- 643 comments by the students on individual resources (URLs)
- 5,400+ ‘assertions’ by students that individual resources are not relevant to engineering and/or education. Such data will eventually help to make future search results more relevant by demoting those resources that relate to other subjects (medicine is a common one) or target audiences (commercial, sales, marketing resources with little/no educational content).
We have also started work on the user interface to display these new paradata ‘types’.
We have spent much of today working on the ranking algorithm to use for search results.
Each time a student interacts with a resource, the event will be captured and published into Learning Registry. Examples of such events include:
- matching a resource with a particular study module (alignment, in LR Paradata parlance)
- recommending a resource for a particular study module
- commenting on a resource
- sharing a resource (e.g. on Facebook, Twitter or LinkedIn)
We are calling these primary interactions, each of which can carry different ‘points’ for weighting subsequent searches.
In addition to these events, we will provide a mechanism for users to like/dislike (or agree/disagree with) other students’ primary interactions, thereby introducing a system of self-moderation or ‘democratisation’. Each user will therefore build up a trustworthiness index, based on how helpful or otherwise their peers judge them to be.
The ranking of the resource will therefore depend on the number of interactions found in LR, the relative importance of these interactions and the trustworthiness of the contributors.
This post presents the results from the online survey that the students completed after road-testing the initial iLIKE and ENGrich user interfaces last week. Some feedback from the ENGrich team is included too.
iLIKE – Module Tab
Is the Module tab potentially useful?
- Good news
Is there any missing or redundant functionality to the Module tab?
- I didn’t find the missing or redundant module tab
- When searching for some modules such as Aircraft Capstone Design, this did not appear whilst other modules did. This may be because the module title has changed but it could not be found even when searching for similar names.
- The results could be automatically linked to the upcoming lectures in that module, effectively a search of pre-defined terms set by the module co-ordinator linked to the module running order.
- Yes, we see the potential for this; the search results would need to be more ‘granular’ than at a module level. Our system would need to read weekly details from VITAL (Liverpool University’s installation of BlackBoard) to narrow the results even further.
- I feel that the iLIKE widget could be linked directly to the course material put on by the lecturer as well. There should be an easy button placed onto the front screen that people can use to add extra material that they think other students should find useful.
- This would require full integration with VITAL/BlackBoard, which isn’t feasible in the immediate future.
Is the display satisfactory; any suggestions?
- One potential problem I found was the number of tabs that could be opened during a 10 minute session, it did become quite tab heavy!
- Yes, we’ll aim to limit the number of windows/tabs that open.
- With only 3 results displayed in the preview results, users might be led to believe that these are specifically filtered i.e. most popular. If this isn’t the case, maybe it could be?
- In future, we aim to order thumbnails by most popular first for the Module tab, and most recent first for the My resources tab, when we implement it.
- I think that the layout is clear to understand and seems easy to use.
iLIKE – Search Tab
What does the Search tab do?
- have been sorted out, related to engineering, can be used for finding useful materials by students’ rating. A Google custom search engine based academic search tool
- The search tab allows a key word search to select a presentation/video which will potential display relevant material. Currently the search is either for presentations or videos. The system communicates with the Google search engine, and then displays 10 results per page. The results may have been recommended by another student which will be indicated with a green and blue arrow.
- It uses a Google custom search to narrow the search to visual resources and then overlays these results with any information in the learning registry.
- The search tab looks through information on Google that is relevant to the word that you have placed in.
How does this Search differ to Google, Bing, Baidu, etc?
- it is more academic. it has more limitations of materials.
- This search focusses on engineering topics, and allows users to interact with the search engine which will moderate the search results with the intent of over time, seperating the good from the bad.
- It only searches visual resources, but includes information about that search result from the learning registry – if the information exists.
- It should bring up information that is more relevant to engineering than the usual Google page and therefore should be more useful.
Is there any missing or redundant functionality you can think of?
- Possibly a different approach may be to pre define the module you are searching for, then any searches that you consider will be linked to this module, you may then have the option to rate the relevance of the document/video to the subject but if it applies to another module, make this option possible as well.
- Yes, we see the potential usefulness here. If search results are presented in the context an individual module, then you could tick checkboxes for all the relevant results and submit the ‘alignments’ in bulk, thereby saving you time.
- Pupils should be able tyo indicate which pages that have come up are the most relevant and which they think is still not relevant to engineering.
- Yes, as above, a system of checkboxes that can be ticked to indicate non-relevance might be efficient.
ENGrich Search/Results Page
What does this page do?
- This is the filtering process of the program and suggests relevant documents/videos and ranks from 1 to 10 on each results page.
- Displays all the results from the iLIKE search.
- This page displays all the results that have come up when you searched a topic.
Is there any missing or redundant functionality you can think of?
- An issue may be the ranking system adopted. A document may be very relevant for a year 4 student but too much detail for a year 1/2 student. Will this be taken into account as this may affect the quality of results that the user obtains. Also the number of tabs that can be opened can become a bit of a headache. I found I could easily close the main results page when I just wanted to close a powerpoint. Maybe a ‘challenge’ could be added to ensure they wish to close the main page.
- We think that this is exactly why we need your comments, which can be shared with the community.
- See comment above regarding tabs
- The results could be split into 2 sections side-by-side: one shows Google results which don’t have information from LR; the other has only results which have LR information.
- At present, the project is about integrating LR into Google search, thereby ‘engriching’ the results. At this stage, we don’t want to develop separate search engines or present their results separately
- This should then be vetted and if an independant person feels the same then there should be an option that stops this website from appearing when people search again.
- Indeed, our algorithm for ordering results will take into account the number of times a resource has been marked ‘not relevant’. Such resources won’t be displayed in subsequent searches.
Is the display, design and layout satisfactory
- The layout is good, text size is fine and the simple design makes it straight forward to navigate.
- Some way of clearly showing, maybe using the Google logo, that this is still a search of the whole Web through Google, not just of liv.ac.uk resources for example.
- Yes, we absolutely agree, thank you
- I feel that the display and layout is very easy to understand and use.
ENGrich Resource Detail Page
What does this page do?
- This gives a preview of a selected document and supples information on whether the document has been reviewed before.
- Displays information relevant to the search result selected.
Is there any missing or redundant functionality you can think of?
- If I find a useful presentation this time, I want to use it next time. I prefer having a link directly from my profile so that I could find it straightforward.
- Yes, the planned ‘My resources’ tab in iLIKE will do exactly that and link to a page in ENGrich which shows all your resources.
- The preview may be a little too small that it requires opening onto a full page. So it may be worthwhile to increase the size of this otherwise it does not offer much use. Would it be possible to save from this page directly instead of having to open and then save?
- For some formats (PPT, DOC, PDF and possibly images) we can provide a “Save as…” button. This won’t work for videos or Flash.
- Some way of the saving this search result, like bookmarking or adding to favourites, but built into the student portal/iLIKE channel/VITAL.
- By interacting with resources, they will automatically be added to ‘My resources’ in iLIKE. VITAL/BlackBoard is outside our control.
- If its possible to show who the source belongs to eg. a university or organisation? Instant recognition of likely reliability of result.
- We are adding the web icon for the source organisation, to help with ‘brand recognition’.
Is the type and format of information presented useful?
- the left side is the presentation or videoes and the right side is only a link to the original website. why don’t we make the space for the link smaller and make the space for the presentation or videoes bigger?
- Yes, we’re still working on the layout and will use the space more efficiently.
- As stated above, the size of the preview for some font sizes may be too small, which will make the preview useless.
- As the LR becomes more popular, a tabulated form of the “What others are saying…” section might be better. Could be information overload otherwise.
- Yes, absolutely. We’re discussing ways of aggregating data in a compact and clear way, e.g. “John Smith and 21 others recommend this resource for the Spaceflight module (AERO319).”
Are there any other ways you would like to “interact” with resources
- for example, I want to find something about Abaqus. It would be good if ilike is connected with the company who design this software. because they publish many things on their website. sometimes I only need this intermediary (bridge) for finding useful things.
- Use normal Google, etc. for finding company information. ENGrich is fine-tuned for finding and interacting with learning resources for Engineering
- Maybe a twitter account that ‘tweets’ whenever a useful document has been found. For example “@CiviliLikeLivUni just recieved a recommendation for CIVE430. by …” I don’t think facebook would really work. Like I said at the session I am more than happy to help as it will be improving student experience in the future. I look forward to the next session. (D Mount)
- Nice suggestion J We’ll look at the Twitter application programming interface (API) to see what we can do.
- Option to share via Facbook/Twitter or add to a resources section in VITAL/BlackBoard.
- We’ll add share buttons for Facebook, Twitter and LinkedIn. VITAL/BlackBoard is outside of our control.
Are you willing to come to future testing sessions?
- Great, thanks
Thanks to all the students who came along to road-test the first alpha test version of the ENGrich visual media search engine. We’ve got lots of notes and video to go through and analyse, but we were pleased to see how well it appeared to go down.
Thanks also to Simon Hatton and Paul Hagan in Computing Services for the work they’ve put in this week and to Phil Barker from CETIS for coming down from Edinburgh.
Following on from earlier work on the iLIKE widget that allows students to view relevant resources by module, we have met again with Simon from Computing Services to outline the functionality of a general search box. We hope to have this ready to try out by the end of this week.