Thesis/Discussion and Conclusion

From Researchwiki

Chapter 4 presented an analysis of data collected throughout this research, the behavioural analysis gained from log file data, and the survey results, used to better understand users' behaviour. This chapter will review the project, draw conclusions from the analysis, and place those conclusions within the available literature, showing how this research contributes to the field. The chapter will also explain the limitations of the research conducted, and identify paths for future research in the area.

Review

This research began by reviewing literature in the areas of online communities, and wikis, with the aim of discovering gaps in the literature, opening possibilities for research. The literature review explored the history of the wiki concept, and followed the growth of the popular Wikipedia, along with its criticisms. Many of these criticisms came out of the fact that Wikipedia calls itself an encyclopedia, and can not provide the expected level of reliability while using a radically open-editing model.

Several theoretical viewpoints of wikis were presented, models behind the design, and visions for the future of wikis. A wide range of uses of wiki technology were explored, personal and collaborative uses were reviewed, along with the various writing, collaboration, and communication styles used. This lead into a discussion on open-content communities, focussing specifically on the Wikipedia community, how the community functions under a self-governance model. Finally on wikis, the literature review compared wiki systems with other technologies, and explored in detail several common wiki implementations.

The literature review concluded with a survey of trust, reputation and attention, exploring the mechanics of trust, and several failures of trust in Wikipedia. Reputation was defined, and several content reputation systems surveyed. Attention was introduced, and an explanation of the use of attention data was provided.

Several gaps were identified during the literature review, leading to several questions. Chapter three selected some of these questions as the focus of this research, and developed methods to answer these questions. The primary goal was to see if there were any methods by which a wiki community can provide some level of authority. One proposed solution, is to have an authoritative group review articles, however this in itself is a huge task. This research tests methods of rating articles, so only good articles need to be validated. The two methods studied were user ratings, and attention data.

A MediaWiki system was installed, and extended with a simple rating mechanism. Over almost two months, users' actions on the wiki were recorded. At the end of the data collection period, the logs from the wiki were analysed, to determine the effectiveness of the system generated ratings, and of the attention data collected. A survey was then distributed to participants of the wiki. The survey sought to elicit from the students their perceptions of the wiki and the rating system, to better explain the behaviours observed in the wiki.

Limitations

The research presented here was completed in partial completion of an Honours research project, and had to meet the requirements thereof. A limit of one school year is a non-negotiable requirement. Ethics approval also had to be sought before research involving human participants could commence.

This research focused on a small group of undergraduate students. Witnessed in this research were how the dynamics of a small group differs from the expected behaviour of a larger wiki. The unexpectedly low amount of data makes for weak answers to the research questions, however, the research exemplifies the limitations of small wikis, and provides valuable lessons for future wiki designers.

The findings presented in this research may not be successfully extrapolated to larger wikis, or wikis used outside of an educational setting.

Findings

Two methods were trialled in this research in a small wiki. While neither provided accurate results in this setting, further research needs to be done with larger and established wikis. Further research may also develop alternate methods to test.

Initial analysis of the data highlighted the care with which MediaWiki log data must be analysed. Discrepancies were found between Apache log data and the MediaWiki statistics available. These are explained in chapter four as being caused by failed attempts at an action, or actions that require more than one step to complete. This inflates the Apache logs, requiring careful filtering of log data.

Analysis of ratings did find that the vast majority of ratings were positive, with at least 78% of ratings found in the top 40% of the rating scale. Dellarocas (2001) and Resnick and Zeckhauser (2002) also found an overwhelming number of positive ratings when studying eBay ratings.

Logs showed that while between 0.84% and 1.1% of page views resulted in ratings, 22% of these were corrections made on previous ratings, and only 49% of these ratings were usable as reliable results. The remainder of ratings were made by users who had also edited the pages they rated.

In small education wikis, no.

In this trial, pages generated by users were assessed manually by the researcher. The analysis involved a comparison of the manually assessed ratings with the system generated ratings. Little correlation was found between the two, and thus it was concluded, that in the wiki community trialled, user rating were not a reliable indicator of article quality.

Biases towards other members was identified as a factor that possibly reduced the accuracy of user ratings. The participant survey determined that this may significant factor. This fact may reduce in significance as the size of the community increases, and the likelihood of interaction between friends lowers.

Current evidence suggests factual quality is most important.

The participant survey tested three possible answers to this question. Results show that all students sampled felt that factual quality is the most important factor. Visual quality and writing style were considered slightly important, however a varied response showed they are not universally considered important. There was not enough data from the Apache logs to allow this question to be tested based on users behaviour.

In small wikis, no, although analysis suggests attention data may be more useful in larger wikis.

Analysis of data showed some correlation between time spent rating pages, and their quality. Comparisons between time taken to view, and article quality showed little correlation. A known problem with using attention data this way, is that users may visit a good quality page that they have seen before, and quickly move on to another page. In a larger wiki with more pages, the dynamics of this behaviour may change.

Apache logs as an attention metric also poses its limitations. Client side monitoring would provider richer data, allowing the researcher to have a better idea if the user was actually reading a page, had simply left the page open in the browser while working on another task, or closed the browser entirely.

Major stakeholders of wiki pages (ie. the principal authors of a page) will often monitor that page for changes. Any change to that page will be reviewed quickly, and often fixed (if the edit was poor), or appended with ideas brought about by the new edit. This creates a very active community, where ideas are exchanged quickly. This behaviour was not observed in this smaller wiki.

There was very little collaboration between users on most pages. Observation suggests students may perceive pages where the entirety of text was written by another student as belonging to that student, and are hesitant to edit it.

It was expected that opening up a wiki would increase editing, by allowing opinion into a wiki, making users feel more free to contribute. No significant exchange of ideas was observed in the wiki.

A possible explanation is that the number of participants constituted a pre-critical-mass, a point at which the wiki has enough members and momentum that any change will be responded to by another user. There was not enough interaction seen in the wiki trial for the allowing of subjective information to have any effect.

Observation of the wiki found little interaction between participants. One of the exercises required of students was to create a shared page, where each student was required to participate by introducing themselves. On these pages, a significant amount of interaction was observed. While it was rare to see any interaction where students posted their personal assignment tasks, most shared pages experienced a high level of collaboration, and evolved quickly. These pages saw students adopt a shared presentation style, and in some cases, a form of group identity, where the group adopted a shared name.

The participant survey elicited mixed feelings towards the wiki as a learning tool. Some saw it as just a form of publishing, whereas others identified the potential for shared learning and collaboration.

The Georgia Institute of Technology with their CoWeb wiki system saw some benefits of wiki use by students (Leuf & Cunningham 2001). In the teaching environment studied in this research it was seen, that in the right conditions, students form a group identity and shared respect. Students also find the wiki an enjoyable and efficient medium for discussion, learning, and sharing of ideas.

The need for further research

Observations made during this research highlight the behavioural differences between small and large wikis. Wikipedia is a common target of study, due to the need to solve problems with its reliability. There is comparably little study done on small and medium wikis. There is a need for future work to study the dynamics and interactions of smaller wikis, determine a software tool-set that best fits their needs, so that they can be most effectively used. Further work is also needed to understand how best to design and manage small wikis in educational settings.

This research observed a high number of potentially "biased" ratings. These ratings were made by participants who rated the same pages they edited. Further work can determine firstly if user ratings produce reliable recommendations in larger wikis. Secondly if the high number of such ratings is a feature only of small wikis, and if such ratings are indeed inaccurate?

Results of this showed a high number of positive ratings. This has also been observed in communities such as eBay (Dellarocas 2001, Resnick & Zeckhauser 2002), however Malda (1999) through his work has created a system (and community) that generates more accurate ratings. Are there any features of the software system, its mechanisms, or the community that can account for these differences?

Wikis comprise a community of editors, and over time, networks of friends develop. Do these social networks affect the way members perceive friends articles, and ultimately ratings of such articles?

Analysis in this research showed inconsistencies between the manual ratings, and the system generated ratings. Is there a need for some form of training so that users understand how articles should be critiqued and assigned an accurate rating?

Established wikis form a culture, a with it a set of norms, expectations and guidelines (written or otherwise), that new members learn and follow when joining the community. Is training required to members in private (corporate) and educational wikis to act as a surrogate to cultivate an effective wiki in its early stages?

Conclusion

This chapter provided an overview of literature reviewed in this research, and revisited the research questions. It summarised the processes followed in this research, and their limitations. The chapter then proceeds to answer the research questions presented in chapter three. Finally, this research concluded by providing several paths for future research focusing on small scale and educational wikis and their communities.

Bibliography