Wikipedia:Evaluating Wikipedia as an encyclopedia

This page is about encyclopedic evaluations of Wikipedia. For a description of what Wikipedia is not, see Wikipedia:Wikipedia is an encyclopedia.

This essay is still in its larval stage, but aspires to be a NPOV assessment of Wikipedia. All well-meaning Wikipedians are encouraged to help its metamorphosis into something useful and good!

The scope of this essay is to evaluate Wikipedia's success or failure as an encyclopedia, using the standard accepted criteria for all encyclopedias: overall size, organization and navigation, breadth of coverage, depth of coverage, timeliness, readability, biases, and reliability. Other encyclopedic criteria peculiar to Wikipedia may be considered as well, such as the fraction of high-quality articles and the stability of its high-quality articles. Of course, Wikipedia has many non-encyclopedic aspects, but evaluating those aspects falls outside the scope of this essay.

Each encyclopedic criterion is given its own section below, with six subsections: Context, Data, Pro, Con, Overall assessment and Methods for improvement. All Wikipedians are encouraged to provide brief, well-reasoned and well-referenced arguments for both sides of every criterion. The goal is to identify Wikipedia's areas of strength and weakness dispassionately, ideally in comparison to a well-defined standard encyclopedia, such as the Encyclopædia Britannica. We should also note the cases where the presently available data are insufficient to make an evaluation, and where we should strive to obtain decisive data.

Despite the Pro and Con sections, the goal here is not to produce duelling POV polemics. On the contrary, the goal is collaborative insight; we seek to really understand what is right and what wrong with Wikipedia. Ignorance and faulty reasoning are our enemies, not each other. Therefore, we should not avert our gaze from the ugly and foolish aspects of Wikipedia; yet neither should we shrink from glorying in its virtues. We should strive to understand one another, to feel the force of one another's arguments and to recognize when a point cannot be decided. We are literally trying to comprehend Wikipedia and one another; although the goal is more ambitious and more laborious than a simple polemic, the prize is all the more worth winning.

Overall size

Context

Typical general encyclopedias have tens of thousands of articles and tens of millions of words. For example, the Encyclopædia Britannica has ~65,000 short unreferenced articles (<750 words, usually) and ~700 long referenced articles (2–310 pages), comprising roughly 40 million words. Its Index lists 700,000 topics (recently expanded from 400,000).

Data

The English Wikipedia now has 6,928,909 articles. Extrapolating from the statistics compiled from June 2006 through October 2006, Wikipedia now has over 700 million words and over 5 gigabytes of information. Therefore, it is roughly twenty times as big as the Britannica.

As of October 31, 2006, 34% of articles in Wikipedia were over 2,000 bytes (this includes images; without images, this would be roughly 400 words). 45% were between 500 bytes and 2,000 bytes (100 to 400 words); 21% were smaller than 500 bytes (100 words).

Pro

More topics are covered

Con

Many covered topics are better handled in other articles instead of being split up into permutations and combinations of (for example) bilateral relations between every two nations.

Overall assessment

Wikipedia is the largest encyclopedia ever assembled, eclipsing even the Yongle Encyclopedia (1407), which held the record for nearly 600 years.

Methods for improvement

Wikipedia's size is still expanding, with a doubling time of roughly one year. It should be noted that not all encyclopedic topics have been covered in Wikipedia as yet (see "Breadth" discussion below).

Organization and navigation

Context

Traditionally, the problem of finding information in a large reference work is managed by organizing topics alphabetically, by topic, and/or by including an alphabetical index. A more subtle organization is also imposed by the editor's choice of main articles and system of cross-referencing. Print encyclopedias generally give short entries to smaller topics, then cross-reference to a larger article in which smaller topics are discussed. This grouping of topics into larger articles was a key part of the "new plan" that was the basis for the 1st edition of the Encyclopædia Britannica.

Data

Wikipedia's methods for organization and navigation are roughly analogous to those of print encyclopedias, although they differ in key ways. On Wikipedia, information is usually found either by following a hyperlink from one article to another or by using a search engine. Hyperlinks are roughly analogous to traditional cross-references; however. As of October 31, 2006, Wikipedia articles had a total of 32 million internal hyperlinks (more than 20 per article on average), plus about 2 million links to Wikipedia sister projects such as Wikibooks.

A typical Wikipedia article has many more such cross-references than a traditional encyclopedia article. Similarly, using a search engine is roughly analogous to searching the index of a print encyclopedia; unlike such indices, however, there is no practical limit to the number of possible search terms. For comparison, the index of the 2007 Encyclopædia Britannica contains 700,000 terms, less than half the number of articles in Wikipedia.

Wikipedia is also organized by topic, an organization approach sometimes found in print encyclopedias, e.g., The Children's Encyclopedia. Wikipedia articles are grouped into categories that may be searched; this is most closely analogous to the Outline of Knowledge found in the Propædia of the Encyclopædia Britannica. However, unlike that Outline, the categorization of Wikipedia articles is not strictly hierarchical, instead forming a network of ideas. A scientific comparison of the relative advantages of the strictly hierarchical vs. network organization approach has not been undertaken.

Wikipedia has also developed portals intended to provide readers with an overview of a topic and a systematic education in a field.

Con

Wikipedia's ability to store and organize its articles and images is limited by MediaWiki's category system. Large categories are difficult to browse and must be actively maintained to prevent overpopulation. Small categories are often unintuitively named or have substantial overlaps with other categories, leading to related articles being categorized differently. It is impossible with the current software to view the unions or intersections of categories.

Breadth of coverage

Context

Wikipedia is a general encyclopedia; similar to the Encyclopædia Britannica and the World Book Encyclopedia, it seeks to describe as wide a range of topics as possible.

Data

In addition to the exponential growth in number of articles, the average (mean) length of each article has increased steadily in the past two years, although the typical (median) article has not. Between December 2004 and October 2006, the average (mean) length of articles increased from 2216 to 3317 bytes (roughly a 50% increase). During the same period, the percentage of articles that were larger than 2,000 bytes increased from 32% of all articles to 34% of all articles. Some portion – perhaps a very significant portion – of the increase in size is due to a large increase in images: during this same period, links to images (roughly equivalent to the number of images) increased almost sixfold, from 150,000 to 875,000.

The number of featured articles – articles recognized for being of significant quality – has also continued to increase, but as a percentage of total articles has shown a continuing decline since the high in October 2004 of 0.11% of all articles.

In a systematic comparison, Wikipedia now covers all but two of the topics covered in the Macropædia, the chief exceptions being industrial "Beverage production" and "Arts of Native American peoples". Although the Macropædia are generally much longer (2–310 pages) than the corresponding Wikipedia entry, this page-count comparison may be misleading, since the English Wikipedia's policy is to limit the length of articles, and the Macropædia articles are generally compilations of smaller articles. Therefore, a more appropriate comparison would be to compare the Macropædia with the set of articles found in the corresponding Wikipedia category.

Wikipedia does not yet contain an article for each of the 700,000 topics listed in the Index of the Encyclopædia Britannica. The relative fraction of the EB topics covered in Wikipedia has not yet been assessed, but is likely to be at least as high as 1% (7,000 topics), judging from the results of the analogous study of the 2007 Macropædia, and conceivably could be much higher. A WikiProject maintains lists of topics covered by other encyclopedia that are not yet present in Wikipedia.

Wikipedia:List of encyclopedia topics currently lists around 25,000 articles that can be found in one or more traditional encyclopedias but not Wikipedia; this number is slowly falling.

Pro

Wikipedia covers subjects that would be expected in traditional encyclopedias, as well as many cultural and technical topics often absent from such works. For example, several featured articles, such as Cyclol and Laplace-Runge-Lenz vector, are not even mentioned in the Encyclopædia Britannica, while the coverage of others, such as Photon, is far more complete, better illustrated, better referenced and better written than their counterparts in the Britannica. Even many non-FA articles are superior to those of other encyclopedias; for example, Wikipedia has far more information on the Encyclopædia Britannica than does the Britannica itself.

Wikipedia also does not shy from covering topics such as sexual harassment that are sometimes ignored by other encyclopedias.

Con

Wikipedia also does not shy from covering topics of dubious merit, such as pseudoscience. Although it is a public service to debunk such non-mainstream and even fraudulent theories, it could also be argued that even their rebuttal confers a scientific dignity on them that they did not earn with controlled experiments and sound reasoning.

Timeliness

Context

Traditionally, new editions of print encyclopedias were released every few decades, as their information become noticeably outdated. In 1933, Elkan Harrison Powell introduced the idea of "continuous revision", in which every article is revised on a systematic schedule and the entire encyclopedia revised and reprinted every few years to incorporate the changes; most encyclopedias now employ continuous revision.

Data

Wikipedia takes "continuous revision" to its ultimate limit—almost all articles may undergo revision at any time—allowing the project to incorporate articles concerning current events and recent developments in science, politics and culture. For example, the 2004 Indian Ocean earthquake article in the English-language edition, was cited frequently by the press in the days following the event.

Pro

Many times, very recent news events are almost immediately covered on Wikipedia.

Con

Current news stories and modern phenomenons can receive a disproportionally higher amount of attention than older, more established topics. For example, Execution of Saddam Hussein is vastly more referenced and thorough than is the main article, Saddam Hussein. Also, 2006 Winter Olympics is substantially more developed than 1976 Winter Olympics.

Readability

Context

Even the best reference work is of limited use if it is poorly written and unintelligible to its audience. Various readability standards are available for use. One, the "Flesch Kincaid grade level test," may be appropriate. The typical American reads at the 7 to 10 grade level, with college students reading at levels up to 16. For people who learn English as a second language, a level of 16 may be too high for usability. Articles, therefore, with F-K levels over 16 are likely to be not understood by typical users of the encyclopedia. Many articles are currently well over 20 for grade level, which means that even the editors writing them are unlikely to be able to read them.

Pro

Wikipedia benefits from a large community of proofreaders, who may detect errors and ambiguities; for comparison, the Encyclopædia Britannica employs only twelve copy editors. It also seeks to foster the development of clearly-written articles through various methods, e.g., by requiring "excellent writing" a criterion for becoming a "featured article".

Con

On the other hand, the number of individual contributors to the project—millions, if unregistered users are included—is far greater than any print encyclopedia; this can result in large variations in tone and style between (and sometimes within) articles; common policies and style guidelines within each language edition attempt to address this.

Biases

Context

Encyclopedia editors have a responsibility to keep articles as free of bias as possible. Historically, even the best works have suffered from bias; for example, the "Lynch Law" article of the 11th edition of the Encyclopædia Britannica describes the Ku Klux Klan as a "protective society" and defends its actions. Even expert editors may exhibit prejudice; a classic example is Dr. George Gleig's rejection of the Newtonian theory of gravity—a theory long accepted within the scientific community—in the 3rd edition of the Britannica.

Pro

Being freely available, Wikipedia need not appeal to the tastes and interests of buyers and, thus, is able to avoid some bias that can skew commercial encyclopedias. As a counterexample, the 2007 Macropædia tends to describe only the Western branch of a field, specifically in histories of architecture, literature, mathematics, music, dance, painting, philosophy, political philosophy, philosophical schools and doctrines, sculpture, theatre, and legal systems. Similarly, the 2007 Macropædia allots only one article each to Buddhism and most other religions, but devotes fourteen articles to Christianity, nearly half of its religion articles. Being international in scope and having an open-editor policy are likely factors that contribute to making Wikipedia articles less parochial than those of the Britannica.

Con

Although Wikipedia has a policy on maintaining a neutral point of view, a few zealous editors may seek to influence the presentation of an article in a biased way. This is usually dealt with swiftly and, in extreme cases, biased editors may be banned from editing. In general, Wikipedia's editors also strive to be complete—to include all aspects of a topic and reflect the prevailing consensus in the scholarly community.

Overall assessment

Absent a way of systematically and neutrally measuring bias across many possible topic fields, an overall assessment is impossible to obtain.

Reliability

Context

Readers of an encyclopedia must have confidence that its assertions are true. In traditional scholarship, confidence is established by appealing to the authority of anonymously peer-reviewed publications and of experienced experts. However, as experts can disagree, and any one may be biased or mistaken, peer-reviewed publications are considered to have higher authority. Most encyclopedias include both of these; for example, the 699 Macropædia articles of the 2007 Encyclopædia Britannica give both references and the names of the authorities that wrote those articles, many of whom are leading experts in their fields. By contrast, most of the ~65,000 Micropædia articles neither give citations nor identify their authors; in such cases, the reader's confidence derives from the reputation of the Britannica itself.

Wikipedia appeals to the authority of peer-reviewed publications rather than the personal authority of experts. It would be difficult to determine truly authoritative users with confidence, since Wikipedia does not require that its contributors give their legal names or provide other information to establish their identity. Although some contributors are authorities in their field, Wikipedia requires that even their contributions be supported by published sources. A drawback of this citation-only approach is that readers may be unable to judge the credibility of a cited source. The reader must be satisfied on two points: first, that the cited source is a genuine publication and, second, that it supports the assertion made by the article. Although the first point is usually easy to check, the second may require significant time, effort or training.

Pro

Wikipedia's reputation has improved in recent years, and its assertions are increasingly used as a source by organizations such as the U.S. Federal courts and the World Intellectual Property Office—though mainly for supporting information rather than information decisive to a case. Wikipedia has also been used as a source in journalism, sometimes without attribution; several reporters have been dismissed for plagiarizing from Wikipedia.

The English-language Wikipedia has introduced a scale against which the quality of articles is judged; roughly 6,640 have passed a rigorous set of criteria to acquire the highest "featured article" status; such articles are intended to provide a thorough, well-written coverage of their topic, and be supported by many references to peer-reviewed publications. Wikipedia has been described as a "work in progress", suggesting that it cannot claim to be authoritative in its own right, unlike more established encyclopedias. Despite its shortcomings, however, many hold out hope for its future. An analogous historical example is provided by the Encyclopædia Britannica, whose first edition was of uneven scholarship, but which rose to great heights in its later editions; the elegant words of its first editor, William Smellie, pertain to Wikipedia as well:

“ With regard to errors in general, whether falling under the denomination of mental, typographical or accidental, we are conscious of being able to point out a greater number than any critic whatever. Men who are acquainted with the innumerable difficulties of attending the execution of a work of such an extensive nature will make proper allowances. To these we appeal, and shall rest satisfied with the judgment they pronounce. ” — William Smellie, in the Preface to the 1st edition of the Encyclopædia Britannica

Number of high-quality articles

Con

Given its large base of editors and its six-year history, it is disappointing that only 3000 articles have been certified as "good" or "excellent".

Coverage of key topics

Context

General encyclopedias such as Wikipedia and the Encyclopædia Britannica are assumed to cover key topics of general interest and importance to their readers. Several lists of such key topics have been assembled such as the List of 1974 Macropædia articles, the List of 2007 Macropædia articles, and Wikipedia's Wikipedia:Vital articles, Core and supplemental Core Topics. It is assumed that such articles should be brought to high quality.

Data

The Vital Articles page lists 1182 topics considered essential to Wikipedia's quality. Of these, only 72 are featured articles (6%). There are also 131 good articles, meaning that roughly 17% of the vital articles are good or featured. At the other extreme, 133 articles (11%) are listed as either stubs or have a cleanup tag. Presumably, the remaining 72% are B-class or start-class on the assessment scale.

Presently, there are roughly 1300 featured articles and roughly one article per day is promoted to featured article status. At this rate, it will take Wikipedia roughly 8 years to produce 4,207 featured articles, matching the number of equivalent articles in the 1974 Macropædia although, naturally, the two sets of articles will not agree.

Pro

Matching the Britannica's quality on the decade time-scale is a reasonable expectation.

Con

The data suggest that a full 83% of the Vital Articles require substantial work before they will match or exceed the standards found in other encyclopaedias.

Many of the Vital Articles on the list are broad, general topics, which tend to be much harder to bring to the comprehensive and well-referenced level of featured articles than smaller, specialized subjects which one person can master.

Stability of high-quality articles

Context

A key quality of good encyclopedias is that their articles change only with advances in their field, and do not spontaneously degrade. However, there are plausible reasons for assuming that previously high-quality articles can decline if their principal contributor neglects to fix vandalism and inexpert edits in a timely way.

Data

Roughly 20% of all articles promoted to Featured Article status (340 articles) were later demoted.

Vandalism to certain featured articles such as Mauna Loa, Sun and Ryanair has gone uncorrected for long times, once their principal author stopped watching over them.

Presumably, such vandalism could be modeled as a Poisson process to predict the rate at which unwatched featured Articles would decline. However, insufficient data have been taken to determine the characteristic rate of vandalous edits.

Pro

Some conjecture that the FA demotions were due to rising standards, particularly for inline citations and thorough referencing, rather than to a significant decline in quality.

Con

Others conjecture that their delisting was due to a real decline in quality.

Overall assessment

The reasons for delisting have not been studied exhaustively, nor has anyone shown that earlier FAs were qualitatively different from recently promoted FAs.

Methods for improvement

A more pro-active automated custodial system would seem like a good idea, although difficult to implement judiciously.

Controversial evaluation criteria

Relative fraction of Featured Articles

A relatively small fraction of Wikipedia's article are Featured Articles. Although this fraction is remaining constant at the German Wikipedia, it is gradually declining at the English Wikipedia, indicating that new articles are being created faster than older articles are being promoted to Featured Article status. If this trend were to continue indefinitely, Wikipedia would never be able make all of its articles into Featured Articles. Therefore, if only featured Articles are considered as good, then the average quality of Wikipedia articles will decline inexorably. From this, some may conclude that Wikipedia is destined to fail.

This argument has been challenged both on its premises and its conclusions.

This reasoning neglects the fact that many articles have not been rated for quality yet. Moreover, editors customarily nominate their articles only once they have reached a perceived apogee of quality and completeness; articles still under improvement may be good without being complete or finished, and therefore may not be nominated for certification as a good or featured article. Moreover, Wikipedia is growing exponentially and, therefore, a significant fraction of its articles were started recently. Naturally, such new articles have not had sufficient time to be improved to high encyclopedic quality.