ACM-SIGMOD
Digital Review: Restaurant Ratings for
Technical Papers
THE BASIC IDEA:
Create a guide to works likely to be of interest to the
SIGMOD membership, to help visitors locate "important" work, through an
electronic review and editorial process designed with the on-line world
in mind. ACM-SIGMOD Digital
Review itself stores no documents, only reviews and pointers. The
primary repositories are the ACM archive on computing related research
(CoRR) and the ACM SIGMOD Anthology, supplemented as needed by private
web pages at authors’ institutions.
THE MOTIVATION:
Our system of refereed journals is based upon a paradigm
where information dissemination is expensive. There is a great deal of
effort devoted to a review process that is meant to ensure a level of quality
in what is published. As publication costs have dropped, the numbers of
journals have proliferated, to the point that most of us today find it
hard to keep up with all published literature even in our own narrow areas
of specialization, let alone in the field at large. Also, the review and
physical paper publication process introduces long delays, to the extent
that many papers are no longer topical by the time they appear in print,
and many researchers choose to focus on conference, rather than journal,
publication.
As conference publication has gained in prestige, we have
also seen an explosion in the number of conferences. There are conferences
today that have few attendees who are not authors, and finance their entire
budget out of registration fees paid by authors of accepted papers. Even
at prestigious conferences, the quality of reviews often leaves authors
dissatisfied. Furthermore, conference publication, though substantially
faster than journal publication, introduces a significant delay itself.
The key point is that the costs of disseminating information
have fallen dramatically, as has the time required for such dissemination.
Review processes designed for a world with high dissemination costs are
not necessarily the best gatekeepers to dissemination in the new world.
In fact, technical reports are often circulated widely by many authors
prior to, and occasionally even in lieu of, publication. Most researchers
provide access to their recent technical writings through pointers from
their web home page. In fast-moving fields, other researchers commonly
build upon such work, without waiting for formal review and publication.
Our attempt here is to rethink the review paradigm from
scratch. If information dissemination is cheap and quick, additional costs
and delays imposed by the reviewing process quickly become burdensome.
Moreover, there is no reason to restrict what is disseminated when the
cost of disseminating everything is close to zero. Of course, human capacity
to read and absorb information is not scaling with changes in communications
technology. So there remains an important role for a review mechanism.
The idea of ACM-SIGMOD Digital
Review is to provide readers with help in choosing what to read
without in any way limiting what they have access to. Think of it like
a movie or restaurant review column: as a consumer you may choose to obtain
guidance from such reviews, you will learn how to calibrate the opinion
of specific reviewers, and you will decide whose reviews to trust. As a
restaurateur you first hope that your restaurant is reviewed at all, and
then you hope that the review is good. Note that customers have access
to all restaurants, irrespective of the reviews. Furthermore customers
have access to diverse opinions when there is a difference of opinion --
a restaurant awarded two stars by Michelin could be panned by Zagat. In
contrast, traditional journals and conferences merge the review and access
functionality so that readers of the journal or conference proceeding have
access only to works that have a positive consensus review.
HOW IT WORKS:
-
Anyone may review any publicly available material, such as
any free and public web page. Documents not available online, or available
only for a fee, may also be reviewed, but only if they appear in a "standard"
forum, covered by the Trier DBLP bibliography. In addition to documents,
other technical material may also be reviewed, including software, demos,
presentations, and bibliographies.
-
Reviews are completely public, as also the reviewer's identity.
Anyone may view any review, along with the name of the reviewer. A password-secured
registration process is used to authenticate reviewer identity.
-
No documents are submitted to the ACM-SIGMOD
Digital Review for review. The reviewer chooses what to review.
In consequence, some hot material may have multiple reviews. The vast majority
of research output will never be recognized with a review.
-
ACM-SIGMOD
Digital Review links to a URL for the reviewed matter when possible.
In the future, the DBLP bibliography will have reverse links from article
headers to reviews in ACM-SIGMOD
Digital Review.
-
We intend in the future that authors be notified, if possible,
when new reviews of their work are submitted. Authors would then have an
opportunity to respond to the reviewer. We intend to experiment with different
mechanisms to encourage such "interactive reviewing". Where possible, we
would like such interaction to be public, even though private interaction
is also encouraged.
-
Readers may of course read any material of their choice.
To help make this choice, readers may read any reviews of their choice
in ACM-SIGMOD Digital Review.
A search form allows readers to select based on properties of the review,
such as reviewer identity and rating given, and also based on properties
of the reviewed material, such as title, author, date, and subject area.
-
Since any one can write a review, it is expected that reviews
may have a high variance in their quality and reliability. To assist readers
in determining the credibility of reviews, we maintain a board of accredited
reviewers comprising most senior leaders in the database research field.
Readers may choose to pay as much weight to this accreditation as they
wish.
-
Program Committee members for major database conferences,
such as SIGMOD, PODS, VLDB, ICDE, PDIS, and EDBT, will automatically have
"guest editor" privileges from the conference submission deadline through
a few weeks after the PC meeting. We will urge PC members to write positive
reviews for any papers they read and liked, particularly if the paper eventually
did not get accepted to the conference in question. (Assuming, of course,
that the conference submission is publicly available, such as through CoRR).
Such guest editor reviews would be so identified explicitly, and may be
used for querying purposes. Guest Editors, like regular Editors, may write
negative reviews if they wish (for papers that were accepted over their
objections), or reviews of any other papers that they happened to read
even if these were not submitted to the conference in question.
-
To encourage discussion with regard to the hot topics of
the day, ACM-SIGMOD Digital
Review maintains a threaded bulletin board as an on-line discussion
forum.
-
A "front page" highlights reviews or articles of particular
interest. While readers are free to read any reviews of their choice, the
front page directs them to a few of particular interest. Thus, the front
page operates as a "meta-review" of sorts, directing users to specific
collections of reviews that are interesting for some reason.
SOME BENEFITS:
The new paradigm has benefits all around, to authors,
to readers, and to reviewers. We consider each in turn.
Benefits to Authors
-
Immediate dissemination of ideas. No delays on account of
slow reviews or long publication queues. A paper is available to the public
as soon as it is written. Each review is available to the author and to
the public as soon as it is written (we do not have a specified set of
reviewers with a need to wait for the last one of them). Readership can
be attracted to the work as soon as favorable reviews appear. Thus, there
is the opportunity to gain quick attention to seminal or controversial
work.
-
The author exercises judgement on what to publish – ultimately
this self-policing will lead to more efficient maintenance of quality than
the current journal system of minimally satisfying nit-picky reviewers.
-
The quality of reviews is improved. In today’s system, it
is often the case that an author gets a review of barely a few lines, or
gets a review that clearly shows that the reviewer missed the point of
the paper. Since reviews are signed in our system, there is an incentive
for reviewers to do a good job.
Benefits to Readers
-
Reviews help create a public assessment system, available
to readers. This is contrast to current conference and journal selection
mechanisms where the end-result is visible to everyone, but the reasoning
is not and the reviews are not. This additional information may be of value.
In particular, where there is diversity of opinion, the reader gets to
know, and can benefit from this knowledge. In contrast, traditional conference
program committees spend the bulk of their effort resolving such differences,
often imperfectly, and then expose only the final decisions.
-
Since reviews are signed, the reader can decide how much
store to set by each review, based both upon the reviewer and the content
of the review itself.
-
The reader’s choice is not abrogated by some editor – the
reader has access to a wide variety of material. For instance, one may
choose to read even mediocre papers in one’s own area of specialization,
while restricting oneself only the most outstanding papers in other sub-areas.
Benefits to Reviewers
-
Papers for review are not assigned by an editor – the reviewer
can choose to review exactly what he or she wishes. In particular, these
are likely to be papers or technical reports that the reviewer would read
any way.
-
The review is written to address the big picture and discuss
the importance of the work being reviewed. In short, the review has the
potential for significant influence. All too often, in the traditional
process, reviewers are forced to spend most of their energy on issues of
detail.
-
Since reviews are public, good reviewers build up a public
reputation. As such, there is a reward for writing good reviews. In contrast,
the traditional review system operates in secrecy, and only an editor gets
to acknowledge reviews that display extraordinary insight or effort.
Of course, with any new effort, there are dangers. The primary
concern is the lack of support for anonymous reviews. This makes it much
harder to write negative reviews. In itself, this is not necessarily
a bad thing, as long as authors have other channels through which to obtain
valuable negative feedback. There is also the fear of reviews being written
in favor of friends or co-authors. A strict conflict-of-interest policy
minimizes this danger. Finally, there is the possibility of reviewers being
coerced, or colluding with authors, or other undesirable behaviors. We’ll
just have to see how the sociology develops around this before taking any
needed measures to avoid or mitigate. Overall, this is an experimental
service that is guaranteed to produce value. The exact ways in which it
will do so depends on what we as a community make of it.
RELATED EFFORTS:
There is no resource available to the database research
community that bears much similarity to ACM-SIGMOD
Digital Review. We point out some of the key differences below.
-
CoRR: The Computing Research Repository is
an on-line repository of technical documents. ACM-SIGMOD
Digital Review is not a repository, but rather a moderated or review-based
guide to the vast amounts of information available. In fact, in large part,
ACM-SIGMOD Digital Review
functions as an optional front-end to CoRR.
-
The Uni-Trier Bibliography: This has become
a critical research tool for members of the database community. However,
its primary function is that of a bibliography (though more recently, electronic
versions of some conference publications are also being made available).
Our intention is to have cross-referencing between ACM-SIGMOD
Digital Review and this bibliography so that a reader browsing the
bibliography can pull up reviews for any paper seen, and also reviews can
reference published articles recorded in the bibliography even if complete
electronic versions are not available on-line.
-
SIGMOD DiSC and Anthology:
These recent SIGMOD efforts make available in digital form vastly more
material than ever before, and thereby make electronic access much more
feasible. However, the focus of these efforts is to make available in electronic
form material that is already available on paper and in traditional forums.
ACM-SIGMOD Digital Review
creates a new electronic review forum. Also, SIGMOD DiSC and Anthology
comprise actual technical articles, just as CoRR does; ACM-SIGMOD
Digital Review has reviews and pointers, but not the technical papers
themselves.
-
ACM TODS and other scholarly publications:
Play a central role in scholarship as we have it today. ACM-SIGMOD
Digital Review is not a journal, even though it may in its own way
play a role in electronic age scholarship.
-
SIGMOD Record: This SIG publication plays a
central role in bonding the database research community. In a limited way,
it provides a forum to comment about important trends or to identify critical
pieces of work. However, this sort of forum is only a very small part of
what SIGMOD Record stands for. ACM-SIGMOD
Digital Review, in contrast, would focus entirely on reviewing scholarly
work. There is no expectation that ACM-SIGMOD
Digital Review would attempt to take over or imitate any of the
central features of SIGMOD Record.
-
ACM Computing Reviews: This journal publishes
reviews of published articles. The reviewers are typically junior people
in the field, and the review typically serves more as an abstract written
by a third party rather than as an evaluation of the work. Furthermore,
in the print medium, it is usually too much trouble for someone browsing
a journal to look up quickly what Computing Reviews has to say, or for
someone to peruse Computing Reviews and then follow "pointers" in print
to selected articles.
ACM-SIGMOD
Digital Review differs in many significant ways. First, it is electronic,
and therefore only a mouse-click away. A potential reader of a specific
article can quickly call up its review, and a person browsing reviews can
click to any full article of interest. (However, Computing Reviews is in
the process of going on-line). Second, by having a significant core of
senior leaders in the field as reviewers, ACM-SIGMOD
Digital Review will have a high credibility with its audience. Third,
there is no limit on the number of reviews for an article -- there may
be none or several depending on how much the piece of work has raised interest.
Thus, it is possible to get multiple points of view for important or controversial
work, while avoiding the need to read reviews that have been written even
though the reviewer did not have a whole lot to say. Fourth, there are
no constraints of publication dates as in a journal -- reviews can appear
as they are generated. One could get relatively immediate reviews, if some
reviewer is quick. One could also get a history of reviews, if the importance
of some article changes with time. Fifth, there is the possibility of a
moderated discussion forum for works of particular controversy. Sixth,
and finally, readers can query for articles that meet specified review
criteria -- such as, articles that received a positive review from Mike
Stonebraker.
CURRENT STATUS:
SIGMOD has generously supported the creation of this new
venture.
ACM-SIGMOD
Digital Review is archived as a "journal" in the Trier
bibliography. Reviews are separated into "issues" based on the date
written. The current "issue" keeps growing each time additional reviews
are archived.
The home page for ACM-SIGMOD
Digital Review is http://www.eecs.umich.edu/digital-review/
Please visit, and enjoy.