Hello everyone!
Welcome to my presentation at virtual CHI 2021.
I am Tazin.
Today, I will present our work on effective interfaces for student-driven revision sessions
for argumentative writing.
We present the design and evaluation of a web-based intelligent writing assistant,
called ArgRewrite, that helps students recognize their revisions of argumentative essays.
It helps students revise their essay by showing them where
and what kinds of revisions they are making.
The goal is to help students write better papers.
We conducted a lab-based Wizard of Oz study with university students.
Students write an essay at home based on provided reading material, and then after few
days revise the essay based on expert feedback on the argumentative elements of the essay.
To understand how our revision assistant can best support students,
we have implemented four versions of our system
with differences in unit span of revision analysis and the level of feedback provided.
Students revise their Draft2 using one of the ArgRewrite conditions,
and finally submit a survey about their experience with the interface.
Now I will describe the interface first,
and then show how we evaluated using the survey and student essay revision.
To start with the interface,
students need to upload their original and revised drafts to the server.
Although in our study the one in charge of data collection did this part.
The System processes the drafts and shows an overview of the
student’s revision process from Draft1 to Draft2.
The ArgRewrite categorizes the revisions made between Draft1
and Draft2 and shows a revision distribution.
It also compares the revisions with the revision map where the left column shows
Draft1 and the right column shows Draft2. Each small tile represents a sentence in the essay.
Students can also click on the sentence or small tiles on the revision map
to see individual revisions.
After observing their previous revision effort,
students process to further revise their essay and create Draft3.
As mentioned before, we have tested four versions of the ArgRewrite interface.
Condition A, where we did not provide any revision categorization or feedback,
this is used as a control condition.
Condition B, with binary revision categorization at sentence-level.
Condition C, with sentence level revision categorization with
six content and three surface level revision categories.
Finally, Condition D with detailed revision categorization like Condition
C but at the sub-sentence level.
To evaluate our research hypothesis that more detailed feedback is more helpful,
we conducted experiments to answer the following research questions.
First, we ask "Do students perceive the ArgRewrite to be clear and easy to use?"
From the perceived ease of use survey questions ANOVA shows conditions A, B,
C, and D are increasingly more difficult to use, where A is significantly easier than D.
Then we ask, "Do students find the ArgRewrite helpful for their writing?"
Perceived usefulness and usage behavior survey question shows C is significantly more helpful
C is significantly more useful than A and B, and D is more useful than A.
Overall we can say that detailed feedback is more useful
than no feedback or binary feedback which supports our hypothesis.
Then we ask, "Is ArgRewrite beneficial for student essay improvement?"
Student's normalized essay score gain shows that
condition C gained significantly more than condition A.
Students in B also gained more than condition A.
But students in D did not gain much compared to B and C.
From this result we can say students found detailed sentence level revision feedback
to be more effective compared to no revision feedback at all.
Finally, we ask, "Is there any difference in student's
revision behavior based on ArgRewrite conditions?"
In A and B, we observe significantly more surface revisions and fewer content
revisions when revising using the system compared to revising without arguing system.
But distribution of types of revision is not significantly different in conditions C and D
In conclusion, our findings consistently showed that feedback on detailed revision
categorization is perceived more helpful than no feedback.
However, we observe an inverse relationship between the granularity
of the feedback and the usability of the system.
We believe we still need to conduct more experiments.
Thank you for attending my talk.
Please let me know if you have any questions.