Assessments of ACC Library Services
Information Literacy Programs

Survey of all faculty done as part of the college's Ad-hoc Committee on Information Literacy, 2001. (PDF document. Go to Appendix A, p. 28)

Survey of Composition I and II faculty on possible changes in our instruction efforts, 1998

Survey of students in Introduction to Business as part of pilot for online tutorial, The Info Game, Fall 2000.

Survey of all Composition I faculty following the first semester of their adoption of The Info Game, December 2001

Survey of students in four sections of Introduction to Sociology after completing pilot of course-integrated research assignment, December 2001.

Analysis of Composition I research papers, Jan. 2003.

Analysis of effectiveness of Info Game online tutorial, 2003.

SURVEY OF COMP I AND COMP II FACULTY ON POSSIBLE CHANGES IN OUR INSTRUCTION EFFORTS, 1998

Seventeen faculty responded. Two only filled out Comp I portion. The responses are in BOLD. In some cases I paraphrased written responses. Anyone, of course, is welcome to look at the originals.

- Red
9/28/1998

LRS Survey of ENG 1613 (Composition I) and
ENG 1623 (Composition II) Faculty

Comp I

1. As part of their research paper, do you ask students to use the Internet/World Wide Web?

• No • OK to use • Require use
ONE TWELVE FOUR

2. As part of the A-paper, do you allow students to use a companion essay found on the Internet/WWW?

• No • Yes
FOUR TWELVE ONE NO RESPONSE

3. Do a significant number of students need help locating the companion essay?

• No • Yes
SIX SIX TWO RESPONDED "UNKNOWN" THREE NO RESPONSE

4. If you think some sort of formal help in locating the companion essay would be useful, should it be:

• simply a descriptive handout of techniques and sources, OR SIX

• a fill-in-the-blanks assignment that leads them through the process, OR FOUR

• something else "Offer several choices [of instruction]" "Hands on walkthrough" "In class presentation or PowerPoint" "Both of the above"

Comp II

5. For the A-paper option that requires documented critical analysis, do a significant number of students need help locating literary criticism?

• No • Yes
FOUR TEN ONE "UNKNOWN"

6. If you think some sort of formal help in locating literary criticism would be useful, should it be:

• simply a descriptive handout of techniques and sources, OR EIGHT

• a fill-in-the-blanks assignment that leads them through the process, OR FOUR

• something else "Offer several choices of instruction" "Both kinds"

7. Any comments on how to improve or expand existing LRS services for the Composition classes?

"Longer hours at CYP library"
"Need video to show in class"
"Get O.E.D. online"
"Handout about using Internet sources with more on bibliographic entries for online sources than current MLA handout"
"More literary criticism"
"Library tours and Internet seminars"

RESULTS OF COMP I FACULTY SURVEY CONDUCTED DEC. 2001 TO EVALUATE "THE INFO GAME"

Eighteen surveys were returned. These results distributed to the Comp I faculty, the library Committee on Information Literacy, and the Dean of Library Services.

(The use of first person below refers to Red Wassenich, chair of the Committee on Information Literacy.)

Question 1: Approximately how many semesters have you taught Comp I?

Responses ranged from 2 to 70 semesters.

Question 2: Did you actually have your students do "The Info Game"?

Yes = 17 No = 1

Question 3: How much of a problem were the quizzes?

No problem = 1 Minor problem = 6 Somewhat of a problem = 8 Big problem = 2

Summary of comments: Obviously we had some serious glitches pop up, so this got a bad result. I think we have cleared up the process and there should be many fewer problems. I apologize for the situation this last semester. Several said it would be nice if the results of the four quizzes could be combined into one document. I will see if there's any way to do this.

Question 4: Concerning advanced warning on this new initiative, a memo was sent to you along with an info sheet for students, and "The Info Game" was mentioned in the cover letter accompanying the blue paper worksheets. Was this adequate?

Yes = 13 No = 3

Summary of comments: More info should be given to students at the start on how to access all the modules from off-campus, i.e., the need for a password to get to some sections

Question 5: Generally, did you notice any change in students' work on the research paper from previous semesters?

Better work = 2 Same work = 12 Worse work = 1

Summary of comments: Few comments. Nothing consistent. The one who said the work was worse said that was due to their lack of skill in documentation, which the Info Game only cursorily mentions. (See final comments at end.)

Question 5: How do you think students generally saw "The Info Game"?

Very useful = 1 Somewhat useful = 8 Neutral = 2 Fairly useless = 2 Very useless = 0

Summary of comments: Several said it was really impossible to answer this because it varied among individuals so much. Many students already knew portions, but there were some who didn't. Ways for the more knowledgeable to skip the basics would help. (See comments at end.)

Question 6: When you or students had questions about "The Info Game," were they addressed adequately?

Yes = 10 No = 1 Not applicable =2

Summary of comments: Few comments. No particular trend to report.

Question 7: Has the addition of "The Info Game" made it possible to delete the initial portions of the blue worksheets that discuss different types of sources and search techniques (pages 2 - 4)?

Yes = 6 No = 9

Summary of comments: Many said the hard copy is needed for both use in the classroom and by the students after they do the Info Game. So the handouts will not be reduced.

Question 8: Any overall comments on "The Info Game," the blue worksheets, or other library-related issues?

Summary of comments:

§ Several mentioned ongoing problems with MLA documentation. My response: Since the Info Game is used in other courses besides Comp I, we didn't go into any detail other than to say documentation is important and giving the URL for our online versions of the handouts. The paper version of the MLA handout has been expanded to 4 pages, but you should be aware that the web version has more examples, since space isn't an issue. Comp I instructors approach documentation in very different ways: some are exacting; some are satisfied with the students grasping the basic concept. Perhaps the task force may want to appoint a small committee to work with me to improve the handout.

§ Some mentioned again the need to make clearer how to get a password to access some online resources. I will add this to the promotional material I send.

§ Some commented that the Info Game led to better questions in classroom discussions.

My general comments: I really appreciate so many answering the survey, not to mention the department's adoption of using the Info Game. Given it was the first semester, I think it went pretty well.

From the results I'd say there is a general endorsement, but no great leap forward. No question got a really negative response except the one on quizzes.

I am working on ways to try to add a method of allowing students to "test out" of some introductory material. One of my concerns is that many students say they already know how to use online information, but in fact they don't. As with all of us, it can be difficult to know that you don't know something.

As always, I welcome your ongoing suggestions and comments.

Red Wassenich
RGC Library
223-3074
redwass@austincc.edu

(Posted March 6, 2002)

EVALUATION BY STUDENTS IN INTRODUCTION TO BUSINESS OF "THE INFO GAME" TUTORIAL, FALL 2000

One section of instructor Charles Quinn did this evaluation during the fall 2000 semester. They took the same survey at the beginning and at the end of the semester.

Pretest had 25 responses. Posttest had 15 responses.

ANALYSIS

Admittedly, the samples are fairly small so very reliable judgment is probably not valid. However the results given below were consistent in both the summer and fall groups, so they may be reliable to some degree.

I see mixed results but with definite progress.

Not too surprisingly, the students usage and self-description of their abilities rose (questions 1 and 2).

Their knowledge of what a periodical index and how to best phrase a search-engine query definitely improved (questions 3 and 4).

Question 5 showed a small increase in the correct answer but a significant number still got it wrong. The realization of the difference between the "free" web information and the databases we subscribe to is one that students have trouble grasping. I really stressed this in the modules, so I'm a bit discouraged they didn't do better here.

Question 6 doesn't have an absolute "correct" answer (and some students may not know what "objective" information means), but this one actually showed a decrease in the best answer (periodical articles). Evaluation skills are the hardest to get across--and arguably the most important. (The rubric analysis of the research papers showed a similar inability to distinguish objective and authoritative sources.) I will try to add some emphasis to the web module on this.

- Red Wassenich
RGC Library
223-3074
redwass@austincc.edu

Posted March 6, 2002

EVALUATION BY INTRODUCTION TO SOCIOLOGY STUDENTS OF PILOT RESEARCH PROJECT

December 2001

Deb Zeigler piloted an information literacy project in her four Intro to Sociology sections in the fall 2001 semester, two at RGC, two at NRG. The project involves looking up a topic in a specialized encyclopedia, finding articles (making the distinction between mags and journals), and writing an annotated bibliography. The project was designed in collaboration with librarians, but no class presentation was given.

Zeigler designed and gave the students the evaluation form. About 75 students filled it out.

They were asked how much they knew about various things before and after the project and given the choices "Nothing," "A little," "Some," A lot." The results below give the before and after responses separated by >.

The existence of specialized encyclopedias.

Before > After: Nothing 7 > 2 A little 31 > 5 Some 32 > 33 A lot 6 > 36

Existence of electronic databases for accessing articles.

Nothing 16 > 4 A little 21 > 4 Some 25 > 21 A lot 15 > 48

Differences between magazines and journals.

Nothing 16 > 3 A little 24 > 12 Some 22 > 28 A lot 15 > 36

How to evaluate the reliability of an article.

Nothing 16 > 2 A little 30 > 12 Some 24 > 26 A lot 7 > 37

How to do a citation of an article.

Nothing 20 > 2 A little 17 > 8 Some 26 > 18 A lot 14 > 44

When asked if they would recommend this project be included in future sections of the course, 68 gave positive responses, 4 negative, and 6 were neutral.

- Red Wassenich, January 2002

Posted March 6, 2002

EVALUATION OF COMPOSITION I PAPERS - OVERVIEW

Jan. 2003

Red Wassenich

The Composition faculty annually reviews student papers in Comp I and II to both evaluate the quality of work the students are doing and to assure that faculty are grading on a reasonably similar scale (two faculty grade each paper and if the scores are notably different, a third grades it).

In Jan. 2003, I participated in the process, although I used a separate rubric focusing on the information literacy skills taught through the Info Game web tutorial and the blue paper "Research Guide." I read 21 Comp I research papers (required of all Comp I students).

I used 7 criteria, which are explained below. The papers were scored on a 1 - 3 point scale for each criterion.

In addition to the formal analysis on the accompanying document, the following thoughts occurred to me during the process:

· The papers were longer than I expected. Of the 21 I read, only one or two was 3-4 pages; most were 5-8 pages.

· There was much more variety to the topics than I expected. There was only one on abortion and one on legalizing marijuana. There were papers on the role of Catholicism in American politics; how women are perceived in Buddhism and Hinduism; pet neutering; history of the Marine Corps, and several other individualistic topics. A few of the papers were really good.

· I saw no instances that looked like plagiarism.

· The Comp faculty, in a brief informal discussion at the end, expressed frustration with the research papers and want to brainstorm possible improvements.

· As frustrating as many of the results of my analysis were, it's important to remember that these are quite young, inexperienced writers for the most part. When we were freshmen, most of us didn't have any clue about the sort of information literacy issues we are teaching, so they're at least getting a glimpse of something we probably didn't.

· It is extremely valuable for librarians to read such papers. Most of us do not ever see the students' finished work.

Evaluation of Comp I Research Papers

Seven criteria were used, based on major concepts covered in the Info Game tutorial.

1. Uses multiple research sources

1 - Used zero or one source in paper
2 - Used two or three sources
3 - Used four or more

SCORE: Average = 2.9

COMMENTS: This was a major surprise. Only two papers used three sources; the others used up to 18. The average number of citations was 6.6

POSSIBLE IMPLICATIONS: Not much. Both the Info Game and the Research Guide discuss the value of using multiple sources, but don't spend much time on it, which evidently they don't need to.

2. Uses a variety of research sources

1 - Used only one type of source, e.g., just the WWW or just periodical articles
2 - Used two types
3 - Used three or more types

SCORE: AVERAGE = 2.3

COMMENTS: I was both pleased and perplexed with this aspect. Only two just used one type of source. (The amazing one was the 18 newspaper sources as the only format.) However, only about four of the 21 papers cited periodicals from our subscription databases. Several more cited free online periodicals. Online sources were the great majority of sources; these included reference, periodicals, government documents, and various web sites. No e-books were cited and about three-fourths cited paper books.

POSSIBLE IMPLICATIONS: Greater use of our subscription databases would presumably lead to better research. Since pushing these is a big portion of the Info Game, it's disheartening to see so few citations to them.

3. Employs both background information and specific information.

1 - Shows no use
2 - Shows some use
3 - Shows clear use

SCORE: Average = 2.2

COMMENTS: They were so-so on this. Most people didn't seem to consciously build their topics from the background to the specific, which would mirror our teaching of reference sources-to-periodicals.

POSSIBLE IMPLICATIONS: Again, this is covered in both the Info Game and the Research Guide. Much of this ends up being the Comp faculty's job, it seems to me, so I don't see a lot more we can do.

4. Uses sources appropriate to assignment
(Errors: irrelevant (not on topic), too simple or too advanced, outdated)

1 - None of the sources were appropriate
2 - Some of the sources were appropriate
3 - All sources were appropriate

SCORE: Average = 2.7

COMMENTS: A pleasant surprise. I was expecting lots of irrelevant web sites; there were only a handful (mind you, these are snap judgments based on titles and analyzing the URLs). The web sites cited generally appeared to be pretty focussed on the topics. Again the lack of citations to our databases was disturbing, but the ones they chose were not too bad.

POSSIBLE IMPLICATIONS: Somehow get more use of our databases.

5. Uses credible sources
(The assessment based on general perspective, without tracking them down in most cases.)

1 - None of the sources seemed credible
2 - Some of the sources were credible
3 - All the sources seemed credible

SCORE: Average = 2.5

COMMENTS: This was the hardest to judge, since I couldn't see the sources. Very few of the web sites cited looked really suspect, e.g., a geocities page or something from a K-12 site. On the other hand, much better sources could have been found for most topics, e.g., our databases.

POSSIBLE IMPLICATIONS: Teaching critical evaluation is the hardest part of information literacy. This is stressed in the Info Game but some additional stuff would be nice.

6. Demonstrates understanding of importance of evaluating sources
(This refers to students explicitly discussing sources in terms of reliability. Above two questions deal with whether or not they actually used them.)
Criteria to consider: Did student mention consideration of author credentials, objective vs. subjective info, currency of info, accuracy of info, scholarly level of info

1 - Does not mention any source's reliability
2 - Mentions reliability for some sources
3 - Consistently discusses sources' reliability

SCORE: Average = 1.2

COMMENTS: Not surprisingly, this was very low. It is probably an aspect of writing that only more advanced writers pay much attention to and I don't think that the Comp faculty stresses it.

POSSIBLE IMPLICATIONS: Probably none, realistically.

7. Uses documentation of sources
(Strict adherence to documentation rules is not the point here. Rather did they give enough information to identify the source and demonstrate an understanding of the concept.)

1 - No sources are adequately documented
2 - Some sources are adequately documented
3 - All sources are adequately documented

SCORE: Average = 2.4

COMMENTS: In fact, no one cited all their sources correctly. For the majority, however, it was easy to tell what they meant. Web pages, not surprisingly, were the most incorrect; in particular, what to use as the "title."

POSSIBLE IMPLICATIONS: Maybe tweak the MLA handout to stress the web page-citation more (but there's not much room on the page).

ANALYSIS OF EFFECTIVENESS OF INFO GAME ONLINE TUTORIAL, 2003