More on Fuchs & Woessmann

Wow, what a great debate!

First David Wiley commented a study whose abstract is

We estimate the relationship between students’ educational achievement and the availability and use of computers at home and at school in the international student-level PISA database. Bivariate analyses show a positive correlation between student achievement and the availability of computers both at home and at schools. However, once we control extensively for family background and school characteristics, the relationship gets negative for home computers and insignificant for school computers. Thus, the mere availability of computers at home seems to distract students from effective learning. But measures of computer use for education and communication at home show a positive conditional relationship with student achievement. The conditional relationship between student achievement and computer and internet use at school has an inverted U-shape, which may reflect either ability bias combined with negative effects of computerized instruction or a low optimal level of computerized instruction.

[my emphasis]

David questioned the methodology and concluded that

Yet another example of people with an agenda “doing research.” What an embarassment. No wonder educational research is completely discredited in the popular mind.

One of his reader commented that David has "an agenda, and not the authors".

David responded with the post More on Fuchs & Woessmann I am opining here.

First thing that interests me is that the Fuchs & Woessmann's paper is based on the data from the PISA 2000 studies. It would be interesting to run the same statistical analysis on the now available PISA 2003 data to see if the same can be concluded.

Fuchs & Woessmann's paper is a pure statistical exercise, basing data from another study and hence the authors have no control of how the initial data was gathered and all the associated assumptions being taken. David's comment is generally applicable to most of such "research", drawing conclusion from statistical correlation: without looking deeply into whether we can establish a causal relationship between the parameters under the study. With that in mind, it is prefectly correct for David to say

Now, let's pretend we are examing a group of 15 year olds, some of whom spend none of their educational and recreational time on computers, some of whom spend some of their educational and recreational time on computers, and some of whom spend much of their educational time and recreational time on computers. Now, let’s imagine two scenarios in which we might measure the academic achievement of a sample of this group of 15 year olds. In one scenario, we will carry out the assessment on computers. In the other scenario, we will carry out the assessment in "more traditional" manner. Can we not form a strong hypothesis, ahead of data collection or analysis, about which sub-group will perform best in each scenario?

Methodology is one thing. But, let's consider something else.

Some sample items from the 2000 studies are available here. From the first reading unit (page 32 of the pdf file), two figures are shown: first about the depth of Lake Chad and second about kind of animals in the rock arts, both plotted along a timeline. 5 items are associated to Lake Chad: first 2 of which test the 15 years old ability to read the first figure, the third item is "Why has the author chosen to start the graph at this point?" which is checking whether the testee has read the introduction to the figures. Question 4 is:

Figure B is based on the assumption that
A. the animals in the rock art were present in the area at the time they were drawn.
B. the artists who drew the animals were highly skilled.
C. the artists who draw the animals were able to travel widely.
D. there was no attempt to domesticate the animals which were depicted in the rock art.

This particular item assumes the 15 years old know what is a rock art.

Here is another sample (page 95 of the pdf) about "speed of a racing car". There is a graph showing how the speed of a racing car varies along a flat 3 kilometre track during its second lap.

The first multiple choice item related to "speed of racing car" is

What is the approximate distance from the starting line to the beginning of the longest straight section of the track?

and so on...

This set of questions would have trapped me! With the benefit of reading the answer, I noticed that this is an unusual graph. Most Physics textbook would plot speed against time and I would have find the area under the curve to find the distance traveled. Well, the graph is speed along distance. Hence we only need to be able to deduce that the race car will decrease its speed only because it is turning a corner and hence find the points of lowest speed to determine whether the corners are.

After looking at the sample items, I found that these are quite demanding items. Those who can stay focussed for 2 hours continuously in a desk obviously will perform better than digital natives who used to multi-task, short attention span and always used to have, at their finger tips, supports from peers from a distance and information whenever they need.

My question would be to question whether these digital natives, given their characteristics would be able to meet the challenge in 2010 when they have to contribute to real economic productivity; or, are the items really testing the required skills these digital natives would need to be productive?

Methodology is only part of the bigger debate!