Income by Degree for Undergraduate Majors

Addendum to this article: I contacted the CEW about these concerns, and information they shared with me is in RED below.

The Wall Street Journal (WSJ) has a database of incomes by undergraduate major here.  The WSJ got the data from the Center on Education and the Workforce (CEW) here, which they compiled from Federal data sources.  I have a few of problems with this data as it is presented.

First, the numbers between the CEW and WSJ sites do not match up.
The WSJ Numbers were different because they were based on 2010 ACS Survey data rather than 2009 ACS data.


Second, the WSJ says that the data used is from the 2010 US Census, which has to be false.  The Census Bureau did not collect any data on earnings or education in the 2010 Census.  Yes, they used to, and their failure to collect this data in 2010 will be seen as a huge mistake in the future, because small area studies relying on Census data can no longer be done.  By small I mean the Census Tract or Block Group Level.  In any case, the study used the 2009 "American Community Survey", which is administered by the Census Bureau.
As above, they used 2010 ACS data, which should be carefully distinguished from 2010 Census Data.

Third, and most important, is that no effort is made to express the accuracy of these numbers.  Let me demonstrate why this is so important.   In order to demonstrate this, I am relying on combining two sources: The data at the CEW, and also a table from the ACS 2009 with summary statistics AND margins of error.  Here is a link to it, but be forewarned it is pretty big.  According to the 2009 ACS, 35,494,367 (+/- 120,221) people in the US have a 4 year degree as their highest education level, out of 201,952,383 people over 25 years old.  Looking here at computers/math majors in the CEW report, they say that 7,829 (+/- not given) people have degrees in "Math and Computer Science" (I am assuming a double major?).

Here comes my point: The ACS in 2009 surveyed 1,917,748 households, and let us suppose that translates into around 3,000,000 people who are 25 and older (around 1.5 people over 25 per household).  That means that surveys covered around 3 (million) out of every 202 (million) people, or 1.5%.  If the ACS surveyed approximately 1.5% of the 7,829 Math and Computer Science majors, the quartiles for income given in the report are based on surveys of around .015*7,829≈117 people.  Of course, we don't know how many people they actually surveyed with this major, but this seems small.  How large might the standard error be for the $98,000 median?  Here come some back-of-the-envelope calculations!  Watch out for LOTS of assumptions!

To simplify things, let us assume that the distribution of salaries for the majors is a normal distribution-- of course this is not true, but this is a rough calculation, after all (and doing a better job requires having the raw data!).  Then, the $75,000 and $134,000 1st and 3rd quartiles would be around 0.67 standard deviations above and below the mean.  These two numbers are 23 and 36 thousand dollars away from the median, respectively (providing evidence of non-normality!), so lets guess that 0.67 standard deviations is in the neighborhood of 30,000, making 1 standard deviation around 30/0.67 ≈$45,000.

Givens:
Sample size around n≈117
s.d. around $45,000
Calculations:
standard error for a mean= 45,000/sqrt(117)≈$4,160
standard error for a median= 1.25*standard error for mean≈$5,200
95%confidence interval for the median earnings = $98,000+/-$10,192  (1.96*5,200)

Now, given that the distribution is not normally distributed, I would bet that the confidence interval would be even wider.  I think that reports like this should give you a clue that some of their estimates have confidence intervals that are over $20,000 wide!
I was told that in a Future, updated report which will merge the 2009 and 2010 ACS data, the CEW plans to issue some guidance about standard errors in an appendix.  I think that reporting sample sizes for each major would largely satisfy my concerns.

Now, to be fair, I am picking on one of the less frequent majors and highest variation majors in the tables... most of them will be better than this.  However, these reports should still at least MENTION that this is sample data, and the numbers are only estimates, and that comparing numbers across majors might be unwise due to errors.  Additionally, these survey forms are typically filled out by one person in the household for all household members-- this introduces other forms of survey error into the mix.  Ask yourself-- how accurately do I know what my spouse makes in a year?  How many thousands might you be off if I asked you on the spot?  For that matter, how well do you know your own?  Or, how accurately would you report your own income? (Copies of the Survey forms can be found here)

Cheers-
Dr. B

Photo Credit: digitalart / FreeDigitalPhotos.net