You won’t be around any sort of professional circle, social media, or talent network long without hearing about the extreme talent shortfalls for data analytics, data science, data engineering, and data analysis along with related skill sets in math, statistics, computer science, and quantitative business and social science. This leads one to wonder to what extent this talent shortfall is true, what is causing it, and who is responsible.

Generally speaking, the so-called talent shortfall in (but not restricted to) data related occupations is self-created by employers and/or inflicted by their current reliance on the university system for employment credentialing. We live in a society with one of the most sought after university systems in the world, spend more and more money on education every year, have online access to virtually unlimited training materials, yet somehow experience, as some would have us believe, an unsurmountable skills shortage even in the occupations most desired by employees and rewarded by employers.

It’s well known that loyalty between companies and employees has been in decline for decades. Many companies realized they could increase short term profits by making layoffs after every blip in the stock market and by poaching employees from other firms instead of investing in their own. This situation has spiralled to the point that companies refuse to train employees in marketable skills or even allow them to perform to their potential in fear that they will simply take their new found talent elsewhere. Most employees have responded to this treatment by obliging these predictions. If employers do train their employees anymore, it’s usually in something like ‘leadership’ or ‘communication skills’ as opposed to the skills that are marketable and which they claim to need.

The current refusal of employers to invest in employees ignores the fact that if all companies do this there will be no one left to hire. This approach trades the long term ability to find skilled talent to prevent the short term loss of losing a trained employee. Companies would be far better off if they invested in their employees.

That said, modern employees must learn to invest in themselves. Many have, and the opportunities in terms of online training abound. Nowadays, you can train in almost any skill online for low cost or even free. That includes material found in almost any university education program. Unfortunately, despite the very large number of employees educated in non-university online programs, employers still claim to have a skill shortage all while many of these employees remain unhired and under-utilized. There are at least two reasons for this.

First, while university education is correlated with many unintented factors (see below), it is also correlated with many desirable attributes such as intelligence, productivity, ability to meet deadlines, and ability to follow instructions. It’s well known in economics that the type of people with a university education would likely have been more productive than those without even if they had never attended college. Generally speaking, completing online training programs is thought less well correlated with natural ability than university education.

The second reason that online education has muted impact is due to resistance from those more formally qualified. Few things are derided to the same extent as free/inexpensive online training programs. It’s important to understand that these programs reduce barriers to entry into fields in which some people currently enjoy heavily inflated wages and benefits.

I want to continue the thought from the previous paragraph. In many ways the skills shortage is being manufactured by those already in that field. To a large extent, the people who we depend upon to determine what is really required to complete a job are the same people who directly benefit from labor supply shortages. What you often find are job descriptions with ridiculous requirements compared to what is actually done in the job followed by claims after hundreds of applications that the job can’t be filled.

While the employer based explanations account for much of the so called skill shortages we see today, I’d be remiss to ignore the role higher education has in qualifying individuals for data related occupations. Universities are great marketers that have gained an inflated position in our society. Most people could quickly recite the names of a dozen universities in their state, but would probably struggle to remember the name of a single trade school.

It’s ironic that academics are famous for lecturing the rest of the society about social equality. Yet you’d be hard pressed to find a more unequal institution than higher education. My undergraduate alma mattter recently announced it would provide free tuition to any in state student from a family earning approximately less than the state median income. In doing so, they admitted that only 12% of their student population meets this criteria. Note that this isn’t an expensive private liberal arts school or ivy league university I’m talking about here. A public land grant university enrolls students overwhelmingly from more affluent families.

Now that we have decided that all children from well-to-do families should have a college degree regardless of their academic talent, college costs have exploded. Student debt has increased wildly especially amongst students not fortunate enough to have parents footing the bill. Many of these students are also less likely to achieve the promised higher paying jobs and struggle to pay this debt.

Today, society is addressing the issue of bloated student debt which universities greedily helped create. Studies have shown that universities increased tuition and costs in lockstep with increased student loan availability essentially viewing student loans as a way for them to make money. And universities have found many ways to make as much money as possible, for instance recreation, room, board, and meals. Even working adults would consider it an expensive luxury to eat out three meals a day every day. Yet first year or, now, even second year students are required to live on campus and do just that with a ridiculous and expensive buffet meal plan, omelette bars, and all you can eat ice cream.

Another way universities make a lot of money is foreign students. Foreign students attend U.S. universities despite higher tuition not only because U.S. universities have a good reputation but because they know this is probably the best shot they have at immigrating to the U.S. To be clear, we are a nation of immigrants and, within reasonable numbers, we benefit by allowing intelligent, loyal, and non-hostile foreign nationals immigrate here. However, I don’t believe that immigrants should be the solutation to self-inflicted skill shortages. It’s interesting that so many facets of the so-called skill shortages fall back to universities making more money.

While colleges earn more and more, their degrees mean less and less. Now that everyone who can afford college gets a degree, departments inflate grades so weak and uncommitted students can pass. Professors themselves are rarely hired, promoted, or evaluated based on true teaching ability. Rather they are educated in PhD programs and admired for their research prowess. Ironically, students taught by these faculty frequently fare better only because the alternative is being taught by desperate and underpaid lecturers or overworked graduate students.

Finally, universities have grown further and further detached from reality and marketable job skills. It’s not uncommon to find departments where far left ideologies are the norm and actively forced upon students and others. Dissenting opinions or really voices of reason are shouted down at public speaking events. I do believe in academic freedom and dissenting voices, but it’s instructive that some departments have more in common with the CCP than U.S. society. Frankly, while the propensity of universities to grant degrees in extremist ideology might be a low hanging criticsm, it is a specific symptom of the detachment of higher education curriculum from the economy and marketable job skills. It’s worth mentioning that most university instructurs have never had a real non-academic job in their entire lives.

So what has caused the skill shortage in data related occupations? For starters, an archaic university credentialing system that caters overwhelmingly to children from affluent families, suffers from spiraling costs, and is detached from the society and workplace it is training people for. Employers over rely on this credentialing system and refuse to train employees themselves in fear that a competitor will benefit from their investment. Then, when employees invest in themselves, much of the readily available online and non-university training is heavily discounted by those already an insider to the industry. The solution is to either fix or partially abandon the university system, invest in employees, and become more open minded on training sources.