Jesse Lawson

buy me a coffee ☕ / GitHub / Mastodon

Dec 7, 2022

Who decides what goes in the training data?

#Ai #Chatgpt

I’m watching people experience ChatGPT, and I remember having that same sense of wonder and excitement the first time I interacted with a really good chat bot and how that experinece was one of the reasons I got really into data science for a while. ChatGPT looks really powerful, like we’re witnessing the birth of an entirely new universe, or perhaps several universes, where the original promise of instant access to information that attracted us to the internet is evolving right before our eyes.

Folks familiar with how algorithmically generated conversations work behind the scenes know that ChatGPT, in addition to being extremely complex, must have consumed an enormous corpus of training data in order to generalize its utility across so many people and their questions. Having consumed more words than any one of us could read in our entire lifetime, we can ask it a question and then complicated mathematical formula are used to derive a response – sometimes even asking follow-up questions.

The tech is nothing short of incredible, but like most technological invention over the past twenty years, there’s a shadow to it that I find myself unable to ignore. It’s not that the technology doesn’t have extreme utility in improving humanity; it’s that something with this much opportunity is, by the very nature of the world in which it has been created and the amount of funding and infrastructure required to produce these human-like experiences, bound to be used to exploit people.

How? By controlling the training data. By deploying models with carefully curated information that serves a specific commercial/political agenda.

If I could convey one thing to the world about these kinds of generative text models, it’s this: You can’t trust what an AI chat bot tells you unless you are intimately familiar with the data used to train it or are a subject-matter expert and are qualified to validate the information it gives you.

Chat bots will, by their very nature, impute a sense of authority in their responses – even if they’re demonstrably wrong by subject-matter experts – in the same way that a textbook conveys a sense of authority to its reader. As illustrated here, this is because of how these systems are designed:

For instance, suppose the human demonstrator is more knowledgeable than the model at common sense: then, the human will ask questions about common sense much less frequently than the model should. However, with BC, the model will ask those questions at the exact same rate as the human, and then because now it has strictly less information than the human, it will have to marginalize over the possible values of the unobserved variables using its prior to be able to imitate the human’s actions. Factoring out the model’s prior over unobserved information, this is equivalent[1] to taking a guess at the remaining relevant info conditioned on all the other info it has (!!!), and then act as confidently as if it had actually observed that info, since that’s how a human would act (since the human really observed that information outside of the episode, but our model has no way of knowing that). This is, needless to say, a really bad thing for safety; we want our models to ask us or otherwise seek out information whenever they don’t know something, not randomly hallucinate facts.

Of course we shouldn’t trust anything we read online unless we can verify it, but let’s be honest: the world is increasingly more difficult to exist in, and while renewed interest in federated social media (I guess we have Elon to thank for that?) has taught and reminded many of the potential to have digital spaces where we can let down our guard against the subtle commercialization and monetization of every aspect of our lives, most people don’t have the knowledge or opportunity to understand what’s going on and why we need to do something about it.

I want to believe that it’s just a matter of education – that if we could only get everyone to be responsible technology users, we could collectively safeguard the authenticity of online human connection and not be influenced by bad actors. In reality, though, I know we’re up against well-funded, extremely sophisticated manipulation tools that are and that will be deployed by unregulated and unscrupulous entities. It will be a digital information war, and we will lose unless we collectively do something.

ChatGPT is more than just a national security issue

Chat bots are already used today for a variety of unethical purposes, from conversational AI used in on a company’s website for customer service and marketing to highly sophisticated disinformation networks of social media accounts attempting to sway public sentiment and even stochastically incite terrorism. As the technology advances, so too does the information battlefield.

ChatGPT, in addition to the commercial exploitation that it’s bound to influence, has profound national security implications. Farms of bots programmatically undermining authentic public discourse and swaying sentiment have been a problem for well over ten years, growing more sophisticated and harder to detect as the technology matures. Meanwhile our government is largely filled with technologically illiterate career contestants in an ongoing nationwide performative populism contest, many of whom were already fully shaped adults with their understanding of the world fully cemented before the first smartphone existed.

We really can’t sit on our hands, here; we ought to be concerned about production use of models in any form that do not have strict oversight rules and accountability of training data – especially when those models are deployed to digital social spaces where we expect human community and interaction. Entire communities will be overrun with seemingly human accounts (just like they are now, just more believable and harder to weed out) deployed by rich people who need to perpetuate myths about the sanctity of the things that make us different (borders, languages, skin color, assigned sex at birth, etc) so that we can never collectively organize around the things that we all have in common.

This is bigger than just national security. It’s not about nations or borders. It’s about privilege and power. Those with the privileges of knowledge and opportunity and the power of money in capitalism will be the ones shaping sentiment and continuing to convince the poor why they should hate each other instead of rise up against the oligarchs. We need international, global, transparent controls over the data used to train ML models and the algorithms through which content, recommendations, and inferences are provided to the general public, otherwise what is bound to happen is commercial interests (which a U.S. president has already admitted are more important than peace) will create massive amounts of pseudo-signal in digital spaces, on the one hand capitalizing on psychological effects of exposure and social proof to continue selling products and political positions, and on the other hand, exponentially stoking and exacerbating terrorism and violence.

It’s not a personal responsibility issue – it’s a community responsibility issue

Strict controls and transparency over training data wont be enough, since the general public is unlikely to ever have the requisite time and energy to inspect the data and recognize when models have been trained for lawful evil purposes and then petition their government for a redress of these grievances in a way that will lead to positive legislative action for healthy digital communities. (I think this task will be relegated to the fringes of society just like it is now, with journalists from big corporate outlets really only interested in these topics as a means of capitalizing on controversy.)

So what do we do? How do we prevent information pollution in digital spaces when commercial interests and state actors have both the means and motive to carry out widespread campaigns of social influence? What happens when a company decides to spend a ton of money to train models with a corpus that was hand-crafted to ensure certain political messages and ideas would underpin the rationality behind a great many conclusions that it draws? Would we need to reconsider how we as people, corporations, and governments treat digital spaces – perhaps considering them as “the means of connectedness” to drive home the distinction between human digital connectedness as a tool for interpersonal communication versus a tool for mass influence? (Is that even possible under our current socioeconomic systems?)

Of course there’s an aspect to this that involves personal accountability and responsibility. We ought not believe everthing we read on the internet without a healthy skepticism, and we ought to validate anything presented as facts if we are going to base our beliefs and opinions off of those facts.

While that may be nice for people to wax poetic about, the reality of most people’s lives is that they don’t have the time, energy, or headspace to dig into whether, for example, something that has gone viral did in fact happen the way it is being presented, and that there’s no deliberate reframing of context in order to fit a particular narrative. People work. People have kids. People have lives. We are all at different phases of our journeys through life and, though we need to practice vigilance when it comes to consuming information online, our lives, our communities, and even our whole world would be a better place if we all could trust that when we are expecting something to have been written by a human, it was in fact written by a fellow human.

I’ve always wondered what would be different if we treated online public spaces like national parks. What would we allow and not allow? What could people count on – and what could they trust (and why) about existing in that space and sharing information with each other?

As these models mature and grow in utility, I don’t find myself cautiously optimistic like I thought I would be. I’m only feeling cautious. I know good people with great imaginations are using ChatGPT and experiencing the same thrill they felt when they first used a cell phone (hey, back in the day it was mind-blowing to be able to call someone from anywhere). On the other hand, I also know that there are bad people with great imaginations using ChatGPT who are seeing an advanced method of exploitation in the name of profit.