The vocabulary, where the field came from, what the machines could do before AI — and what genuinely changed when language models arrived.
Every day, several billion people post things in public. Some of it is about your organisation, your industry, or a decision you are about to make.
An entire industry exists to tell you what was said and, increasingly, what it meant. It has a confusing vocabulary — monitoring, listening, intelligence, analytics — often used interchangeably by people selling quite different things.
This is a plain guide to the field: what the words mean, where it came from, what the machines could and couldn’t do, and what genuinely changed when AI arrived.
These get used loosely. The differences matter when you’re buying.
Did we appear? The oldest of the four. Tracks mentions across newspapers, television, radio and online. Answers questions of record: where were we covered, how often, next to what.
What are people saying? Watches public conversation on social platforms and measures it — volume, reach, sentiment, share of voice against competitors. The output is largely quantitative.
What does it mean, and what should we do? Adds interpretation: which narratives are forming, who is driving them, what a shift implies. Usually involves an analyst, whether they work for you or a vendor.
How is this being talked about? Borrowed from linguistics. Less concerned with how many people said something than with how it is framed, which words carry the argument, and how that differs between communities.
Underneath sit a handful of technical terms you’ll meet in every product demo. Sentiment analysis scores text as positive, negative or neutral. Named entity recognition pulls out the people, places and organisations mentioned. Topic modelling groups documents by what they’re about, without being told the topics in advance. Stance detection asks something harder than sentiment — not whether a post is angry, but whether it supports or opposes a specific thing. Share of voice is simply your share of the total conversation in a category.
The business is older than the internet by a century. As cities filled with competing newspapers in the late nineteenth century, press clipping bureaus appeared to read them on your behalf. Henry Romeike’s London operation is the one usually credited, generally dated to 1881, though some sources place an earlier start in 1852. Burrelle’s followed in New York in 1888.
The method stayed unchanged for roughly a hundred years: people read newspapers, found your name, cut out the article, and posted it to you. Everything since has been an attempt to do that faster and at larger scale.
The web broke the model twice. First by moving publication online, where a crawler could do the reading. Then, harder, by letting everyone publish. A clipping bureau tracked a few hundred newspapers. Social platforms produce more text in an hour than those newspapers produced in a year, and most of it is written by people with no editor, in no fixed register, often in dialect.
That volume is what forced the field to become computational. Nobody was going to read it.
For most of the field’s computational history, the tooling was narrow and brittle, and it is worth knowing why — because the limits explain what today’s vendors are still selling against.
Keyword and boolean search came first, and much of the industry still runs on it. Cheap, transparent, and blind to anything phrased unexpectedly.
Bag-of-words models treated a document as a pile of words with the order thrown away, weighted by how distinctive each word was. Enough to cluster documents. Not enough to notice that “not bad” and “bad” differ.
Lexicon-based sentiment scored text against a dictionary of words tagged positive or negative. It was fast, needed no training data, and fell over on negation, sarcasm and context. “Great, another outage” scores positive.
Supervised classifiers were the real step forward. In 2002, Pang, Lee and Vaithyanathan showed that standard machine-learning methods trained on labelled examples beat hand-built word lists at judging sentiment — a paper that shaped the next decade of commercial sentiment analysis. The catch was in the word labelled: the models needed thousands of human-tagged examples, per language and per domain, and a classifier trained on film reviews was useless on airline complaints.
Topic modelling could group documents into themes nobody had specified in advance. Useful for discovery, though the themes came out as word clusters that a human still had to name and interpret.
Taken together, these could tell you how much was being said and roughly in what tone. They could not tell you what was being argued, or why it mattered. And they worked far better in English than anywhere else, because that is where the labelled data was.
Three shifts, roughly a decade apart, moved the field from counting words to reading them.
Word embeddings (from 2013). Instead of treating words as unrelated symbols, models learned to place them in a space where related words sit near each other. Suddenly a system could tell that “furious” and “livid” were close without anyone writing that down. Meaning became geometry — but each word still had exactly one position, so bank meant one thing whether you were fishing or borrowing.
Transformers and BERT (2017–2018). The transformer architecture, introduced in 2017, let models weigh every word against every other word in a passage. BERT applied it to language understanding a year later. Now a word’s representation depended on its sentence. Context stopped being a problem to work around and became the mechanism itself.
This mattered enormously outside English. Arabic-specific models followed — AraBERT, released in early 2020 by Wissam Antoun, Fady Baly and Hazem Hajj, trained on roughly 2.5 billion tokens of Arabic text, and outperforming the multilingual models that had treated Arabic as one language among a hundred. Languages with fewer resources stopped being an afterthought.
Large language models (from 2020, and mainstream from late 2022). The practical break was that you no longer needed a trained model per task. One model could translate a post, identify what it was about, extract who was named, judge its stance and summarise a thousand others — described in plain instructions rather than assembled from labelled datasets. Work that took a research team and a tagging budget became a prompt.
Three things, concretely.
Non-English stopped being second class. The old bottleneck was labelled training data, which existed in English and thinly elsewhere. A model that follows instructions can work in Arabic without anyone hand-tagging ten thousand Arabic examples first.
The unit of analysis got larger. Older systems classified one post at a time. Current models can read a few hundred and describe what is happening across them — which is closer to the question anyone actually had.
Interpretation became automatable — partly. A model can now propose what a narrative means, not just count it. This is genuinely new, and it is also where the field’s new failure mode lives.
Because the other half of the truth is that these systems produce fluent, confident, well-structured output whether or not they are right. A word-count could be wrong but rarely lied convincingly. A language model will explain a narrative that does not exist in a tone indistinguishable from one that does. Anyone buying analysis generated this way should be asking a simple question: can I see the posts each claim came from?
Sarcasm and irony remain unsolved in any language, because the signal is often outside the text entirely.
Dialect and code-switching. Arabic is the clearest case — Gulf, Levantine, Egyptian and Maghrebi communities differ enough that merging them produces an average describing nobody. People also switch between formal and dialect mid-sentence, and write Arabic in Latin characters.
Knowing what is missing. Every tool reports on what it collected. None can tell you about the conversation happening in a closed group, a WhatsApp thread, or a platform you are not monitoring. Coverage limits are rarely printed on the dashboard.
Volume is not importance. A thousand identical reposts and a thousand people independently reaching the same conclusion look similar in a chart and mean entirely different things.
Strip away the category names and there are four recurring reasons.
Something is going wrong and you found out late. The classic driver. A story forms overnight in a language nobody internally reads, and the first anyone hears of it is a journalist calling.
You need to know whether a decision landed. A campaign, a price change, a public statement. The internal view of how it went is rarely the audience’s view.
Your audience isn’t one audience. Particularly across regions, where the same message can be well received in one place and read as an insult in another. That divergence is invisible in an aggregate score.
Someone senior asked a question you cannot answer from internal data. Why do people think this about us? Who is driving the criticism? What changed last month? These aren’t analytics questions; they’re research questions that happen to use public data.
The first two are usually measurement problems, and the platforms handle them well. The second two are interpretation problems, and they need somebody — human or model, but supervised — to read.
If you are at the stage of comparing vendors, we have written a separate guide on five Arabic social media intelligence platforms and what each one actually sells.
What is social listening?
Social listening is the practice of tracking public conversation on social platforms to measure what is being said about a brand, topic or organisation. It typically reports volume, reach, sentiment and share of voice, and is usually delivered as software you operate yourself.
What is the difference between social listening and social media monitoring?
Monitoring is about detection — finding mentions as they appear, one at a time. Listening is about aggregation — measuring patterns across many mentions over time. In practice vendors use the terms interchangeably, so it is worth asking which one a product actually does.
What is social media intelligence?
Social media intelligence adds interpretation to measurement. Rather than reporting that mentions rose 40%, it addresses which narratives formed, who carried them, and what the shift implies for a decision. It normally involves an analyst, whether employed by you or by a vendor.
What is sentiment analysis, and how accurate is it?
Sentiment analysis classifies text as positive, negative or neutral. Modern approaches are considerably better than the dictionary-based methods used through the 2000s, but sarcasm, negation and context still cause errors, and accuracy varies by language and subject. Vendor accuracy figures are usually self-reported and measured on their own test data, so they are best read as positioning rather than a benchmark.
How has AI changed social listening?
The biggest change is that a single model can now handle translation, classification, entity extraction and summarisation without task-specific training data, which particularly benefits languages other than English. The trade-off is that these systems generate fluent, confident output whether or not it is correct, so being able to trace any claim back to the posts behind it matters more than it used to.
Historical and technical claims above draw on: Pang, Lee and Vaithyanathan, “Thumbs up? Sentiment Classification using Machine Learning Techniques” (EMNLP 2002); Antoun, Baly and Hajj, “AraBERT: Transformer-based Model for Arabic Language Understanding” (2020); and standard reference material on the history of press clipping services, where the founding date of Romeike’s London bureau is given variously as 1852 or, more commonly, 1881.