|
Welcome to Import AI, a newsletter about artificial intelligence. Forward this email to give your chums an AI upgrade. Subscribe here.
Oh hooray, the BERT-generated headlines are here:
...Being able to search over text is cool, but do you know what's cooler? Clickable headlines…
Researchers with Amazon and german publisher Axel Springer have built an AI tool that uses BERT to generate search engine optimization (SEO)-friendly headlines. The system uses recent advances in natural language processing to make it cheaper and easier to generate a bunch of different headlines that editors can then select from. "By recommending search engine optimized titles to the editors, we aim to accelerate the production of articles and increase organic search traffic", the authors write.
How to use AI to generate an SEA headline: They build the headlines out of two main priors - a one sentence length-constrained summary of the article, and a set of keywords that relate to the text and are expected to rank well on Google. The described system combines these two bits of information to generate short, descriptive, keyword-filled headlines. To help them, they train a BERT-style summarization model (named BERTSUMABS) on ~500,000 articles from Axel Springer publication 'WELT'.
How well do humans think it works? In tests, human experts said "the German BERTSUMABS generates articles with mostly correct grammar and which rarely contain false information". The system still has some problems - it can't easily avoid grammatical mistakes or outputting misleading and false information (though has that ever stopped the media industry? Kidding! - Ed).
Why this matters: The internet is an ecology of different, interacting systems. For the past few decades, the Internet has mostly been made up of humans and expert systems designed by humans, though smarter AI tools have been used increasingly to filter and catalog this system. Now, the Internet is becoming an ecology containing multiple content-generating systems which are more autonomous and inscrutable than the things we had in the past. We're heading towards a future where ML-optimized publications will generate ML-catnip headlines for ML-based classifiers which feed into ML-based recommenders that then try to tempt human-based eyeballs towards some content. The effects of this increasingly cybernetic ecology will be bizarre and emergent.
Read more: DeepTitle -- Leveraging BERT to generate Search Engine Optimized Headlines (arXiv).
####################################################
Massive pre-training comes for reinforcement learning - and the results are impressive:
...DeepMind shows that RL agents can (somewhat) generalize via pre-training…
Pre-training - where you train a network on a massive dataset to improve generalization on downstream tasks - is a powerful, simple concept in AI. Pre-training techniques have, in recent years, massively improved the performance of computer vision systems and, notoriously, NLP systems such as GPT-2 and GPT-3. Now, researchers with DeepMind have found a way to get pre-training to work for reinforcement learning agents - and the results are impressive.
What they did: Simulations, curriculums, and more: The key here is 'XLand', which is basically a programmatically specifiable game engine. XLand lets DeepMind automatically compose different types of games in different types of terrain within the simulator - think of this as domain randomization/synthetic data but for gameworlds, rather than static data such as variations on real images or audio streams. DeepMind then uses population-based training to feed different Rl agents different games to play and it basically breeds successively smarter agents via distilling good ones into the next generation. "The agent’s capabilities improve iteratively as a response to the challenges that arise in training, with the learning process continually refining the training tasks so the agent never stops learning," DeepMind writes.
Generalization: XLand, combined with PBT, combined with relatively simple agents, means DeepMind is able to create agents that can succeed on tasks they've never seen before, such as object-finding challenges to "complex games like hide and seek and capture the flag". Most intriguing, they "find the agent exhibits general, heuristic behaviours such as experimentation, behaviours that are widely applicable to many tasks rather than specialised to an individual task". Now, this isn't full generalization (after all, the agents are only shown to generalize within the bounds of the unseen games within the same simulator), but it is impressive. It also suggests that we might start to see more progress in reinforcement learning, as being able to do massive pre-training gives us a way to build more capable agents.
AI history trivia - Universe: A few years ago, OpenAI had a similar idea via 'OpenAI Universe', which sought to train RL agents on a massive distribution of games (predominantly 2D flash games gathered on the Internet). The implementation details were quite different, but it gestured at the ideas present in this work from DeepMind. My sense is that one of the important differences here is the use of a simulator which lets DeepMind have a tighter link between the agent and its environment (whereas Universe had to simulate the games within virtual browsers), as well as the usage of slightly more complex RL agents with a greater ability to attend over internal states with regard to goals.
Why this matters: As one of Shakespeare's characters once said regarding the magic of subjective consciousness: "I could be bounded in a nutshell and call myself a king of infinite space" - who is to say that XLand isn't a nutshell and that within it DeepMind has started to create agents that have a sense of how to get things done and experiment within this bounded universe. Obviously, we're quite far away from the agents being able to deliver soliloquies about their experience, but it's an open question as to where the general behaviors exhibited here top out.
Read more: Generally capable agents emerge from open-ended play (DeepMind blog).
Read the paper here: Open-Ended Learning Leads to Generally Capable Agents (DeepMind).
####################################################
Want to close the compute gap between the private and public sector in the USA? Respond to this RFI:
...Help the National AI Research Resource do intelligent, useful things…
In recent years, power differences in AI development have caused a bunch of problems - put simply, a small number of AI developers have immense financial resources which they've used to spend big on computation which they've used to drive new frontier results (e.g, AlphaGo, GPT-3, pre-training image systems on billions of images, etc). This has been useful for developing capabilities, but it has also furthered information asymmetries that exist between a small number of private sector actors and the rest of the world.
Now, a new project in the Biden administration wants to change this by bringing people together to think about building a National AI Research Resource (NAIRR). The point of the NAIRR is to "democratize access to the cyberinfrastructure that fuels AI research and development". There's also a recently formed taskforce whose task is to "investigate the feasibility and advisability of establishing and sustaining a NAIRR and propose a roadmap detailing how such a resource should be established and sustained".
Now, the government wants the help of other interested parties to build out a sensible NAIRR and has published an RFI seeking expert input. The RFI asks questions like which capabilities should be prioritized within the NAIRR, how the NAIRR can be used to reinforce principles of ethical and response research, and more.
Deadline: September 1st, 2021.
Why this matters: Societies tend to be more robust if power is distributed more evenly through them. Right now, power is distributed inequitably within the AI ecosystem. By developing things like a NAIRR, the US has an opportunity to create shared compute-heavy infrastructure that others can access. It'd be good for interested members of the AI community to contribute ideas in response to this RFI, as the better the NAIRR is, the more robust the AI ecosystem will become.
Read more: Request for Information (RFI) on an Implementation Plan for a National Artificial Intelligence Research Resource (Federal Register).
####################################################
Industrial robots + Intelligence = $80 million in new funding:
...Covariant raises a whopping Series B…
Covariant, a startup that aims to combine recent advances in deep learning and reinforcement learning with industrial robots, has raised $80m in new funding, bringing its total raises to $147 million. The company makes AI tools that can help robots do tasks as varied as pick-and-place, induction, and sorting. Covariant's president is Pieter Abeel, a professor of robotics at UC Berkeley, and its CEO is Peter Chen - much of Covariant's founding team is ex-OpenAI (disclaimer: I used to work with them. Very nice people!).Why this matters:Last week, we wrote about Google spinning out an industrial robots startup called Intrinsic. This week, we've got some investors flinging a ton of money at Covariant. These things add further evidence to the idea that industrial robots are about to get a lot smarter - if they do, that'll have big effects on factory automation, and could lead to the emergence of entirely new AI-robot capabilities as well.
Read more: Robotic AI firm Covariant raises another $80 million (TechCrunch).
####################################################
Aleph Alpha raises $27m for European AI:
...New money for nationalistic AI...
Aleph Alpha, a European AI startup, has raised €23 Million ($27M) to help it "build Europe's largest, sovereign AI language models". The idea behind Aleph Alpha is "to establish a European alternative to OpenAI and the Beijing Academy of AI (BAAI) and establish a globally leading AI-research institution with European values at its core," writes one of the venture capital firms that invested in the company.
Aleph Alpha + Eleuther: Aleph Alpha also hired a bunch of people connected to Eleuther, the open source cyberpunk AI collective. Eleuther has made some nice GPT-style models including GPT-J, a 6billion parameter code&language model (Import AI 253).
Why this matters - multi-polar AI: We live in an era of multi-polarity in the AI ecosystem; after several years of centralization (e.g, the growth of DeepMind and OpenAI relative to other startups), it feels like we're entering an era that'll be defined by the proliferation of new AI actors and capabilities - some of them with nationalistic flavors, like Aleph Alpha. Other recent AI startups include Cohere, Anthropic, and - as mentioned - Eleuther. It's an open question as to whether the shift into a multi-polar era will make it harder or easier for AI developers to coordinate.
Read more: Twitter announcement (Aleph Alpha Twitter).
Find out more at the official Aleph Alpha website.
Read about why one of the VC firms invested:Europe's shot for Artificial General Intelligence - Why we invested in Aleph Alpha (Medium).
####################################################
Tech Tales:
Arch-AI-ologists
Now we've all been on Internet 5 for a few years now, and Internet 4 before that, and so on. No one ever spends time on the original Internet, if they can help it. It's mostly spam now. Spam and the chattering of old machines and old viruses.
But there are some people that still use it: the Arch-AI-ologists, or Arch-AIs; software programs we built to try and find some of the original 'content' that was used to train some of the AI systems around us. We build these agents and then we send them out to find the memories that defined their forebears.
There are some people that say this is unethical - they compare it to deep sea fishing, or interference in a foreign ecology. Some people claim that we should preserve the old Internet as a living, breathing ecology full of spambots and AIs and counter-AIs. Other people say we should mine it for what is useful and then be done with it - erase as much of it as we can and slowly and painfiully replace the dependencies.
We haven't thought to ask our Arch-AI-ologists about this, yet. Instead some of us try to celebrate their findings: here's an original 'grumpy cat' meme found on an old server by one of the Arch-AIs. Here is a cache of vintage Vine movies that someone found on someone's dormant profile on a dead social network. And here is an early meme about a robot that wants to forget its bad memories and so it holds a magnet against its head until it falls asleep. "Magnets, alcohol for robots" is one of the captions.
Things that inspired this story: How memory and representation interplace in neural networks; generative models; agent-based models; thinking about the internet as an ecology.
Thanks for reading. If you have suggestions, comments or other thoughts you can reach me at jack@jack-clark.net or tweet at me@jackclarksf
|
