Wednesday, AI 101, Technique: Mixture of Depth

Friday, Friday, AI Unicorns: Perplexity (we apologize for the delay with this article – the common cold has hit us hard.)

Master 50 essential Excel tricks with step-by-step GIF tutorials

Create stunning data visualizations using done-for-you graph templates

Learn time-saving shortcuts the pros use daily

Access game-changing formulas and functions explained simply

Mixture-of-Transformers (MoT): A Sparse and Scalable Architecture for Multi-Modal Foundation Models proposed by researchers from Meta and Stanford. MoT architecture is important because it addresses the high computational costs and inefficiencies involved in training large, multi-modal models. Traditional dense models process multiple data types (text, images, speech) in a unified way, which demands significant resources, limits scalability, and complicates training. MoT’s approach introduces sparsity by activating only relevant model components per modality, reducing FLOPs and computational load while maintaining model performance →read the paper

Agent K v1.0: Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level introduced by researchers from Huawei Noah’s Ark and UCL developed Agent K v1.0, an autonomous data science agent that manages the entire data science lifecycle by learning from experience. Agent K v1.0 is important because it automates complex data science tasks, achieving expert-level performance on Kaggle, which shows that LLMs can autonomously handle workflows that typically require skilled human data scientists. This scalability enhances productivity and serves as a benchmark for using AI in high-level problem-solving, demonstrating AI’s potential to learn, adapt, and improve with experience →read the paper

Decoding Dark Matter: Specialized Sparse Autoencoders (SSAEs) for Interpreting Rare Concepts in Foundation Models introduced by researchers from Carnegie Mellon. This research matters because it improves our ability to interpret foundation models (FMs) by capturing rare, domain-specific features that are usually overlooked. These “dark matter” concepts are important for AI safety and fairness, as they can include subtle biases or unintentional behaviors that may otherwise go unnoticed. SSAEs help isolate and control these features, which could lead to fairer models, safer use in specific fields like healthcare, and a clearer understanding of how FMs function →read the paper

Artificial Intelligence, Scientific Discovery, and Product Innovation by Aidan Toner-Rodgers. The key findings reveal that AI-assisted scientists discovered 44% more materials, which led to a 39% increase in patent filings and a 17% rise in downstream product innovation. These discoveries also resulted in novel compounds and radical innovations, with significant effects among high-ability scientists, whose output nearly doubled. However, lower-ability researchers didn’t see a lot of benefits, widening productivity disparities →read the paper

Is it really over for LLMS? A balanced thought piece by Devansh

Microsoft

• Microsoft’s Magentic-One introduces a coordinated team of AI agents like WebSurfer and FileSurfer, handling complex web and file workflows with a safety-first approach →their GitHub

Microsoft and OpenAI

• Medprompt by Microsoft and OpenAI enhances diagnostic accuracy with chain-of-thought reasoning, elevating medical model performance without traditional prompt tuning →read the paper

OpenAI

• Facing slower improvements, OpenAI shifts Orion training to synthetic data, indicating a potential slowing in the industry’s AGI ambitions →The Infromation

• Meanwhile, Sam Altman says AGI arrives in 2025 🚂 →on YouTube

• Good news for OpenAI, it dismissed claims of copyright misuse in a lawsuit, marking a pivotal moment for copyright in generative AI and setting precedents for future disputes →Reuters

• OpenAI’s “Predicted Outputs” feature reduces GPT-4o latency, allowing for quicker responses in fast-paced applications and an overall smoother experience →read their blog

Google

• Gemini is now accessible from the OpenAI Library →read their blog

Defense Llama: Scale AI’s National Security Specialist

• Scale AI’s Defense Llama, a secure Llama 3 variant, supports U.S. defense operations, with capabilities for mission planning and intelligence analysis in high-security settings →read their blog

Department of Defence shows more and more interest

• Jericho Security wins the Pentagon’s first AI contract, using adaptive simulations to combat phishing and deepfake threats – an AI milestone in national defense →VentureBeat

Mistral API Adds Precision to Content Moderation

• Mistral’s Ministral 8B model brings nuanced content moderation, covering nine sensitive categories and diverse languages for a global audience →check their blog

NVIDIA

• NVIDIA expands NeMo with NeMo Curator and Cosmos tokenizers, boosting generative AI development across video, image, and text. Faster data processing and high-quality tokenization mean efficient, high-fidelity visuals for industries like robotics and automotive. Cosmos tokenizers’ 12x speed gain sets a new standard →read their blog

The Semantic Hub Hypothesis proposes a unified semantic processing hub across languages and data types in LLMs, enhancing versatility but embedding potential biases.

Self-Consistency Preference Optimization improves reasoning by preferring consistent responses, boosting zero-shot accuracy without needing labeled data.

Sample-Efficient Alignment For LLMs introduces a sampling-based alignment approach, improving efficiency under limited feedback.

SALSA: Soup-Based Alignment Learning enhances model stability in reinforcement learning through a "model soup" of averaged weights.

Give Me BF16 Or Give Me Death? analyzes quantization formats, balancing accuracy and cost for efficient model deployment.

BitNet a4.8: 4-Bit Activations For 1-Bit LLMs reduces parameter requirements by using 4-bit activations, supporting fast, large-scale deployment.

SPARSING LAW examines neuron sparsity in LLMs, identifying efficient patterns for activation reduction.

Inference Optimal VLMs Need Only One Visual Token shows that fewer visual tokens but larger model size can improve VLM efficiency.

A Systematic Analysis Of Multimodal LLM Data Contamination detects data contamination in multimodal models, highlighting the need for clean datasets.

LLM2CLIP: Language Models Unlock Richer Visual Representation integrates LLMs to enhance multimodal learning, improving cross-lingual retrieval.

WEBRL: Training LLM Web Agents trains web agents with a curriculum that evolves through agent learning, improving task success rates.

DynaSaur: Large Language Agents Beyond Predefined Actions allows agents to create actions on-the-fly, handling unforeseen tasks with Python-based adaptability.

THANOS: Skill-Of-Mind-Infused Agents enhances conversational agents with social skills, improving response accuracy and empathy.

DELIFT: Data Efficient Language Model Instruction Fine-Tuning optimizes fine-tuning by selecting the most informative data, cutting dataset size significantly.

HtmlRAG: HTML Is Better Than Plain Text improves RAG systems by preserving HTML structure, enhancing retrieval quality.

M3DOCRAG: Multi-Modal Retrieval For Document Understanding introduces a multimodal RAG framework to handle multi-page and document QA tasks with visual data.

Needle Threading: LLMs For Long-Context Retrieval examines LLMs’ retrieval capabilities, identifying limits in handling extended contexts.

HtmlRAG: HTML-Based RAG System utilizes HTML structure in retrieval-augmented generation, improving document comprehension.

Survey Of Cultural Awareness In Language Models reviews cultural inclusivity in LLMs, emphasizing diverse and ethically sound datasets.

OPENCODER: The Open Cookbook For Code Models provides a comprehensive open-source guide for building high-performance code LLMs.

Polynomial Composition Activations enhances model expressivity using polynomial activations, optimizing parameter efficiency.

Hunyuan-Large: An Open-Source MoE Model presents a large-scale MoE model, excelling across language, math, and coding tasks.

Balancing Pipeline Parallelism With Vocabulary Parallelism improves transformer training efficiency by balancing memory across vocabulary layers.