Predicting Apple’s A.I. Play

iSolutions
10 min readFeb 18, 2024

Studying the clues from various releases to predict whats next for Apple in A.I.

Apple’s AI “breadcrumbs” timeline

Amid the flurry of headlines dominated by tech giants making bold strides in artificial intelligence, Apple appears to navigate a quieter path, seemingly absent from AI industry news.

However, a closer examination reveals a trail of breadcrumbs, hinting at the company’s strategic foray into AI that could soon reshape the landscape.

This understated approach, characterized by selective acquisitions, covert project developments, and meticulous integration of AI into its ecosystem, suggests Apple is crafting a future where AI enhances its suite of products in uniquely Apple ways.

By piecing together these clues, we get a glimpse what might be Apple’s AI play.

Acquisitions:

In January of 2020, Apple acquired Xnor.ai , which began as a process for making machine learning algorithms highly efficient — so efficient that they could run on even the lowest tier of hardware out there, things like mobile devices that use only a modicum of power. Yet using Xnor’s algorithms they could accomplish tasks like object recognition, which in other circumstances might require a powerful processor or connection to the cloud.

Later in 2020, Apple acquired Vilynx, who developed a self-learning AI platform for Media to understand content and personalize the user experience on publisher websites, mobile/OTT apps and optimize content creation and distribution. Deep metadata tagging is the base to drive automated previews, recommendations and smart search.

What this could mean: An edge computing-enhanced Siri that processes user data locally could transform how users interact with their Apple devices, making Siri a more integral and trusted part of daily life. This would not only improve the user experience by making Siri more responsive and capable but also reinforce Apple’s commitment to user privacy and security.

AJAX:

In July of 2023, rumors began about “AJAX”, Apple’s internal code name for “Apple GPT” based on Apple’s proprietary language model framework, Ajax.

The development of Apple GPT is part of Apple’s broader efforts in the field of artificial intelligence. The company has been relatively quiet about these efforts compared to other tech giants.

The internal name for Apple’s language model framework is “Ajax”. The origin of this name is unclear, but it could be a combination of Google’s “JAX” and the “A” from Apple.

The Apple GPT project is still under development. However, it’s reported that Apple is planning to make a major AI-related announcement in 2024, which could be the general release of Apple GPT.

Since, this author has confirmed the existence of AJAX internally within Apple.

What this could mean: The rumored development of AJAX or “Apple GPT” could herald a new era of AI integration within Apple’s ecosystem, potentially integrating it with Safari and Spotlight for smarter, more context-aware search results, AI-driven health and fitness advice, advanced educational tools, creative and content generation applications, to new forms of interactive entertainment. Apple’s focus on privacy and the user experience could set AJAX apart in the crowded field of AI and language models.

Ferrett:

In October 2023, Apple released Ferret, an open-source, multimodal Large Language Model (LLM) developed in collaboration with Cornell University, which represents a significant pivot from its traditionally secretive approach towards a more open stance in the AI domain.

Ferret distinguishes itself by integrating language understanding with image analysis, enabling it to not only comprehend text but also analyze specific regions within images to identify elements and use these in queries. This capability allows for more nuanced interactions, where Ferret can provide contextual responses based on both text and visual inputs​.

What this could mean: The introduction of Ferret into Apple’s ecosystem could potentially revolutionize how users interact with Apple devices, offering enhanced image-based interactions and augmented user assistance. For instance, Siri could leverage Ferret’s capabilities to understand queries about images or perform actions based on visual content, significantly improving the user experience by providing more accurate and context-aware responses. Ferret could enrich media and content understanding, improving the organization, search functionality within Apple’s Photos app, and even offering more personalized content recommendations across Apple’s services​

MLX:

In December 2023, Apple released MLX and MLX Data signifies a pivotal shift towards empowering developers to create more sophisticated AI applications that are optimized for Apple Silicon.

The MLX framework, inspired by PyTorch, Jax, and ArrayFire but with the unique feature of shared memory, simplifies the process for developers to build models that work seamlessly across CPUs and GPUs without the need to transfer data.

MLX is a NumPy-like array framework designed for efficient and flexible machine learning on Apple silicon, brought to you by Apple machine learning research.

The Python API closely follows NumPy with a few exceptions. MLX also has a fully featured C++ API which closely follows the Python API.

What this could mean: This could lead to a new era of generative AI apps on MacBooks, which may include capabilities similar to Meta’s Llama or Stable Diffusion.

HUGS:

Also in December 2023, Apple Machine Learning released “HUGS: Human Gaussian Splats,” in collaboration with the Max Planck Institute for Intelligent Systems, introduces a novel approach to create animatable human avatars and scenes from monocular videos using 3D Gaussian Splatting.

This method efficiently separates and animates humans within scenes, achieving state-of-the-art rendering quality and speed. It addresses challenges in animating 3D Gaussians, optimizing for realistic movement and enabling novel pose and view synthesis at high speeds, significantly outperforming previous methods in both training time and rendering speed.

https://mlr.cdn-apple.com/video/novel_pose_view_2_79fa4a7a4f.mp4
https://mlr.cdn-apple.com/video/novel_multihuman_scene_1_c5b2b300a.mp4

What this could mean: Apple’s “HUGS: Human Gaussian Splats” could enable more realistic and interactive 3D animations from simple video inputs. This could lead to advancements in augmented reality experiences, improved virtual assistants, and more immersive gaming and social media applications on Apple devices. The technology’s efficiency in rendering and animating could also enhance user experiences across the Apple ecosystem, making digital interactions more lifelike and engaging.

Flash Memory:

In December 2023, Apple released a research paper titled “LLM in a flash: Efficient Large Language Model Inference with Limited Memory,” noted that flash storage is more abundant in mobile devices than the RAM traditionally used for running LLMs.

Their method cleverly bypasses the limitation using two key techniques that minimize data transfer and maximize flash memory throughput:

  1. Windowing: Think of this as a recycling method. Instead of loading new data every time, the AI model reuses some of the data it already processed. This reduces the need for constant memory fetching, making the process faster and smoother.
  2. Row-Column Bundling: This technique is like reading a book in larger chunks instead of one word at a time. By grouping data more efficiently, it can be read faster from the flash memory, speeding up the AI’s ability to understand and generate language.

The combination of these methods allows AI models to run up to twice the size of the iPhone’s available memory, according to the paper. This translates to a 4–5 times increase in speed on standard processors (CPUs) and an impressive 20–25 times faster on graphics processors (GPUs). “This breakthrough is particularly crucial for deploying advanced LLMs in resource-limited environments, thereby expanding their applicability and accessibility,” write the authors.

What this could mean: Apple’s research on running Large Language Models (LLMs) efficiently on mobile devices could herald a new era of on-device AI processing. By optimizing data handling through “windowing” and “row-column bundling,” Apple has potentially developed a method to significantly speed up AI inference on standard CPUs and GPUs. This breakthrough could lead to faster and more powerful AI functionalities directly on iPhones and iPads without relying on cloud computing, enhancing user experience while maintaining Apple’s strong stance on privacy.

Siri Summarizations:

In January 2024, leaks on iOS 18 revealed Siri Summarization functionality that specifically reference OPENAI.

The leaked images show specific functions referencing both AJAX and OpenAI.

Specific reference to “OpenAIGPT”

The mentions of “OpenAISettings” section and references to issues such as “SummarizationOpenAIError” and a missing OpenAI API key.

This suggests that there’s an attempt to make an API call to OpenAI’s service, likely for a feature involving text summarization.

Specific reference to “AJAXGPTonDevice”

What this could mean: This could indicate that Apple is testing or developing a feature using OpenAI’s language models for summarization purposes within its software, as suggested by the “SummarizationOpenAIError.” The code seems to be part of an internal testing process for integrating OpenAI’s language models into Apple’s ecosystem, potentially for improving Siri’s functionalities or other text-based features in iOS.

MGIE:

In February of 2024, Apple released MGIE, representing a significant advancement in instruction-based image editing.

Utilizing multimodal large language models (MLLMs), MGIE interprets natural language commands for precise pixel-level image manipulations, covering a spectrum from Photoshop-style modifications to global photo enhancements and detailed local edits.

Developed in collaboration with the University of California, Santa Barbara, and showcased at ICLR 2024, MGIE underscores Apple’s growing AI research capabilities, offering a practical tool for creative tasks across personal and professional domains.

What this could mean: MGIE’s release could significantly propel Apple’s AI capabilities, especially in creative and personalization applications. This move may lead to more intuitive interfaces for content creation, potentially integrating into existing Apple products to offer advanced editing features directly within the ecosystem. MGIE’s integration into Apple’s software or hardware offerings could revolutionize user interaction with devices, making advanced image editing accessible directly from iPhones, iPads, or Macs. Imagine Siri or Photos app leveraging MGIE for editing commands, enhancing user creativity without complex software

Keyframer:

Also in February 2024, Apple released Keyframer, a generative AI tool for animating 2D images using text descriptions, showcasing the potential of LLMs in animation.

It simplifies the animation process, allowing users to animate SVG images through text prompts without coding knowledge. While promising, Keyframer is in the prototype stage, highlighting the evolving landscape of AI in creative fields and suggesting a future where AI tools could significantly augment creative workflows.

SVG images and text descriptions fed into Keyframer are automatically converted into animation code. Image: Apple

What this could mean: Keyframer could be integrated into Apple’s software ecosystem, enhancing creative tools in applications like Final Cut Pro, iMovie, or even Pages and Keynote, by enabling easy animation creation.

XCode AI:

In February of 2024, reports of an Apple Xcode Code AI Tool emerged. A report from Bloomberg says Apple has expanded internal testing of new generative AI features for its Xcode programming software and plans to release them to third-party developers this year.

Apple also reportedly looked at potential uses for generative AI in consumer-facing products, like automatic playlist creation in Apple Music, slideshows in Keynote, or AI chatbot-like search features for Spotlight search.

What this could mean: Apple’s development of AI coding tools could be transformative for its suite of developer services, particularly Xcode. These tools, potentially rivaling GitHub’s Copilot, would assist developers in writing code more efficiently, likely by suggesting code snippets, auto-completing lines of code, or even generating code from comments. This could greatly speed up the development process, reduce bugs, and make development more accessible to a broader range of skill levels.

iWork.ai:

In Feb of 2024, According to BuyAIDomains, Apple became the owner of the domain very recently, with its company name and business address of Apple Park in the owner records of “iWork.ai”.

What this could mean: This domain offers speculation that its office suite, consisting of Pages, Keynote, and Numbers, could all be getting some big artificial intelligence features soon.

WWDC:

Apple’s annual Worldwide Developer Conference typically takes place in June. Apple has not yet announced the dates for the event.

A new rumor claims that Apple’s generative AI technology will be included in Siri not just locally on iPhones, but also integrated into other services being announced at Apple’s 2024 Worldwide Developer Conference.

What this could mean: Get your popcorn ready, clarity on these rumors could be coming in June.

As Apple continues to navigate the rapidly evolving landscape of artificial intelligence, its strategic investments and innovations underscore a commitment to not just participate, but lead in the AI revolution.

By prioritizing user privacy and pushing the boundaries of AI capabilities, Apple is setting a new standard for how technology companies can blend advanced AI functionalities with ethical considerations.

In doing so, Apple is not just shaping the future of its products but is also influencing the broader trajectory of the tech industry towards a more responsible use of AI.

--

--

iSolutions

Multiple award-winning experts in custom applications, machine learning models and artificial intelligence for business.