Recently, I can often see people using NVIDIA DGX Spark to deploy tens of billions or hundreds of billions of models, which makes many people itch.
However, DGX Spark does have strong performance and the price is not close to the people. For most developers, the real thing is the computer at hand that has been in service for several years.
Recently, some netizens shared a set of 2022 first-generation Mac Studio (M1 Max /32GB) deployed on Ling-3.0-tiny INT4 The complete plan.
Originally, this netizen thought that it was difficult for this machine to undertake today's large-scale model reasoning tasks.
However, the actual test results were somewhat unexpected.
The model not only supports chatting, Think Mode and Tool Calling, but also maintains an appointment on Apple Silicon. 64 Token/s The stable generation speed of is quite smooth for scenarios such as local chat, code assistance, translation, and personal Agent.
More importantly, it is not a "1B model" in the traditional sense.
Why is Ling-3.0-tiny worth paying attention?
The total parameter size of Ling-3.0-tiny is approximately 7.9B。
However, it uses MoE (Mixture of Experts) sparse expert architecture。
Simply put:
- The model has approximately 128 experts (Experts)
- Each Token activates only 8 experts
- Then cooperate with shared experts to participate in the calculation
- Only about 1.3B Parameter
This design takes into account model capacity and reasoning costs, and also allows it to achieve good reasoning speed on Apple Silicon.
test environment
The device configuration used by this netizen is as follows:
- Apple Mac Studio（2022）
- Apple M1 Max
- 24-core GPU
- 32GB Unified Memory
- macOS 15.7.4
- arm64
Although Ling-3.0-tiny supports longer contexts, in order to balance memory usage and stability, this test sets the Context to 8K。
This is enough for chatting, code completion and daily Agent applications, while also preventing KV Cache from continuously occupying a large amount of unified memory.
The entire deployment architecture is as follows:
Ling-3.0-tiny INT4
│
支持 Ling 的 Ollama 实验分支
│
MLX Runner
│
Metal / Apple Silicon GPU
│
Ollama API（11434）The reason why the stable version of Olama is not directly used is that the current support of Ling and Bailing MoE V3 still relies on the Olama experimental branch.
Inference frameworks such as vLLM and SGLang are more suitable for NVIDIA CUDA environments and are not the best choice on Apple Silicon.
Plan your disk space before deploying
Throughout the deployment process, the most easily underestimated thing is not the model itself, but the disk usage.
The final catalog occupancy is roughly as follows:
| content | size |
|---|---|
| Hugging Face INT4 raw weights | Approximately 5.4GB |
| Ollama Import Model | Approximately 5.8GB |
| Olama source code, MLX and compilation results | Approximately 1.3GB |
| Cache and Configuration | Dozens of MB |
The entire project is final 13GB。
Consider downloading temporary files, compilation caches, and macOS Swap. It is recommended to reserve at least 20GB Available space, preparation is recommended 30GB above.
An easy to ignore pit: Git
The biggest experience shared by this netizen was not the model, but Git.
Since the entire deployment process is completed in the Codex project directory, the project enables version snapshots by default.
After downloading the model, Git saved a copy of the entire model catalog.
As a result, a model of about 5 GB quickly expanded to more than 10 GB, and the system disk space decreased rapidly.
The final investigation found that the model catalogue was not added in advance. .gitignore。
It is recommended to add in advance:
.cache/
models/
ollama-models/
ollama-ling/After re-cleaning Git Objects, the entire .git Only about is left in the catalog 132KB。
If you are ready to deploy a large model, it is best to complete this step ahead of schedule.
Download Ling-3.0-tiny INT4
Officials have provided Hugging Face INT4 weights.
The entire warehouse contains:
- 44 documents
- 32 safetensers are divided into weights
During the download process, this netizen mainly encountered two problems:
First, the Xet download speed has been stagnant for a long time.
Later, Xet was closed and normal HTTP downloads were switched to.
Second, timeouts occur frequently during the download process.
Since the Hugging Face CLI supports breakpoints to resume, you only need to increase the timeout time appropriately to continue downloading.
After the download was over, the model was not directly considered complete.
Instead, three checks were performed:
- Check whether all 32 weight slices exist
- Check whether the model index is fully referenced
- Calculate SHA-256 for all fragments
Final result:
- File exception: 0
- Hash exception: 0
- Missing fragment: 0
For models of several gigabytes, it is recommended to complete at least the number of shards or hash verification, rather than judging the model is complete based on just the download completion prompt.
Compiled Olama that supports Ling
At present, the official stable version of Olama cannot directly run Ling, so you need to switch experimental branches and compile it yourself.
During the whole process, what really stuck this netizen was Metal Toolchain。
The machine has been installed:
- Xcode
- Command Line Tools
However, MLX compilation still reported an error.
It was later discovered that Metal Toolchain needed to be installed separately.
What's even more troublesome is that even if the installation is complete, the system may not find the Metal compiler.
After finally viewing the Toolchain Identifier, you need to set it manually:
export TOOLCHAINS=...It should be noted that this value will change with different versions of macOS and Xcode, and it is not recommended to directly copy other people's configurations.
After compilation is completed, it will generate:
- Ollama main program
- MLX Runner
- Metal Kernel
- dynamic library
Need special attention, don't just keep ollama Executable file.
The actual running model also relies on Runner and dynamic libraries in the build directory.
Import INT4 models
In order to avoid mixing it with the stable version of Olama in the system, this netizen placed the model catalog separately within the project.
After the import is complete, Ollama outputs:
26947 tensors
479 layers
preserving source quantizationDescription INT4 weights remain in the original quantization format and do not need to be converted again.
The final model name is:
ling-tiny-int4:latestHow to make sure the GPU is really working?
This is very important.
Because models may silently fall back to the CPU.
After deployment is complete, you can check the log to confirm whether similar information appears:
library=Metal
Apple M1 Max
MLX engine initialized device=gpuOnly by seeing these logs can you confirm that the entire reasoning process is actually running on the Apple GPU.
M1 Max Measured Speed
In order to avoid the first Token delay affecting the results, this netizen first warmed up the model.
Three consecutive rounds of full generation testing were then conducted.
The test conditions are unified as follows:
- Context：8192
- Temperature：0
- Think: Close
- Output length: 384 Tokens
The final results are as follows:
| test | Token/s |
|---|---|
| the first round | 64.26 |
| the second round | 65.18 |
| the third round | 63.67 |
Final average:
64.37 Token/s
Approximately time to load the model for the first time:
6.93 seconds
After the model is stored, it takes up about unified video memory:
6.48GB
For an M1 Max that has been released for several years, this performance is quite good.
How was the actual experience?
After the deployment was completed, the netizen connected the model to a screenshot translation tool commonly used on Mac Bob。
Since Bob can directly call the Ollama API, there is no need to configure an additional cloud interface.
From the actual experience, the local translation response speed is very fast.
Especially in the screenshot translation scenario, the result can be returned almost after clicking.
Compared with online translation services, although translation quality is different, the local model has two obvious advantages:
First, the response speed is faster;
Second, all data is retained locally, making it more suitable for processing private content.
In addition, it also performed well in the following scenarios:
- daily chat
- document summarizes
- AI translation
- code assist
- local knowledge base
- Personal Agent
The overall experience has been able to meet daily use needs.
written in the end
Judging from the entire deployment process, this netizen's biggest gain was not "Can Mac run a big model?"
Instead, Apple Silicon's local AI ecosystem has become increasingly mature.
With the development of MLX, Metal, MoE and more efficient quantitative solutions, a M1 Max from a few years ago has also provided a good local reasoning experience.
Of course, it still cannot replace high-end NVIDIA GPUs, nor is it suitable for high-concurrency production environments.
But for individual developers, it is enough:
- local chat
- AI translation
- Programming assistant
- Private knowledge base
- Personal Agent
- Private text processing
If you have a limited budget and want to experience a truly local model, then Ling-3.0-tiny INT4 + Apple Silicon + MLX + OllamaIt is still a combination worth trying.
At the very least, it proves that Mac Studio, released a few years ago, still has good native AI reasoning capabilities.