{"id":352,"date":"2025-01-13T01:16:22","date_gmt":"2025-01-13T01:16:22","guid":{"rendered":"https:\/\/nocoffei.com\/?p=352"},"modified":"2025-01-13T01:18:00","modified_gmt":"2025-01-13T01:18:00","slug":"switch-ai-%e2%9c%a8","status":"publish","type":"post","link":"https:\/\/nocoffei.com\/?p=352","title":{"rendered":"Switch AI \u2728"},"content":{"rendered":"\n
\n

Man, I should really write a clickbait article about the Switch before the Switch 2 comes out.<\/p>\n\n\n\n

– Me<\/em><\/p>\n<\/blockquote>\n\n\n\n

What’s more clickbaity than AI \u2728? We’re now in the third year of being promised the world– AI \u2728 is going to change everything, everywhere, and if you’re not adopting it into every aspect of your life and career you’re going to get left in the dust.<\/p>\n\n\n\n

The Switch runs an off-the-shelf Nvidia Tegra X1 with 4GB of RAM. It was Nvidias second desktop GPU architecture in a mobile chip, after the Nvidia Tegra K1, containing 256 Maxwell CUDA cores. Announced ten years ago now (On January 5, 2015) it was originally intended for tablets and the automotive industry<\/a>, before Nintendo picked it up. The Jetson TX1 dev kit was released and it was also present in the somewhat popular and well-known Jetson Nano (though with only half the CUDA cores enabled.) A fully functional (albeit largely proprietary, in traditional Nvidia fashion) Linux4Tegra distribution was shipped, with a working CUDA development environment.<\/p>\n\n\n\n

CUDA? The compute platform that’s driving the AI revolution \u2728? In my Nintendo Switch?<\/p>\n\n\n\n

Surely you see where I’m going with this.<\/p>\n\n\n\n

Still image version<\/summary>\n
\"\"<\/figure>\n\n\n\n

<\/p>\n<\/details>\n\n\n\n

Pure text version<\/summary>\n
qwe@localhost:~\/llama.cpp\/build\/bin$ .\/llama-cli -m ~\/Llama-3.2-3B-Instruct-Q4_K_M.gguf -ngl 100 -cnv --chat-template  llama3\nggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no\nggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\nggml_cuda_init: found 1 CUDA devices:\n  Device 0: NVIDIA Tegra X1, compute capability 5.3, VMM: no\nregister_backend: registered backend CUDA (1 devices)\nregister_device: registered device CUDA0 (NVIDIA Tegra X1)\nregister_backend: registered backend CPU (1 devices)\nregister_device: registered device CPU (ARMv8 Processor rev 1 (v8l))\nbuild: 4418 (b56f079e) with gcc (GCC) 8.5.0 for aarch64-unknown-linux-gnu (debug)\nmain: llama backend init\nmain: load the model and apply lora adapter, if any\nllama_load_model_from_file: using device CUDA0 (NVIDIA Tegra X1) - 2755 MiB free\nllama_model_loader: loaded meta data with 35 key-value pairs and 255 tensors from \/home\/qwe\/Llama-3.2-3B-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))\nllama_model_loader: Dumping metadata keys\/values. Note: KV overrides do not apply in this output.\nllama_model_loader: - kv   0:                       general.architecture str              = llama\nllama_model_loader: - kv   1:                               general.type str              = model\nllama_model_loader: - kv   2:                               general.name str              = Llama 3.2 3B Instruct\nllama_model_loader: - kv   3:                           general.finetune str              = Instruct\nllama_model_loader: - kv   4:                           general.basename str              = Llama-3.2\nllama_model_loader: - kv   5:                         general.size_label str              = 3B\nllama_model_loader: - kv   6:                            general.license str              = llama3.2\nllama_model_loader: - kv   7:                               general.tags arr[str,6]       = [\"facebook\", \"meta\", \"pytorch\", \"llam...\nllama_model_loader: - kv   8:                          general.languages arr[str,8]       = [\"en\", \"de\", \"fr\", \"it\", \"pt\", \"hi\", ...\nllama_model_loader: - kv   9:                          llama.block_count u32              = 28\nllama_model_loader: - kv  10:                       llama.context_length u32              = 131072\nllama_model_loader: - kv  11:                     llama.embedding_length u32              = 3072\nllama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192\nllama_model_loader: - kv  13:                 llama.attention.head_count u32              = 24\nllama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8\nllama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000\nllama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\nllama_model_loader: - kv  17:                 llama.attention.key_length u32              = 128\nllama_model_loader: - kv  18:               llama.attention.value_length u32              = 128\nllama_model_loader: - kv  19:                          general.file_type u32              = 15\nllama_model_loader: - kv  20:                           llama.vocab_size u32              = 128256\nllama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128\nllama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2\nllama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = llama-bpe\nllama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,128256]  = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\nllama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\nllama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,280147]  = [\"\u0120 \u0120\", \"\u0120 \u0120\u0120\u0120\", \"\u0120\u0120 \u0120\u0120\", \"...\nllama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 128000\nllama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 128009\nllama_model_loader: - kv  29:                    tokenizer.chat_template str              = {{- bos_token }}\\n{%- if custom_tools ...\nllama_model_loader: - kv  30:               general.quantization_version u32              = 2\nllama_model_loader: - kv  31:                      quantize.imatrix.file str              = \/models_out\/Llama-3.2-3B-Instruct-GGU...\nllama_model_loader: - kv  32:                   quantize.imatrix.dataset str              = \/training_dir\/calibration_datav3.txt\nllama_model_loader: - kv  33:             quantize.imatrix.entries_count i32              = 196\nllama_model_loader: - kv  34:              quantize.imatrix.chunks_count i32              = 125\nllama_model_loader: - type  f32:   58 tensors\nllama_model_loader: - type q4_K:  168 tensors\nllama_model_loader: - type q6_K:   29 tensors\nllm_load_vocab: special tokens cache size = 256\nllm_load_vocab: token to piece cache size = 0.7999 MB\nllm_load_print_meta: format           = GGUF V3 (latest)\nllm_load_print_meta: arch             = llama\nllm_load_print_meta: vocab type       = BPE\nllm_load_print_meta: n_vocab          = 128256\nllm_load_print_meta: n_merges         = 280147\nllm_load_print_meta: vocab_only       = 0\nllm_load_print_meta: n_ctx_train      = 131072\nllm_load_print_meta: n_embd           = 3072\nllm_load_print_meta: n_layer          = 28\nllm_load_print_meta: n_head           = 24\nllm_load_print_meta: n_head_kv        = 8\nllm_load_print_meta: n_rot            = 128\nllm_load_print_meta: n_swa            = 0\nllm_load_print_meta: n_embd_head_k    = 128\nllm_load_print_meta: n_embd_head_v    = 128\nllm_load_print_meta: n_gqa            = 3\nllm_load_print_meta: n_embd_k_gqa     = 1024\nllm_load_print_meta: n_embd_v_gqa     = 1024\nllm_load_print_meta: f_norm_eps       = 0.0e+00\nllm_load_print_meta: f_norm_rms_eps   = 1.0e-05\nllm_load_print_meta: f_clamp_kqv      = 0.0e+00\nllm_load_print_meta: f_max_alibi_bias = 0.0e+00\nllm_load_print_meta: f_logit_scale    = 0.0e+00\nllm_load_print_meta: n_ff             = 8192\nllm_load_print_meta: n_expert         = 0\nllm_load_print_meta: n_expert_used    = 0\nllm_load_print_meta: causal attn      = 1\nllm_load_print_meta: pooling type     = 0\nllm_load_print_meta: rope type        = 0\nllm_load_print_meta: rope scaling     = linear\nllm_load_print_meta: freq_base_train  = 500000.0\nllm_load_print_meta: freq_scale_train = 1\nllm_load_print_meta: n_ctx_orig_yarn  = 131072\nllm_load_print_meta: rope_finetuned   = unknown\nllm_load_print_meta: ssm_d_conv       = 0\nllm_load_print_meta: ssm_d_inner      = 0\nllm_load_print_meta: ssm_d_state      = 0\nllm_load_print_meta: ssm_dt_rank      = 0\nllm_load_print_meta: ssm_dt_b_c_rms   = 0\nllm_load_print_meta: model type       = 3B\nllm_load_print_meta: model ftype      = Q4_K - Medium\nllm_load_print_meta: model params     = 3.21 B\nllm_load_print_meta: model size       = 1.87 GiB (5.01 BPW) \nllm_load_print_meta: general.name     = Llama 3.2 3B Instruct\nllm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'\nllm_load_print_meta: EOS token        = 128009 '<|eot_id|>'\nllm_load_print_meta: EOT token        = 128009 '<|eot_id|>'\nllm_load_print_meta: EOM token        = 128008 '<|eom_id|>'\nllm_load_print_meta: LF token         = 128 '\u00c4'\nllm_load_print_meta: EOG token        = 128008 '<|eom_id|>'\nllm_load_print_meta: EOG token        = 128009 '<|eot_id|>'\nllm_load_print_meta: max token length = 256\nllm_load_tensors: offloading 28 repeating layers to GPU\nllm_load_tensors: offloading output layer to GPU\nllm_load_tensors: offloaded 29\/29 layers to GPU\nllm_load_tensors:   CPU_Mapped model buffer size =   308.23 MiB\nllm_load_tensors:        CUDA0 model buffer size =  1918.35 MiB\n...........................................................................\nllama_new_context_with_model: n_seq_max     = 1\nllama_new_context_with_model: n_ctx         = 4096\nllama_new_context_with_model: n_ctx_per_seq = 4096\nllama_new_context_with_model: n_batch       = 2048\nllama_new_context_with_model: n_ubatch      = 512\nllama_new_context_with_model: flash_attn    = 0\nllama_new_context_with_model: freq_base     = 500000.0\nllama_new_context_with_model: freq_scale    = 1\nllama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized\nllama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1\nllama_kv_cache_init:      CUDA0 KV buffer size =   448.00 MiB\nllama_new_context_with_model: KV self size  =  448.00 MiB, K (f16):  224.00 MiB, V (f16):  224.00 MiB\nllama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB\nllama_new_context_with_model:      CUDA0 compute buffer size =   256.50 MiB\nllama_new_context_with_model:  CUDA_Host compute buffer size =    14.01 MiB\nllama_new_context_with_model: graph nodes  = 902\nllama_new_context_with_model: graph splits = 2\ncommon_init_from_params: setting dry_penalty_last_n to ctx_size = 4096\ncommon_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)\nmain: llama threadpool init, n_threads = 4\nmain: chat template example:\n<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHi there<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHow are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n\n\nsystem_info: n_threads = 4 (n_threads_batch = 4) \/ 4 | CUDA : USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | \n\nmain: interactive mode on.\nsampler seed: 2854367793\nsampler params: \n        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000\n        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096\n        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800\n        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\nsampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist \ngenerate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 1\n\n== Running in interactive mode. ==\n - Press Ctrl+C to interject at any time.\n - Press Return to return control to the AI.\n - To return control without starting a new line, end your input with '\/'.\n - If you want to submit another line, end your input with '\\'.\n\n\n> Llama, tell me what a Nintendo Switch is\nThe Nintendo Switch is a hybrid video game console developed and published by Nintendo. It was released in 2017 and has since become one of the most popular gaming platforms in the world.\n\nThe Switch is designed to be a versatile device that can be used in multiple modes:\n\n1. **Portable mode**: The Switch can be held like a tablet, allowing players to take it anywhere and play games on the go.\n2. **Tabletop mode**: The Switch can be placed on a flat surface, detach its Joy-Con controllers, and be used like a television. This mode is ideal for multiplayer games or when playing with a large group.\n3. **TV mode**: The Switch can be connected to a television using an HDMI cable, providing a bigger screen and more immersive gaming experience.\n\nThe Switch has a robust library of games, including popular titles like Super Mario Odyssey, The Legend of Zelda: Breath of the Wild, Mario Kart 8 Deluxe, and Splatoon 2. It also has a range of indie games, eShop releases, and Nintendo's own franchises.\n\nSome of the key features of the Switch include:\n\n* A 6.2-inch capacitive touchscreen display\n* A 32-bit Tegra X1 processor\n* 4GB or 32GB of RAM\n* A battery life of up to 2.5 to 6 hours, depending on the game and brightness settings\n* Support for Joy-Con controllers, which can be attached to the sides of the console or used separately\n* Compatibility with Nintendo's Amiibo figures and other accessories\n\nOverall, the Nintendo Switch is a unique and innovative gaming platform that offers a distinct gaming experience, making it a popular choice among gamers of all ages!\n\n> Can a Nintendo Switch run a large language model?\nWhile the Nintendo Switch is a powerful gaming console, its hardware might not be the most suitable for running large language models.\n\nHere are some reasons why:\n\n1. **Processing power**: The Switch's Tegra X1 processor, which is a 32-bit processor, might not be powerful enough to handle the complex computations required for training and running large language models. Language models typically require significant computational resources, especially when it comes to large-scale models like BERT, RoBERTa, or transformer-based models.\n2. **Memory**: The Switch has 4GB or 32GB of RAM, which is relatively limited compared to the requirements of large language models. These models often require massive amounts of memory to store their weights and activations, especially when fine-tuning or generating text.\n3. **Graphics processing unit (GPU)**: The Switch's GPU is optimized for gaming and not specifically designed for AI computing. This means it might not be able to handle the intense computational tasks required for running large language models.\n\nHowever, it's not impossible to run a large language model on a Nintendo Switch. Some researchers and developers have explored using the Switch for running smaller, more lightweight language models, such as those based on transformer architectures. In these cases, the Switch's processor and memory can be sufficient to run the model.\n\nIf you're interested in running a large language model on a Nintendo Switch, you might consider the following:\n\n1. **Downsample the model**: Use a smaller, more efficient version of the language model to reduce its computational requirements.\n2. **Use a more lightweight framework**: Explore alternative frameworks, such as TensorFlow Lite or PyTorch, which are optimized for mobile and embedded devices.\n3. **Use a cloud-based service**: Consider using a cloud-based service, such as AWS SageMaker or Google Cloud AI Platform, which can provide more powerful computing resources and scalability.\n\nKeep in mind that running a large language model on a Nintendo Switch will likely require significant optimization, and the results might not be as impressive as running the model on a more powerful device.\n\nWould you like to explore any of these options further?\n\n> \nllama_perf_sampler_print:    sampling time =     784.87 ms \/   445 runs   (    1.76 ms per token,   566.97 tokens per second)\nllama_perf_context_print:        load time =   57389.43 ms\nllama_perf_context_print: prompt eval time =   16092.76 ms \/    40 tokens (  402.32 ms per token,     2.49 tokens per second)\nllama_perf_context_print:        eval time =  220373.49 ms \/   771 runs   (  285.83 ms per token,     3.50 tokens per second)\nllama_perf_context_print:       total time =  396023.26 ms \/   811 tokens\nInterrupted by user<\/code><\/pre>\n<\/details>\n\n\n\n
That’s right folks, with only mild<\/em> amounts of hacking we can get modern LLMs \u2728 running at a pretty reasonable clip [4 tokens per second] on a Nintendo Switch! Here’s how it works, and what hacks were needed to get there.<\/p>\n\n\n\n
Basic Premise<\/h2>\n\n\n\n
There’s an organization of folks collectively named Switchroot<\/a> who have done an unbelievable amount of work to make a pleasant and easy-to-use port of Linux4Tegra to the Switch. There are ports of three different Ubuntu versions, Fedora and Android. The Joy-Cons work, there are built in power profiles including overclocking for both CPU and GPU, touchscreen and HDMI out work, auto rotate works, sleep works… Everything works. <\/p>\n\n\n\n
Including CUDA. At least, on the original Ubuntu version L4T shipped with, 18.04.<\/p>\n\n\n\n
CUDA is a platform for universal GPU acceleration across all Nvidia GPUs. Anything from the lowliest $50 GPU in the back of an old desktop to the most powerful server accelerator card can be programmed using one unified API. That means that, in theory, the code that runs LLM \u2728 inference on CUDA will Just Work when compiled to run on the Tegra X1.<\/p>\n\n\n\n
\n
Meanwhile, Nvidias competitors are still not capable of getting their compute stack to run well on their highest-end five-figure accelerators<\/a>, let alone any of their desktop GPUs.<\/p>\n\n\n\n
For reference, Nvidia still actively supports desktop Maxwell GPUs, released almost 11 years ago<\/em>.<\/p>\n\n\n\n
At least things are improving. I was able to run an LLM on my new AMD laptop just the other day<\/a> after lots of compiling experimental forks; better than my past experiences, which are just “give up, it’s broken.”<\/p>\n<\/blockquote>\n\n\n\n
Of course, there’s a catch– Nvidia dropped support for the Tegra X1 quite some time ago, and we’re left with CUDA version 10.2.<\/p>\n\n\n\n
llama.cpp<\/a>, arguably the most popular project for running LLMs \u2728 at home, uses GGML<\/a> as the core code that actually performs the high-speed matrix multiplication for AI \u2728. So all we have to do is backport GGML to CUDA 10.2 and we’ll be all set to run AI on the Switch \u2728.<\/p>\n\n\n\n
Warning: Gory configuration, patches, and dependencies ahead<\/h2>\n\n\n\n
This all took about a weekend to figure out. It’s not that bad, there’s just a lot of it, and it’s not all very interesting. I’m just putting it here so others can replicate this process.<\/p>\n\n\n\n
We start here:<\/p>\n\n\n\n
cmake -B build -DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=\/usr\/local\/cuda\/bin\/nvcc <\/code><\/pre>\n\n\n\n
Problem<\/strong>: Linux4Tegra ships with CUDA 10.0 out of the box. We need CUDA 10.2 so we can have cublasLt, which GGML relies on.<\/p>\n\n\n\n
The Switchroot folks have you covered. There’s a quick update procedure on their Discord<\/a> (yes, I know…) which will get you updated.<\/p>\n\n\n\n
Problem: <\/strong>GGML freaks out immediately when using such an old compiler:<\/p>\n\n\n\n
CMake Error in src\/ggml-cuda\/CMakeLists.txt:\n  Target \"ggml-cuda\" requires the language dialect \"CUDA17\" (with compiler\n  extensions).  But the current compiler \"NVIDIA\" does not support this, or\n  CMake does not know the flags to enable it.<\/code><\/pre>\n\n\n\n
I found these CMake flags lying around that make the error go away.<\/p>\n\n\n\n
cmake -B build -DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=\/usr\/local\/cuda\/bin\/nvcc -DCMAKE_CUDA_STANDARD=14 -DCMAKE_CUDA_STANDARD_REQUIRED=true <\/code><\/pre>\n\n\n\n
Problem: <\/strong>Tegra X1 has its own CUDA architecture code (53), separate from desktop Maxwell (52). Specify this:<\/p>\n\n\n\n
cmake -B build -DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=\/usr\/local\/cuda\/bin\/nvcc -DCMAKE_CUDA_ARCHITECTURES=53 -DCMAKE_CUDA_STANDARD=14 -DCMAKE_CUDA_STANDARD_REQUIRED=true <\/code><\/pre>\n\n\n\n
Problem: <\/strong>GGML is currently bugged<\/a> when compiling for ARM. Add more CMake flags to make it go away.<\/p>\n\n\n\n
cmake -B build -DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=\/usr\/local\/cuda\/bin\/nvcc -DCMAKE_CUDA_ARCHITECTURES=53 -DCMAKE_CUDA_STANDARD=14 -DCMAKE_CUDA_STANDARD_REQUIRED=true -DGGML_NATIVE=off -DGGML_CPU_ARM_ARCH=armv8-a<\/code><\/pre>\n\n\n\n
Problem<\/strong>: Ubuntu 18.04 ships with GCC 7. CUDA 10.2 will not compile on anything newer than GCC 8.5. We need a new version of GCC, which adds support for compiler intrinsics<\/a> used for GGMLs ARM64 backend. GGML will not compile without these. <\/p>\n\n\n\n
So basically, you have to backport this patch to GCC 8.5 and build GCC from source yourself. You can cherry-pick commit 84d713f6f45bef44b8bea84b02caa8a165bff1b5<\/code>, it applies cleanly to the 8.5 release tag. Then just wait a couple hours for the Switch’s anemic CPU to compile a compiler. (Or git gud and figure out how to cross-compile a native compiler. I am not gud enough to figure that out.)<\/p>\n\n\n\n
Once that’s applied, you can continue building GGML with your shiny new-old compiler. Don’t forget to set up update-alternatives<\/code> so CUDA looks at GCC and is happy. Our final CMake flags are:<\/p>\n\n\n\n
cmake -B build -DGGML_CUDA=on -DCMAKE_CUDA_COMPILER=\/usr\/local\/cuda\/bin\/nvcc -DCMAKE_CUDA_ARCHITECTURES=53 -DCMAKE_CUDA_STANDARD=14 -DCMAKE_CUDA_STANDARD_REQUIRED=true -DGGML_NATIVE=off -DGGML_CPU_ARM_ARCH=armv8-a -DCMAKE_C_COMPILER=\/home\/qwe\/gcc-install\/home\/qwe\/gcc\/bin\/gcc -DCMAKE_CXX_COMPILER=\/home\/qwe\/gcc-install\/home\/qwe\/gcc\/bin\/g++<\/code><\/pre>\n\n\n\n
Problem:<\/strong> We’re making headway. Now we’re getting to actual<\/em> compiler errors from actual<\/em> missing features. <\/p>\n\n\n\n
GGML throws this:<\/p>\n\n\n\n
\/home\/qwe\/ggml\/src\/ggml-cuda\/common.cuh(348): error: A __device__ variable cannot be marked constexpr<\/code><\/pre>\n\n\n\n
That’s a CUDA C++17 feature. CUDA 10.2 doesn’t support C++17. The fix is easy, thankfully, just remove the constexpr<\/code>.<\/p>\n\n\n\n
\/home\/qwe\/ggml\/src\/ggml-cuda\/fattn-common.cuh(532): error: identifier \"__builtin_assume\" is undefined<\/code><\/pre>\n\n\n\n
This is splattered around in a few places . It’s just a compiler hint for performance, you can just comment it out.<\/p>\n\n\n\n
Problem: Compilation succeeds, but linker errors are getting thrown out the wazoo:<\/p>\n\n\n\n
..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::filesystem_error::_M_gen_what()'\n..\/src\/libggml.so: undefined reference to `vtable for std::filesystem::__cxx11::filesystem_error'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::directory_iterator::operator*() const'\n..\/src\/libggml.so: undefined reference to `typeinfo for std::filesystem::__cxx11::filesystem_error'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::path::_M_find_extension() const'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::filesystem_error::~filesystem_error()'\n..\/src\/libggml.so: undefined reference to `std::filesystem::status(std::filesystem::__cxx11::path const&)'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::path::_M_split_cmpts()'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::directory_iterator::operator++()'\n..\/src\/libggml.so: undefined reference to `std::filesystem::__cxx11::directory_iterator::directory_iterator(std::filesystem::__cxx11::path const&, std::filesystem::directory_options, std::error_code*)'<\/code><\/pre>\n\n\n\n
This one was a bit of a stumper. I wound up spending a couple hours trying to figure it out.<\/p>\n\n\n\n
I’m still not entirely sure what’s wrong, but I think GCC 8.5, being from an era where C++17 support was still new, doesn’t correctly link shared libraries to std::filesystem<\/code>. So it throws errors whenever you try to link it into a final binary, since those symbols are left unresolved until final linkage, and then fail to link.<\/p>\n\n\n\n
You can add some linker commands to grab GCC by the lapels and say “link this crap, punk” in ggml\/CMakeLists.txt.<\/p>\n\n\n\n
diff --git a\/CMakeLists.txt b\/CMakeLists.txt\nindex 3935065..3376657 100644\n--- a\/CMakeLists.txt\n+++ b\/CMakeLists.txt\n@@ -246,6 +246,8 @@ set(GGML_PUBLIC_HEADERS\n     include\/ggml-vulkan.h)\n \n set_target_properties(ggml PROPERTIES PUBLIC_HEADER \"${GGML_PUBLIC_HEADERS}\")\n+target_link_libraries(ggml PRIVATE stdc++fs)\n+add_link_options(-Wl,--copy-dt-needed-entries)\n #if (GGML_METAL)\n #    set_target_properties(ggml PROPERTIES RESOURCE \"${CMAKE_CURRENT_SOURCE_DIR}\/src\/ggml-metal.metal\")\n #endif()\n@@ -260,3 +262,9 @@ if (GGML_STANDALONE)\n     install(FILES ${CMAKE_CURRENT_BINARY_DIR}\/ggml.pc\n         DESTINATION share\/pkgconfig)\n endif()<\/code><\/pre>\n\n\n\n
I don’t know why these precise commands do the trick, but managed to scrape them together from hours of searching.<\/p>\n\n\n\n
We now have a working build!!<\/h2>\n\n\n\n
We’re so close… llama.cpp can load a large AI model \u2728 and write text! I’m loading Llama-3.2-3B-Instruct-Q4_K_M.<\/p>\n\n\n\n
Logs of llama.cpp loading<\/summary>\n
qwe@localhost:~\/llama.cpp\/build\/bin$ .\/llama-cli -m ~\/Llama-3.2-3B-Instruct-Q4_K_M.gguf -ngl 100 -cnv --chat-template  llama3\nggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no\nggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\nggml_cuda_init: found 1 CUDA devices:\n  Device 0: NVIDIA Tegra X1, compute capability 5.3, VMM: no\nregister_backend: registered backend CUDA (1 devices)\nregister_device: registered device CUDA0 (NVIDIA Tegra X1)\nregister_backend: registered backend CPU (1 devices)\nregister_device: registered device CPU (ARMv8 Processor rev 1 (v8l))\nbuild: 4418 (b56f079e) with gcc (GCC) 8.5.0 for aarch64-unknown-linux-gnu (debug)\nmain: llama backend init\nmain: load the model and apply lora adapter, if any\nllama_load_model_from_file: using device CUDA0 (NVIDIA Tegra X1) - 2007 MiB free\nllama_model_loader: loaded meta data with 35 key-value pairs and 255 tensors from \/home\/qwe\/Llama-3.2-3B-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))\nllama_model_loader: Dumping metadata keys\/values. Note: KV overrides do not apply in this output.\nllama_model_loader: - kv   0:                       general.architecture str              = llama\nllama_model_loader: - kv   1:                               general.type str              = model\nllama_model_loader: - kv   2:                               general.name str              = Llama 3.2 3B Instruct\nllama_model_loader: - kv   3:                           general.finetune str              = Instruct\nllama_model_loader: - kv   4:                           general.basename str              = Llama-3.2\nllama_model_loader: - kv   5:                         general.size_label str              = 3B\nllama_model_loader: - kv   6:                            general.license str              = llama3.2\nllama_model_loader: - kv   7:                               general.tags arr[str,6]       = [\"facebook\", \"meta\", \"pytorch\", \"llam...\nllama_model_loader: - kv   8:                          general.languages arr[str,8]       = [\"en\", \"de\", \"fr\", \"it\", \"pt\", \"hi\", ...\nllama_model_loader: - kv   9:                          llama.block_count u32              = 28\nllama_model_loader: - kv  10:                       llama.context_length u32              = 131072\nllama_model_loader: - kv  11:                     llama.embedding_length u32              = 3072\nllama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192\nllama_model_loader: - kv  13:                 llama.attention.head_count u32              = 24\nllama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8\nllama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000\nllama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\nllama_model_loader: - kv  17:                 llama.attention.key_length u32              = 128\nllama_model_loader: - kv  18:               llama.attention.value_length u32              = 128\nllama_model_loader: - kv  19:                          general.file_type u32              = 15\nllama_model_loader: - kv  20:                           llama.vocab_size u32              = 128256\nllama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128\nllama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2\nllama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = llama-bpe\nllama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,128256]  = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\nllama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\nllama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,280147]  = [\"\u0120 \u0120\", \"\u0120 \u0120\u0120\u0120\", \"\u0120\u0120 \u0120\u0120\", \"...\nllama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 128000\nllama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 128009\nllama_model_loader: - kv  29:                    tokenizer.chat_template str              = {{- bos_token }}\\n{%- if custom_tools ...\nllama_model_loader: - kv  30:               general.quantization_version u32              = 2\nllama_model_loader: - kv  31:                      quantize.imatrix.file str              = \/models_out\/Llama-3.2-3B-Instruct-GGU...\nllama_model_loader: - kv  32:                   quantize.imatrix.dataset str              = \/training_dir\/calibration_datav3.txt\nllama_model_loader: - kv  33:             quantize.imatrix.entries_count i32              = 196\nllama_model_loader: - kv  34:              quantize.imatrix.chunks_count i32              = 125\nllama_model_loader: - type  f32:   58 tensors\nllama_model_loader: - type q4_K:  168 tensors\nllama_model_loader: - type q6_K:   29 tensors\nllm_load_vocab: special tokens cache size = 256\nllm_load_vocab: token to piece cache size = 0.7999 MB\nllm_load_print_meta: format           = GGUF V3 (latest)\nllm_load_print_meta: arch             = llama\nllm_load_print_meta: vocab type       = BPE\nllm_load_print_meta: n_vocab          = 128256\nllm_load_print_meta: n_merges         = 280147\nllm_load_print_meta: vocab_only       = 0\nllm_load_print_meta: n_ctx_train      = 131072\nllm_load_print_meta: n_embd           = 3072\nllm_load_print_meta: n_layer          = 28\nllm_load_print_meta: n_head           = 24\nllm_load_print_meta: n_head_kv        = 8\nllm_load_print_meta: n_rot            = 128\nllm_load_print_meta: n_swa            = 0\nllm_load_print_meta: n_embd_head_k    = 128\nllm_load_print_meta: n_embd_head_v    = 128\nllm_load_print_meta: n_gqa            = 3\nllm_load_print_meta: n_embd_k_gqa     = 1024\nllm_load_print_meta: n_embd_v_gqa     = 1024\nllm_load_print_meta: f_norm_eps       = 0.0e+00\nllm_load_print_meta: f_norm_rms_eps   = 1.0e-05\nllm_load_print_meta: f_clamp_kqv      = 0.0e+00\nllm_load_print_meta: f_max_alibi_bias = 0.0e+00\nllm_load_print_meta: f_logit_scale    = 0.0e+00\nllm_load_print_meta: n_ff             = 8192\nllm_load_print_meta: n_expert         = 0\nllm_load_print_meta: n_expert_used    = 0\nllm_load_print_meta: causal attn      = 1\nllm_load_print_meta: pooling type     = 0\nllm_load_print_meta: rope type        = 0\nllm_load_print_meta: rope scaling     = linear\nllm_load_print_meta: freq_base_train  = 500000.0\nllm_load_print_meta: freq_scale_train = 1\nllm_load_print_meta: n_ctx_orig_yarn  = 131072\nllm_load_print_meta: rope_finetuned   = unknown\nllm_load_print_meta: ssm_d_conv       = 0\nllm_load_print_meta: ssm_d_inner      = 0\nllm_load_print_meta: ssm_d_state      = 0\nllm_load_print_meta: ssm_dt_rank      = 0\nllm_load_print_meta: ssm_dt_b_c_rms   = 0\nllm_load_print_meta: model type       = 3B\nllm_load_print_meta: model ftype      = Q4_K - Medium\nllm_load_print_meta: model params     = 3.21 B\nllm_load_print_meta: model size       = 1.87 GiB (5.01 BPW) \nllm_load_print_meta: general.name     = Llama 3.2 3B Instruct\nllm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'\nllm_load_print_meta: EOS token        = 128009 '<|eot_id|>'\nllm_load_print_meta: EOT token        = 128009 '<|eot_id|>'\nllm_load_print_meta: EOM token        = 128008 '<|eom_id|>'\nllm_load_print_meta: LF token         = 128 '\u00c4'\nllm_load_print_meta: EOG token        = 128008 '<|eom_id|>'\nllm_load_print_meta: EOG token        = 128009 '<|eot_id|>'\nllm_load_print_meta: max token length = 256\nllm_load_tensors: offloading 28 repeating layers to GPU\nllm_load_tensors: offloading output layer to GPU\nllm_load_tensors: offloaded 29\/29 layers to GPU\nllm_load_tensors:   CPU_Mapped model buffer size =   308.23 MiB\nllm_load_tensors:        CUDA0 model buffer size =  1918.35 MiB\n...........................................................................\nllama_new_context_with_model: n_seq_max     = 1\nllama_new_context_with_model: n_ctx         = 4096\nllama_new_context_with_model: n_ctx_per_seq = 4096\nllama_new_context_with_model: n_batch       = 2048\nllama_new_context_with_model: n_ubatch      = 512\nllama_new_context_with_model: flash_attn    = 0\nllama_new_context_with_model: freq_base     = 500000.0\nllama_new_context_with_model: freq_scale    = 1\nllama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized\nllama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1\nllama_kv_cache_init:      CUDA0 KV buffer size =   448.00 MiB\nllama_new_context_with_model: KV self size  =  448.00 MiB, K (f16):  224.00 MiB, V (f16):  224.00 MiB\nllama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB\nllama_new_context_with_model:      CUDA0 compute buffer size =   256.50 MiB\nllama_new_context_with_model:  CUDA_Host compute buffer size =    14.01 MiB\nllama_new_context_with_model: graph nodes  = 902\nllama_new_context_with_model: graph splits = 2\ncommon_init_from_params: setting dry_penalty_last_n to ctx_size = 4096\ncommon_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)\nmain: llama threadpool init, n_threads = 4\nmain: chat template example:\n<|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHi there<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHow are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n\n\nsystem_info: n_threads = 4 (n_threads_batch = 4) \/ 4 | CUDA : USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | \n\nmain: interactive mode on.\nsampler seed: 2791425704\nsampler params: \n        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000\n        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096\n        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800\n        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000\nsampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist \ngenerate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 1\n\n== Running in interactive mode. ==\n - Press Ctrl+C to interject at any time.\n - Press Return to return control to the AI.\n - To return control without starting a new line, end your input with '\/'.\n - If you want to submit another line, end your input with '\\'.\n\n\n> <\/code><\/pre>\n<\/details>\n\n\n\n
But what’s this? After I type a prompt, it crashes…<\/p>\n\n\n\n
> What's up?\n\nKilled\nqwe@localhost:~\/llama.cpp\/build\/bin$ <\/code><\/pre>\n\n\n\n
Insidious, deep bug somewhere in the depths that I’ll never find?<\/p>\n\n\n\n
Thankfully, no. The system is just running too low on memory and the OOM killer assassinates the process.<\/p>\n\n\n\n
It turns out that GDM3 and Unity, the default display manager and desktop environment, are way too RAM heavy to give a 3B model the 3 gigabtes of breathing room (out of 4GB total) it needs. Llama 3.2 1B runs fine, and if you take out all graphical stuff with sudo systemctl isolate multi-user.target<\/code> the 3B model will work fine over SSH. But that’s boring, so we’ll install lightdm<\/code> and LXDE to get a working desktop environment. There’s a quick and easy script from Switchroot that’ll take care of LXDE:<\/p>\n\n\n\n
bash <( curl https:\/\/raw.githubusercontent.com\/cobalt2727\/L4T-Megascript\/master\/helper.sh ) \"scripts\/desktop_environments\/lxde.sh\"<\/code><\/pre>\n\n\n\n
And lightdm<\/code> is already installed, just set it as the default with sudo dpkg-reconfigure lightdm<\/code> and reboot.<\/p>\n\n\n\n
You can make the OOM killer less trigger happy, and Ubuntu less overly swap-happy, like so:<\/p>\n\n\n\n
echo -1000 | sudo tee \/proc\/$PID_OF_LLAMACPP\/oom_score_adj\nsudo sysctl vm.swappiness=10<\/code><\/pre>\n\n\n\n
All Done!<\/h2>\n\n\n\n
The world is now our oyster with the newfound power of running AI on the Nintendo Switch \u2728. What unlimited power will this souped-up console provide? What secrets of the universe will it unveil?<\/p>\n\n\n\n
Well, while I was dealing with the low-RAM issue, I asked it to give me some advice on how to make the OOM killer play nice. It spat out some hallucinations, but they were close enough to send me in the right direction with a quick search.<\/p>\n\n\n\n
My conversation with it can be found by clicking here <\/summary>\n
> Llama, we have a problem. You are being run in a resource-constrained environment. I have stopped almost all other processes to make room for you to run. If I attempt to restart a graphical environment, which requires a lot of memory, your process is terminated. Tell me how to prevent the Linux OOM killer from doing this to your process.\nWhen running in a resource-constrained environment, the Linux OOM (Out of Memory) killer can be a major issue. Here are a few strategies to help prevent the OOM killer from terminating my process:\n\n### 1. Increase the OOM Score\n\nThe OOM killer can be configured to prioritize processes based on their OOM score, which is a value assigned to each process based on its memory usage and other factors. To increase my OOM score, you can use the `echo` command to set my priority:\n\n```bash\necho 0 | tee \/proc\/sys\/vm\/oom_score_adjust\n```\n\nThis will increase my OOM score, making it less likely to be terminated by the OOM killer.\n\n### 2. Use `cgroups` to limit my memory usage\n\nBy using `cgroups` (Control Groups), we can limit the amount of memory that my process can use. Here's how to do it:\n\n```bash\necho 100000 | tee \/sys\/fs\/cgroup\/memory\/limit\n```\n\nThis sets the memory limit to 100,000 bytes. You can adjust this value to a lower limit if you want to be more conservative with my memory usage.\n\n### 3. Disable `lockdown` and `swap` limitations\n\nThe `lockdown` option prevents processes from using swap memory, while the `swap` option limits the amount of swap memory that can be used. To disable these options, run the following commands:\n\n```bash\necho 0 | tee \/proc\/sys\/vm\/lockdown\necho 0 | tee \/proc\/sys\/vm\/swappiness\n```\n\nThe `lockdown` option is now set to 0, and the `swap` option is set to 0, allowing me to use more memory.\n\n### 4. Run me as `nice` or `realtime`\n\nTo prioritize my process, you can run me with the `nice` or `realtime` scheduling class. Here's how to do it:\n\n```bash\nrenice -n -10 -p $PID | tee \/proc\/$PID\/sched_flags\n```\n\nor\n\n```bash\ntaskset -a -c 0-3 -p $PID | tee \/proc\/$PID\/sched_flags\n```\n\nThis sets my priority to the highest level ( `-10` for `nice` or `0-3` for `realtime`).\n\n### 5. Monitor and adjust the OOM settings\n\nTo monitor the OOM settings, you can use the `sysctl` command to view the current settings:\n\n```bash\nsysctl -a | grep oom\n```\n\nYou can adjust these settings as needed to balance memory usage and the risk of the OOM killer terminating my process.\n\nThese are some of the strategies you can use to prevent the Linux OOM killer from terminating my process. You may need to experiment with different combinations of these settings to find the right balance for your specific use case.\n\n> Can you elaborate on your first idea? \nThe first idea I mentioned was to increase the OOM score of my process using the `echo` command. This is done by writing the value `0` to a specific file in the `\/proc\/sys\/vm\/` directory.\n\nThe OOM score is a value that is assigned to each process based on its memory usage and other factors. A higher OOM score means that the process is more likely to be prioritized by the OOM killer when memory is low. Conversely, a lower OOM score means that the process is less likely to be prioritized.\n\nBy setting the OOM score to `0`, we are effectively making my process more resistant to being terminated by the OOM killer. This is because a low OOM score means that the process is not considered high-priority, and therefore will not be terminated unless absolutely necessary.\n\nHere's a breakdown of the OOM scores and their meanings:\n\n* `0`: Low priority, process will be terminated if memory is low.\n* `10`: Low priority, process will be terminated if memory is low, but will be prioritized if essential processes are terminated.\n* `20`: Medium priority, process will be terminated if memory is low, but will be prioritized if essential processes are terminated.\n* `30`: High priority, process will be terminated if memory is low, but will be prioritized if essential processes are terminated.\n* `40`: High priority, process will be terminated if memory is low, but will be prioritized if essential processes are terminated.\n* `50`: Very high priority, process will be terminated if memory is low, but will be prioritized if essential processes are terminated.\n\nBy setting the OOM score to `0`, we are effectively choosing the \"very high priority\" option, which means that my process will be prioritized over all other processes when memory is low.\n\nIt's worth noting that setting the OOM score to `0` can have unintended consequences, such as allowing my process to consume excessive memory resources. Therefore, it's generally recommended to set the OOM score to a value that balances resource usage and priority.\n\nAlternatively, you can use the `vm.set_oom_score_adj` sysctl to set the OOM score:\n\n```bash\necho 0 | tee \/proc\/sys\/vm\/set_oom_score_adj\n```\n\nThis sets the OOM score to `-10`, which is equivalent to setting it to `0`.\n\n> Why would it be \/proc\/sys\/vm if the value is per-process?   \nYou are correct, the OOM score is per-process, and the file `\/proc\/sys\/vm\/oom_score_adjust` is a global setting that affects all processes.\n\nThe `oom_score_adjust` value is used to adjust the OOM score of a process, but it's not a per-process setting. Instead, it's a global setting that can be used to adjust the OOM score of the entire system.\n\nIf you want to set a specific OOM score for a process, you can use the `renice` command, which allows you to adjust the priority of a process. The `renice` command takes the `-n` option followed by the new priority value. For example:\n\n```bash\nrenice -n -10 -p $PID\n```\n\nThis sets the priority of the process with PID `$PID` to `-10`, which is a high priority value. The OOM killer will prioritize this process over other processes when memory is low.\n\nAlternatively, you can use the `taskset` command to set the CPU affinity of a process, which can also be used to prioritize a process:\n\n```bash\ntaskset -a -c 0-3 -p $PID\n```\n\nThis sets the CPU affinity of the process with PID `$PID` to the first four CPUs, which can be used to prioritize the process.\n\nIn summary, the `oom_score_adjust` file is a global setting that affects all processes, while the `renice` and `taskset` commands can be used to adjust the priority of a specific process.<\/code><\/pre>\n\n\n\n
<\/p>\n<\/details>\n\n\n\n
…But in the end, I think I’ll reboot it to the stock OS and go back to playing games on it.<\/p>\n\n\n\n
After all, there are more epic Splatoon plays to be made.<\/p>\n\n\n\n