> For the complete documentation index, see [llms.txt](https://docs.neevcloud.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.neevcloud.com/neevcloud-guide/neevcloud-knowledgebase/deployment-of-nvidia-nemotron-3-nano.md).

# Deployment of NVIDIA Nemotron-3-Nano

NVIDIA Nemotron-3-Nano-4B is a compact large language model developed by NVIDIA for agentic AI tasks. It runs on GPUs using llama.cpp with full CUDA acceleration, enabling fast text generation and reasoning on a single Tesla T4 GPU. It exposes an OpenAI-compatible REST API, making it easy to integrate into any application.

***

### Minimum Requirements for Nemotron-3-Nano-4B

#### Hardware Requirements

| Component               | Minimum                     | Recommended                |
| ----------------------- | --------------------------- | -------------------------- |
| GPU                     | NVIDIA Tesla T4 (16GB VRAM) | A30 / RTX 4090 (24GB VRAM) |
| CUDA Compute Capability | >= 7.5                      | >= 8.0                     |
| VRAM                    | 16GB                        | 24GB+                      |
| System RAM              | 16GB                        | 32GB+                      |
| Storage                 | 20GB free                   | 40GB+                      |

#### Software Requirements

* Ubuntu **24.04** (NeevCloud **Ubuntu - Tesla T4** GPU image)
* NVIDIA Driver **>= 570**
* CUDA Toolkit **12.8**
* Python **3.12**
* HuggingFace Hub CLI (`hf`)
* llama.cpp (built from source with CUDA)
* tmux

***

### Step 1: Create a Server on NeevCloud

Log in to the NeevCloud dashboard and navigate to **Cloud > Servers > Create Server**.

Configure the server with the following settings:

| Field      | Value                                    |
| ---------- | ---------------------------------------- |
| Location   | Central India                            |
| Image      | GPU Operating System > Ubuntu - Tesla T4 |
| Hardware   | GPU Enabled > Select a T4 flavor         |
| Networking | Public                                   |
| SSH Key    | Select your existing SSH key             |

Click **Create Server** and wait for the status to show **Active**.<br>

<figure><img src="/files/onjpXAFjsYecaOl3suLv" alt=""><figcaption></figcaption></figure>

***

### Step 2: Connect to the Server

Open a terminal or PowerShell and SSH into the server using the public IP shown on the server overview page.

```
ssh root@<your-server-ip>
```

You will see the Ubuntu 24.04 welcome prompt once connected.

***

### Step 3: Verify the GPU

Run the following command to confirm the Tesla T4 is detected and the NVIDIA driver is active.

```
nvidia-smi
```

<figure><img src="/files/HdijdGjIDMAONnQRsMl9" alt=""><figcaption></figcaption></figure>

If the GPU details appear, continue to the next step. If not, the server may not have booted with the GPU image correctly. Contact NeevCloud support.

***

### Step 4: Install System Dependencies

Update the package list and install all required build tools.

```
sudo apt update && sudo apt install -y \
  git \
  cmake \
  build-essential \
  curl \
  libcurl4-openssl-dev \
  python3-pip \
  tmux \
  wget
```

This installs the compiler toolchain, cmake build system, and supporting libraries required to build llama.cpp from source.

***

### Step 5: Install HuggingFace CLI

Install the HuggingFace Hub CLI to download the model from the official NVIDIA repository.

```
pip install -U huggingface_hub hf_transfer \
  --break-system-packages --ignore-installed rich
```

Verify the installation. Note that the new CLI command is `hf`, not `huggingface-cli`.

```
hf --help
```

> If you see a deprecation warning for `huggingface-cli`, this is expected. Use `hf` for all subsequent commands.

***

### Step 6: Locate the CUDA Compiler

The NeevCloud Tesla T4 image ships with CUDA 12.8. The `nvcc` compiler must be found and added to PATH before building llama.cpp.

```
find /usr/local/cuda* -name "nvcc" 2>/dev/null
```

Expected output:

```
/usr/local/cuda-12.8/bin/nvcc
```

Set the required environment variables:

```
export CUDACXX=/usr/local/cuda-12.8/bin/nvcc
export PATH=/usr/local/cuda-12.8/bin:$PATH
```

Verify the compiler is accessible:

```
nvcc --version
```

> These exports apply to the current SSH session only. They are made permanent in Step 9 via the systemd service configuration.

***

### Step 7: Build llama.cpp with CUDA Support

Clone the llama.cpp repository and compile it with CUDA enabled. This step takes approximately 8 to 10 minutes.

**Clone the repository:**

```
git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
```

**Configure the build:**

```
cmake /opt/llama.cpp -B /opt/llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc \
  -DCMAKE_BUILD_TYPE=Release
```

A successful configuration ends with:

```
-- Configuring done
-- Build files have been written to: /opt/llama.cpp/build
```

> If you see `No CMAKE_CUDA_COMPILER could be found`, the environment variables from Step 6 were not set. Re-run those exports and retry.

**Compile the server and CLI binaries:**

```
cmake --build /opt/llama.cpp/build \
  --config Release \
  -j$(nproc) \
  --target llama-server llama-cli
```

**Copy binaries to the main directory:**

```
cp /opt/llama.cpp/build/bin/llama-* /opt/llama.cpp/
```

**Verify the build:**

```
ls -lh /opt/llama.cpp/llama-server
```

***

### Step 8: Download the Nemotron-3-Nano-4B Model

Create the model directory and download the Q4\_K\_M quantized GGUF from the official NVIDIA organization on HuggingFace.

```
mkdir -p /opt/nemotron

HF_XET_HIGH_PERFORMANCE=1 hf download \
  nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF \
  --include "*Q4_K_M*" \
  --local-dir /opt/nemotron
```

| Quantization | File Size | VRAM Used | Quality                   |
| ------------ | --------- | --------- | ------------------------- |
| Q4\_K\_M     | 2.7 GB    | \~3 GB    | Good — recommended for T4 |
| Q5\_K\_M     | 3.3 GB    | \~4 GB    | Better                    |
| Q8\_0        | 5.0 GB    | \~5.5 GB  | Near-lossless             |

Verify the downloaded file:

```
ls -lh /opt/nemotron/
```

Expected output:

```
-rw-r--r-- 1 root root 2.7G  NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf
```

***

### Step 9: Launch the Model Server

Start a tmux session so the server continues running after you disconnect from SSH.

```
tmux new -s nemotron
```

Launch llama-server inside the tmux session:

```
/opt/llama.cpp/llama-server \
  --model /opt/nemotron/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
  --alias "nemotron-nano-4b" \
  --port 8080 \
  --host 0.0.0.0 \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --temp 0.6 \
  --top-p 0.95
```

| Flag                | Description                                           |
| ------------------- | ----------------------------------------------------- |
| `--model`           | Path to the downloaded GGUF file                      |
| `--alias`           | Name used when calling the API                        |
| `--port 8080`       | Port the API listens on                               |
| `--host 0.0.0.0`    | Accept connections from any IP                        |
| `--ctx-size 8192`   | Maximum context window per session (\~6000 words)     |
| `--n-gpu-layers 99` | Offload all layers to GPU for full acceleration       |
| `--temp 0.6`        | Temperature — 0.6 recommended by NVIDIA for chat      |
| `--top-p 0.95`      | Top-p sampling as recommended in NVIDIA documentation |

The server is ready when you see:

```
llama_server: listening on http://0.0.0.0:8080
```

Detach from tmux to keep the server running in the background:

```
Press Ctrl+B then D
```

***

### Step 10: Verify the GPU is Being Used

Open a new SSH session and run:

```
nvidia-smi
```

<figure><img src="/files/wf6TGjyBWm4Yra1SWG0h" alt=""><figcaption></figcaption></figure>

The llama-server process should appear in the Processes section, with approximately 3000 MiB of VRAM consumed.

***

### Step 11: Test the API

Send a test request from the server itself to confirm inference is working.

**Basic chat test:**

```
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-nano-4b",
    "messages": [
      {"role": "system", "content": "detailed thinking off"},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100
  }'
```

A successful response will contain:

json

```json
{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "Paris."
    },
    "finish_reason": "stop"
  }]
}
```

**Test from an external machine:**

Replace `<your-server-ip>` with the public IP from the NeevCloud dashboard.

```
curl http://<your-server-ip>:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-nano-4b",
    "messages": [
      {"role": "user", "content": "Hello, are you working?"}
    ],
    "max_tokens": 100
  }'
```

***

### Step 12: Configure Auto-Start on Reboot (Systemd)

To ensure the model server starts automatically whenever the VM reboots, create a systemd service.

```
cat > /etc/systemd/system/nemotron.service << 'EOF'
[Unit]
Description=Nemotron-3-Nano-4B LLM Server
After=network.target

[Service]
Type=simple
ExecStart=/opt/llama.cpp/llama-server \
  --model /opt/nemotron/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
  --alias nemotron-nano-4b \
  --port 8080 \
  --host 0.0.0.0 \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --temp 0.6 \
  --top-p 0.95
Restart=always
RestartSec=15
Environment=CUDA_VISIBLE_DEVICES=0
Environment=PATH=/usr/local/cuda-12.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[Install]
WantedBy=multi-user.target
EOF
```

Enable and start the service:

```
systemctl daemon-reload
systemctl enable nemotron
systemctl start nemotron
```

Check the service status:

```
systemctl status nemotron
```

View live logs:

```
journalctl -u nemotron -f
```

***

### Reasoning Mode Control

Nemotron-3-Nano-4B supports toggling chain-of-thought reasoning via the system prompt. This is an official NVIDIA feature.

**Reasoning OFF** — faster responses, direct answers:

json

```json
{"role": "system", "content": "detailed thinking off"}
```

**Reasoning ON** — higher accuracy, model shows its thinking process:

json

```json
{"role": "system", "content": "detailed thinking on"}
```

> Enabling reasoning generally produces higher-quality answers at the cost of more tokens generated. The `reasoning_content` field in the response contains the internal thinking trace.

***

### Performance Reference

Observed performance on NeevCloud Tesla T4 with Q4\_K\_M quantization:

| Metric                   | Value                        |
| ------------------------ | ---------------------------- |
| Prompt evaluation speed  | 111 to 222 tokens/second     |
| Generation speed         | 63 tokens/second             |
| Total response time      | \~3.4 seconds for 232 tokens |
| VRAM consumed            | \~3000 MiB of 15360 MiB      |
| Concurrent request slots | 4                            |
| Context window           | 8192 tokens                  |
| GPU layer offload        | 99 of 99 (full GPU)          |

***

### Troubleshooting

**cmake fails with "No CMAKE\_CUDA\_COMPILER could be found"**

The CUDA compiler path was not exported. Run the following and retry cmake:

```
export CUDACXX=/usr/local/cuda-12.8/bin/nvcc
export PATH=/usr/local/cuda-12.8/bin:$PATH
rm -rf /opt/llama.cpp/build
```

**pip install fails with "Cannot uninstall rich"**

Add the `--ignore-installed` flag:

```
pip install -U huggingface_hub hf_transfer \
  --break-system-packages --ignore-installed rich
```

**`huggingface-cli` command not found or deprecated**

The new CLI name is `hf`. Use `hf download` instead of `huggingface-cli download`.

**Server stops when SSH session closes**

Always run the server inside a tmux session. To reattach to an existing session:

```
tmux attach -t nemotron
```

Alternatively, use the systemd service from Step 12 which survives disconnects and reboots.

**Model responds with "I am Qwen" instead of Nemotron**

This is known behavior. Nemotron-3-Nano-4B was trained on data derived from Qwen. Override the identity using the system prompt:

json

```json
{"role": "system", "content": "detailed thinking off. You are Nemotron, an AI assistant built by NVIDIA."}
```

Ensure `max_tokens` is set to at least 200 to allow a complete response.

**API returns connection refused from external machine**

Confirm port 8080 is open in your NeevCloud Security Group. Navigate to **Cloud > Networking > Security Groups** and add an inbound rule for TCP port 8080.

***

### Official Resources

| Resource                      | URL                                                            |
| ----------------------------- | -------------------------------------------------------------- |
| NVIDIA Nemotron Research Page | <https://research.nvidia.com/labs/nemotron/Nemotron-3/>        |
| Official Model on HuggingFace | <https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF> |
| NVIDIA Nemotron v3 Collection | <https://huggingface.co/collections/nvidia/nvidia-nemotron-v3> |
| llama.cpp Repository          | <https://github.com/ggml-org/llama.cpp>                        |
| NVIDIA Blog Post              | <https://huggingface.co/blog/nvidia/nemotron-3-nano-4b>        |