Deployment of NVIDIA Nemotron-3-Nano

NVIDIA Nemotron-3-Nano-4B is a compact large language model developed by NVIDIA for agentic AI tasks. It runs on GPUs using llama.cpp with full CUDA acceleration, enabling fast text generation and reasoning on a single Tesla T4 GPU. It exposes an OpenAI-compatible REST API, making it easy to integrate into any application.

Minimum Requirements for Nemotron-3-Nano-4B

Hardware Requirements

Component

Minimum

Recommended

GPU

NVIDIA Tesla T4 (16GB VRAM)

A30 / RTX 4090 (24GB VRAM)

CUDA Compute Capability

>= 7.5

>= 8.0

VRAM

16GB

24GB+

System RAM

16GB

32GB+

Storage

20GB free

40GB+

Software Requirements

Ubuntu 24.04 (NeevCloud Ubuntu - Tesla T4 GPU image)
NVIDIA Driver >= 570
CUDA Toolkit 12.8
Python 3.12
HuggingFace Hub CLI (hf)
llama.cpp (built from source with CUDA)
tmux

Step 1: Create a Server on NeevCloud

Configure the server with the following settings:

Field

Value

Location

Central India

Image

GPU Operating System > Ubuntu - Tesla T4

Hardware

GPU Enabled > Select a T4 flavor

Networking

Public

SSH Key

Select your existing SSH key

Click Create Server and wait for the status to show Active.

Step 2: Connect to the Server

Open a terminal or PowerShell and SSH into the server using the public IP shown on the server overview page.

ssh root@<your-server-ip>

You will see the Ubuntu 24.04 welcome prompt once connected.

Step 3: Verify the GPU

Run the following command to confirm the Tesla T4 is detected and the NVIDIA driver is active.

nvidia-smi

If the GPU details appear, continue to the next step. If not, the server may not have booted with the GPU image correctly. Contact NeevCloud support.

Step 4: Install System Dependencies

Update the package list and install all required build tools.

sudo apt update && sudo apt install -y \
  git \
  cmake \
  build-essential \
  curl \
  libcurl4-openssl-dev \
  python3-pip \
  tmux \
  wget

This installs the compiler toolchain, cmake build system, and supporting libraries required to build llama.cpp from source.

Step 5: Install HuggingFace CLI

Install the HuggingFace Hub CLI to download the model from the official NVIDIA repository.

pip install -U huggingface_hub hf_transfer \
  --break-system-packages --ignore-installed rich

Verify the installation. Note that the new CLI command is hf, not huggingface-cli.

hf --help

If you see a deprecation warning for huggingface-cli, this is expected. Use hf for all subsequent commands.

Step 6: Locate the CUDA Compiler

The NeevCloud Tesla T4 image ships with CUDA 12.8. The nvcc compiler must be found and added to PATH before building llama.cpp.

find /usr/local/cuda* -name "nvcc" 2>/dev/null

Expected output:

/usr/local/cuda-12.8/bin/nvcc

Set the required environment variables:

export CUDACXX=/usr/local/cuda-12.8/bin/nvcc
export PATH=/usr/local/cuda-12.8/bin:$PATH

Verify the compiler is accessible:

nvcc --version

These exports apply to the current SSH session only. They are made permanent in Step 9 via the systemd service configuration.

Step 7: Build llama.cpp with CUDA Support

Clone the llama.cpp repository and compile it with CUDA enabled. This step takes approximately 8 to 10 minutes.

Clone the repository:

git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp

Configure the build:

cmake /opt/llama.cpp -B /opt/llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc \
  -DCMAKE_BUILD_TYPE=Release

A successful configuration ends with:

-- Configuring done
-- Build files have been written to: /opt/llama.cpp/build

If you see No CMAKE_CUDA_COMPILER could be found, the environment variables from Step 6 were not set. Re-run those exports and retry.

Compile the server and CLI binaries:

cmake --build /opt/llama.cpp/build \
  --config Release \
  -j$(nproc) \
  --target llama-server llama-cli

Copy binaries to the main directory:

cp /opt/llama.cpp/build/bin/llama-* /opt/llama.cpp/

Verify the build:

ls -lh /opt/llama.cpp/llama-server

Step 8: Download the Nemotron-3-Nano-4B Model

Create the model directory and download the Q4_K_M quantized GGUF from the official NVIDIA organization on HuggingFace.

mkdir -p /opt/nemotron

HF_XET_HIGH_PERFORMANCE=1 hf download \
  nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF \
  --include "*Q4_K_M*" \
  --local-dir /opt/nemotron

Quantization

File Size

VRAM Used

Quality

Q4_K_M

2.7 GB

~3 GB

Good — recommended for T4

Q5_K_M

3.3 GB

~4 GB

Better

Q8_0

5.0 GB

~5.5 GB

Near-lossless

Verify the downloaded file:

ls -lh /opt/nemotron/

Expected output:

-rw-r--r-- 1 root root 2.7G  NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf

Step 9: Launch the Model Server

Start a tmux session so the server continues running after you disconnect from SSH.

tmux new -s nemotron

Launch llama-server inside the tmux session:

/opt/llama.cpp/llama-server \
  --model /opt/nemotron/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
  --alias "nemotron-nano-4b" \
  --port 8080 \
  --host 0.0.0.0 \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --temp 0.6 \
  --top-p 0.95

Flag

Description

--model

Path to the downloaded GGUF file

--alias

Name used when calling the API

--port 8080

Port the API listens on

--host 0.0.0.0

Accept connections from any IP

--ctx-size 8192

Maximum context window per session (~6000 words)

--n-gpu-layers 99

Offload all layers to GPU for full acceleration

--temp 0.6

Temperature — 0.6 recommended by NVIDIA for chat

--top-p 0.95

Top-p sampling as recommended in NVIDIA documentation

The server is ready when you see:

llama_server: listening on http://0.0.0.0:8080

Detach from tmux to keep the server running in the background:

Press Ctrl+B then D

Step 10: Verify the GPU is Being Used

Open a new SSH session and run:

nvidia-smi

The llama-server process should appear in the Processes section, with approximately 3000 MiB of VRAM consumed.

Step 11: Test the API

Send a test request from the server itself to confirm inference is working.

Basic chat test:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-nano-4b",
    "messages": [
      {"role": "system", "content": "detailed thinking off"},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100
  }'

A successful response will contain:

json

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "Paris."
    },
    "finish_reason": "stop"
  }]
}

Test from an external machine:

Replace <your-server-ip> with the public IP from the NeevCloud dashboard.

curl http://<your-server-ip>:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-nano-4b",
    "messages": [
      {"role": "user", "content": "Hello, are you working?"}
    ],
    "max_tokens": 100
  }'

Step 12: Configure Auto-Start on Reboot (Systemd)

To ensure the model server starts automatically whenever the VM reboots, create a systemd service.

cat > /etc/systemd/system/nemotron.service << 'EOF'
[Unit]
Description=Nemotron-3-Nano-4B LLM Server
After=network.target

[Service]
Type=simple
ExecStart=/opt/llama.cpp/llama-server \
  --model /opt/nemotron/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf \
  --alias nemotron-nano-4b \
  --port 8080 \
  --host 0.0.0.0 \
  --ctx-size 8192 \
  --n-gpu-layers 99 \
  --temp 0.6 \
  --top-p 0.95
Restart=always
RestartSec=15
Environment=CUDA_VISIBLE_DEVICES=0
Environment=PATH=/usr/local/cuda-12.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[Install]
WantedBy=multi-user.target
EOF

Enable and start the service:

systemctl daemon-reload
systemctl enable nemotron
systemctl start nemotron

Check the service status:

systemctl status nemotron

View live logs:

journalctl -u nemotron -f

Reasoning Mode Control

Nemotron-3-Nano-4B supports toggling chain-of-thought reasoning via the system prompt. This is an official NVIDIA feature.

Reasoning OFF — faster responses, direct answers:

json

{"role": "system", "content": "detailed thinking off"}

Reasoning ON — higher accuracy, model shows its thinking process:

json

{"role": "system", "content": "detailed thinking on"}

Enabling reasoning generally produces higher-quality answers at the cost of more tokens generated. The reasoning_content field in the response contains the internal thinking trace.

Performance Reference

Observed performance on NeevCloud Tesla T4 with Q4_K_M quantization:

Metric

Value

Prompt evaluation speed

111 to 222 tokens/second

Generation speed

63 tokens/second

Total response time

~3.4 seconds for 232 tokens

VRAM consumed

~3000 MiB of 15360 MiB

Concurrent request slots

Context window

8192 tokens

GPU layer offload

99 of 99 (full GPU)

Troubleshooting

cmake fails with "No CMAKE_CUDA_COMPILER could be found"

The CUDA compiler path was not exported. Run the following and retry cmake:

export CUDACXX=/usr/local/cuda-12.8/bin/nvcc
export PATH=/usr/local/cuda-12.8/bin:$PATH
rm -rf /opt/llama.cpp/build

pip install fails with "Cannot uninstall rich"

Add the --ignore-installed flag:

pip install -U huggingface_hub hf_transfer \
  --break-system-packages --ignore-installed rich

huggingface-cli command not found or deprecated

The new CLI name is hf. Use hf download instead of huggingface-cli download.

Server stops when SSH session closes

Always run the server inside a tmux session. To reattach to an existing session:

tmux attach -t nemotron

Alternatively, use the systemd service from Step 12 which survives disconnects and reboots.

Model responds with "I am Qwen" instead of Nemotron

This is known behavior. Nemotron-3-Nano-4B was trained on data derived from Qwen. Override the identity using the system prompt:

json

{"role": "system", "content": "detailed thinking off. You are Nemotron, an AI assistant built by NVIDIA."}

Ensure max_tokens is set to at least 200 to allow a complete response.

API returns connection refused from external machine

Confirm port 8080 is open in your NeevCloud Security Group. Navigate to Cloud > Networking > Security Groups and add an inbound rule for TCP port 8080.