Nvidia 3060 显卡 CUDA 环境搭建(Ubuntu22.04+Nvidia 510+Cuda11.6+cudnn8.8)

对每个人而言,真正的职责只有一个:找到自我。然后在心中坚守其一生,全心全意,永不停息。所有其它的路都是不完整的,是人的逃避方式,是对大众理想的懦弱回归,是随波逐流,是对内心的恐惧 ——赫尔曼·黑塞《德米安》

写在前面


  • 工作中遇到,简单整理
  • 理解不足小伙伴帮忙指正

对每个人而言,真正的职责只有一个:找到自我。然后在心中坚守其一生,全心全意,永不停息。所有其它的路都是不完整的,是人的逃避方式,是对大众理想的懦弱回归,是随波逐流,是对内心的恐惧 ——赫尔曼·黑塞《德米安》


当前系统环境

系统环境

1
2
3
4
5
6
7
8
9
10
11
12
┌──[root@test]-[~]
└─$hostnamectl
Static hostname: test
Icon name: computer-desktop
Chassis: desktop
Machine ID: addc7ca21ef24518a9465c499eb3c8b7
Boot ID: 14aa59cc6960431c95d328684b521844
Operating System: Ubuntu 22.04.2 LTS
Kernel: Linux 5.19.0-43-generic
Architecture: x86-64
Hardware Vendor: Micro-Star International Co., Ltd.
Hardware Model: MS-7C83

显卡版本

1
2
3
4
5
┌──[root@test]-[~]
└─$lspci -vnn | grep VGA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] [10de:2504] (rev a1) (prog-if 00 [VGA controller])
┌──[root@test]-[~]
└─$

安装 NVIDIA 驱动,在安装之前,需要禁用 Nouveau 驱动程序。

Nouveau 是一个开源的NVIDIA显卡驱动程序,它由社区开发和维护。它可以在Linux系统上替代NVIDIA官方驱动程序,但它的性能和功能可能不如官方驱动程序。

如果使用 Nouveau 驱动程序,可能无法使用NVIDIA的高级功能,如CUDA和深度学习库。如果您需要使用这些功能,建议安装NVIDIA官方驱动程序。

禁用 Nouveau 驱动程序

1
2
3
4
5
6
7
8
9
┌──[root@test]-[~]
└─$sudo vim /etc/modprobe.d/blacklist-nouveau.conf
┌──[root@test]-[~]
└─$cat /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
options nouveau modeset=0
┌──[root@test]-[~]
└─$sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.19.0-43-generic

重启完机器,没有输出说明操作成功

1
2
3
4
5
6
┌──[root@test]-[~]
└─$reboot
┌──[root@test]-[~]
└─$lsmod | grep nouveau
┌──[root@test]-[~]
└─$

安装Nvidia驱动

这里的版本 nvidia-driver-510 要和后面安装 cuda 的版本一样

如果之前安装过卸载驱动

1
2
3
4
# 查看显卡型号
lspci -vnn | grep VGA
# 卸载旧驱动
sudo apt-get remove --purge nvidia*

离线安装

如果离线环境需要手动安装,下载驱动: https://www.nvidia.com/Download/index.aspx?lang=en-us

1
2
3
4
5
6
7
# 给run文件可执行权限 
sudo chmod a+x NVIDIA-Linux-x86_64-515.86.01.run
# 安装
sudo ./NVIDIA-Linux-x86_64-440.64.run -no-x-check -no-nouveau-check -no-opengl-files
# -no-x-check:安装驱动时关闭X服务
# -no-nouveau-check:安装驱动时禁用nouveau
# -no-opengl-files:只安装驱动文件,不安装OpenGL文件

非离线安装

非离线环境使用包管理工具安装,下面的选择这一种,选择安装驱动版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌──[root@test]-[~]
└─$ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00002504sv00001462sd0000397Dbc03sc00i00
vendor : NVIDIA Corporation
model : GA106 [GeForce RTX 3060 Lite Hash Rate]
driver : nvidia-driver-530-open - distro non-free
driver : nvidia-driver-470 - distro non-free
driver : nvidia-driver-525-open - third-party non-free
driver : nvidia-driver-535 - third-party non-free
driver : nvidia-driver-520 - third-party non-free
driver : nvidia-driver-510 - distro non-free
driver : nvidia-driver-525 - third-party non-free
driver : nvidia-driver-515-server - distro non-free
driver : nvidia-driver-535-open - third-party non-free recommended
driver : nvidia-driver-530 - third-party non-free
driver : nvidia-driver-470-server - distro non-free
driver : nvidia-driver-515-open - distro non-free
driver : nvidia-driver-525-server - distro non-free
driver : nvidia-driver-515 - third-party non-free
driver : xserver-xorg-video-nouveau - distro free builtin

┌──[root@test]-[~]
└─$

安装

1
2
┌──[root@test]-[~]
└─$sudo apt install nvidia-driver-510 -y

重启机器

1
2
┌──[root@test]-[~]
└─$reboot

查看安装是否成功,对应版本信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
┌──[root@test]-[~]
└─$nvidia-smi
Thu Jun 15 11:49:43 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 0% 38C P8 16W / 170W | 172MiB / 12288MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1386 G /usr/lib/xorg/Xorg 60MiB |
| 0 N/A N/A 1650 G /usr/bin/gnome-shell 109MiB |
+-----------------------------------------------------------------------------+
┌──[root@test]-[~]
└─$cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 510.108.03 Thu Oct 20 05:10:45 UTC 2022
GCC version: gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)
┌──[root@test]-[~]
└─$

安装Cuda

CUDA是NVIDIA提供的一种并行计算平台和编程模型,旨在利用GPU的并行计算能力加速计算密集型应用程序。

CUDA包括CUDA驱动程序和CUDA Toolkit。支持多种编程语言,包括C、C++、Fortran和Python等。

  • CUDA驱动程序是GPU和操作系统之间的接口.
  • CUDA Toolkit则包括编译器、库和工具,用于开发CUDA应用程序。

如果以前安装过,卸载

1
2
sudo /usr/local/cuda-11.6/bin/cuda-uninstaller
sudo rm -rf /usr/local/cuda-11.6
1
2
3
4
5
6
7
8
9
10
11
sudo: /usr/local/cuda-11.8/bin/uninstall_cuda_8.0.pl: command not found
┌──[root@test]-[~]
└─$sudo /usr/local/cuda-11.6/bin/
bin2c cuda-gdbserver ncu nsys-ui nv-nsight-cu-cli
computeprof cuda-memcheck ncu-ui nvcc nvprof
compute-sanitizer cuda-uninstaller nsight_ee_plugins_manage.sh __nvcc_device_query nvprune
crt/ cu++filt nsight-sys nvdisasm nvvp
cudafe++ cuobjdump nsys nvlink ptxas
cuda-gdb fatbinary nsys-exporter nv-nsight-cu
┌──[root@test]-[~]
└─$sudo /usr/local/cuda-11.6/bin/cuda-uninstaller

在输出的终端 UI页面,空格选择全部,选择完成,卸载完成之后重新安装

1
2
3
4
5
┌──[root@test]-[~]
└─$sudo /usr/local/cuda-11.6/bin/cuda-uninstaller
Successfully uninstalled
┌──[root@test]-[~]
└─$sudo rm -rf /usr/local/cuda-11.6

官网安装

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=runfile_local

1
2
┌──[root@test]-[~]
└─$chmod +x cuda_*

这里cuda 选择 cuda_11.6.0_510.39.01_linux.run, 510 对应的版本

1
2
3
4
5
6
7
8
9
10
┌──[root@test]-[~]
└─$ll cuda*
-rwxr-xr-x 1 root root 3488951771 1月 11 2022 cuda_11.6.0_510.39.01_linux.run*
-rwxr-xr-x 1 root root 3490450898 5月 5 2022 cuda_11.7.0_515.43.04_linux.run*
-rwxr-xr-x 1 root root 4317456991 4月 17 23:04 cuda_12.1.1_530.30.02_linux.run*
-rwxr-xr-x 1 root root 853 5月 17 19:52 cuda_log.log*
-rw-r--r-- 1 root root 2472241638 7月 29 2021 cuda-repo-ubuntu2004-11-4-local_11.4.1-470.57.02-1_amd64.deb
-rw-r--r-- 1 root root 2699477842 5月 5 2022 cuda-repo-ubuntu2204-11-7-local_11.7.0-515.43.04-1_amd64.deb
┌──[root@test]-[~]
└─$
1
2
┌──[root@test]-[~]
└─$sudo ./cuda_12.1.1_530.30.02_linux.run

上面我们已经安装了驱动,所以不需要选择,直接安装 cuda 相关的就可以,安装成功输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@test]-[~]
└─$sudo ./cuda_11.6.0_510.39.01_linux.run
===========
= Summary =
===========

Driver: Not Selected
Toolkit: Installed in /usr/local/cuda-11.6/

Please make sure that
- PATH includes /usr/local/cuda-11.6/bin
- LD_LIBRARY_PATH includes /usr/local/cuda-11.6/lib64, or, add /usr/local/cuda-11.6/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-11.6/bin
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 510.00 is required for CUDA 11.6 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
sudo <CudaInstaller>.run --silent --driver

Logfile is /var/log/cuda-installer.log

添加环境变量

1
2
3
4
5
6
7
8
┌──[root@test]-[/b1205]
└─$echo $LD_LIBRARY_PATH
/usr/local/cuda-11.6/lib64:/usr/local/cuda-11.6/lib64
┌──[root@test]-[/b1205]
└─$echo $PATH
/usr/local/cuda-11.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
┌──[root@test]-[/b1205]
└─$

安装 cuDNN

cuDNN 是NVIDIA提供的一个用于深度神经网络的加速库,它可以优化卷积、池化、归一化等操作,使得在GPU上运行深度神经网络的速度得到了大幅度提升。cuDNN需要与CUDA配合使用,因此在安装cuDNN之前,需要先安装相应版本的CUDA。

https://developer.nvidia.com/rdp/cudnn-download

这里需要注册账户登录一下,然后在这里下载

https://developer.nvidia.com/rdp/cudnn-archive

选择cuda对应的版本

在这里插入图片描述

1
2
3
┌──[root@test]-[~]
└─$ls cudnn*
cudnn-local-repo-ubuntu2204-8.8.1.3_1.0-1_amd64.deb
1
sudo dpkg -i cudnn-local-repo-ubuntu2204-8.8.1.3_1.0-1_amd64.deb
1
2
3
4
5
sudo cp /var/cudnn-local-repo-*/cudnn-local-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudoapt-get install libcudnn8=8.8.1.3-1+cuda1
sudo apt-get install libcudnn8-dev=8.8.1.3-1+cuda1
sudo apt-get install libcudnn8-samples=8.8.1.3-1+cuda1

确实安装是否成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
┌──[root@test]-[~]
└─$nvcc -V && nvidia-smi
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0
Thu Jun 15 14:42:58 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 0% 51C P8 21W / 170W | 105MiB / 12288MiB | 12% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1386 G /usr/lib/xorg/Xorg 81MiB |
| 0 N/A N/A 1650 G /usr/bin/gnome-shell 22MiB |
+-----------------------------------------------------------------------------+
┌──[root@test]-[~]
└─$

编写测试脚本测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
(py39) test@test:~/code/Face$ cat cuda_vim.py
import numpy as np
import time
from numba import cuda

@cuda.jit
def increment_kernel(array):
idx = cuda.grid(1)
if idx < array.size:
array[idx] += 1

def main():
n = 1000000000
a = np.zeros(n, dtype=np.int32)

threads_per_block = 1024
blocks_per_grid = (n + threads_per_block - 1) // threads_per_block

start = time.time()
increment_kernel[blocks_per_grid, threads_per_block](a)
end = time.time()

print("Time taken: ", end - start)

if __name__ == "__main__":
while True:
main()

(py39) test@test:~/code/Face$
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Every 2.0s: nvidia-smi                                                                test: Thu Jun 15 14:44:47 2023

Thu Jun 15 14:44:47 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.108.03 Driver Version: 510.108.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A |
| 0% 55C P2 51W / 170W | 4025MiB / 12288MiB | 22% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1386 G /usr/lib/xorg/Xorg 81MiB |
| 0 N/A N/A 1650 G /usr/bin/gnome-shell 22MiB |
| 0 N/A N/A 32031 C python 3917MiB |
+-----------------------------------------------------------------------------+

通过运行nvidia-smi命令获取的GPU状态信息,以下是各列的含义:

  • GPU Name: 显卡名称
  • Persistence-M: 持久化模式,表示GPU是否在计算任务完成后保持运行状态
  • Bus-Id: 显卡的总线ID
  • Disp.A: 显示器适配器的ID
  • Volatile Uncorr. ECC: 显存错误校验码(ECC)错误的数量
  • Fan: 风扇转速百分比
  • Temp: 显卡温度
  • Perf: 性能状态
  • Pwr:Usage/Cap: 显卡功率使用率和功率容量
  • Memory-Usage: 显存使用情况
  • GPU-Util: GPU利用率
  • Compute M.: 计算模式
  • MIG M.: 多实例GPU模式

遇到的问题

安装530高版本报下面的错:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
┌──[root@test]-[~]
└─$sudo ./cuda_12.1.1_530.30.02_linux.run
Error! Could not locate dkms.conf file.
File: /var/lib/dkms/nvidia-fs/2.15.3/source/dkms.conf does not exist.
cat: /var/log/nvidia/.uninstallManifests/kernelobjects-components/uninstallManifest-nvidia_fs: No such file or directory
make: *** No rule to make target 'uninstall'. Stop.
Error! DKMS tree already contains: nvidia-fs-2.15.3
You cannot add the same module/version combo more than once.
===========
= Summary =
===========

Driver: Not Selected
Toolkit: Installed in /usr/local/cuda-12.1/

Please make sure that
- PATH includes /usr/local/cuda-12.1/bin
- LD_LIBRARY_PATH includes /usr/local/cuda-12.1/lib64, or, add /usr/local/cuda-12.1/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-12.1/bin
To uninstall the kernel objects, run ko-uninstaller in /usr/local/kernelobjects/bin
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 530.00 is required for CUDA 12.1 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
sudo <CudaInstaller>.run --silent --driver

Logfile is /var/log/cuda-installer.log
┌──[root@test]-[~]
└─$

解决办法,换了低版本的510

运行 nvvp 报错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
┌──[root@test]-[~]
└─$nvvp
Nvvp: Cannot open display:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.eclipse.osgi.storage.FrameworkExtensionInstaller (file:/usr/local/cuda-11.6/libnvvp/plugins/org.eclipse.osgi_3.10.1.v20140909-1633.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.eclipse.osgi.storage.FrameworkExtensionInstaller
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Nvvp: Cannot open display:
Nvvp:
An error has occurred. See the log file
/usr/local/cuda-11.6/libnvvp/configuration/1686795694122.log.
┌──[root@test]-[~]
└─$

ssh 环境不行,需要做桌面环境

在这里插入图片描述

在桌面环境执行,报错

1
Gtk-Message: 09:10:26.571: Failed to load module "canberra-gtk-module"

安装下面的安装包

1
2
┌──[root@test]-[~]
└─$sudo apt-get install libcanberra-gtk-module

nvidia-driver-XXX-open 版本安装报错

nvidia-driver-530-open 是一个在发行版的非自由存储库中提供的NVIDIA驱动程序,它是由发行版的维护者维护的。这意味着它是与发行版的其余部分紧密集成的,并且由发行版的维护者提供支持和更新。

nvidia-driver-530 是一个第三方非自由驱动程序,它不是由发行版的维护者维护的。相反,它是由NVIDIA公司提供的,并且可能需要手动安装和配置。由于它不是由发行版的维护者提供的,因此您可能无法获得与发行版集成和支持相同的级别。

nvidia-driver-530-open是更受支持和更集成的选择,而nvidia-driver-530则需要更多的手动配置和支持。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 nvidia-driver-530-open : Depends: libnvidia-gl-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: nvidia-dkms-530-open (<= 530.41.03-1)
Depends: nvidia-dkms-530-open (>= 530.41.03)
Depends: nvidia-kernel-common-530 (<= 530.41.03-1) but it is not going to be installed
Depends: nvidia-kernel-common-530 (>= 530.41.03) but it is not going to be installed
Depends: nvidia-kernel-source-530-open (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-compute-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-extra-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: nvidia-compute-utils-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-decode-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-encode-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: nvidia-utils-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: xserver-xorg-video-nvidia-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-cfg1-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Depends: libnvidia-fbc1-530 (= 530.41.03-0ubuntu0.22.04.2) but it is not going to be installed
Recommends: libnvidia-compute-530:i386 (= 530.41.03-0ubuntu0.22.04.2)
Recommends: libnvidia-decode-530:i386 (= 530.41.03-0ubuntu0.22.04.2)
Recommends: libnvidia-encode-530:i386 (= 530.41.03-0ubuntu0.22.04.2)
Recommends: libnvidia-fbc1-530:i386 (= 530.41.03-0ubuntu0.22.04.2)
Recommends: libnvidia-gl-530:i386 (= 530.41.03-0ubuntu0.22.04.2)
E: Unable to correct problems, you have held broken packages.

解决办法,发现并未解决,还是不行。换了不带 open 的版本

1
2
3
4
5
6
7
8
9
10
11
# 更新你的软件包列表和已安装的软件包:
sudo apt update
sudo apt upgrade
# 尝试使用以下命令来修复可能存在的损坏软件包:
sudo apt --fix-broken install
# 使用以下命令来清理系统中已经安装的软件包的缓存:
sudo apt clean
# 尝试使用以下命令来删除已经损坏的软件包并重新安装
sudo apt remove nvidia-driver-530-open
sudo apt autoremove
sudo apt install nvidia-driver-530-open

博文部分内容参考

© 文中涉及参考链接内容版权归原作者所有,如有侵权请告知,这是一个开源项目,如果你认可它,不要吝啬星星哦 :)


https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html#id8

https://blog.51cto.com/u_4029519/5909904


© 2018-2023 liruilonger@gmail.com, All rights reserved. 保持署名-非商用-相同方式共享(CC BY-NC-SA 4.0)

发布于

2023-06-15

更新于

2023-06-21

许可协议

评论
Your browser is out-of-date!

Update your browser to view this website correctly.&npsb;Update my browser now

×