![]() So I would like to avoid this approach.Īlso I can use the docker in airflow and mount the docker socket to airflow container such that I can initialise a new container from the airflow. However, I would like to use docker-compose as life will be much easier to run multiple containers and setting up all network. This container has all the libraries that I need. ![]() On the other hand, if I am not using docker-compose I can start a container with docker:ĭocker run -it -gpus all tensorflow/tensorflow:latest-gpu Is there a way to install these libraries automatically (ideally FROM tensorflow/tensorflow:latest-gpu) as these set the CUDA libraries within the container? However CUDA libraries are still missing. This way within the airflow container I can see nvidia-smi. etc/os-release echo $ID$VERSION_ID) curl -s -L $distribution/nvidia-container-runtime.list | \ sudo tee /etc/apt//nvidia-container-runtime.list sudo apt-get updateĪnd I have managed to use the runtime:nvidia in the docker-compose.yaml file. I have installed nvidia-container runtime and set the deamon default runtime in deamon.json fileĬurl -s -L | \ sudo apt-key add - distribution=$(. I am having some difficulties in starting airflow using docker-compose with appropriate GPU libraries to run my machine learning tasks.Īirflow-scheduler_1 | 12:33:36.919960: W tensorflow/stream_executor/platform/default/dso_:64] Could not load dynamic library 'libcudart.so.11.0' dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directoryīasically, there is no CUDA libraries installed in the /usr/local within the airflow container hence the error.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |