transformers + ja-ginza-electra の Dockerfile

2023-04-29

ライブラリ間の依存関係の問題で transformers と ja-ginza_electra を含む Dockerfile のビルドに苦戦したので備忘録的に残しておく。 Dockerfile

FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime

WORKDIR /root

RUN apt-get update &&\
    apt-get install -y apt-utils dialog &&\
    apt-get install -y emacs less &&\
    apt-get upgrade -y &&\
    apt-get install -y make &&\
    apt-get install -y wget &&\
    apt-get install -y gcc &&\
    apt-get install -y g++ &&\
    apt-get install -y bzip2 &&\
    apt-get install -y libboost-dev &&\
    apt-get install -y google-perftools &&\
    apt-get install -y libgoogle-perftools-dev &&\
    apt-get install -y build-essential --fix-missing

ENV RUSTUP_HOME=/opt/rust CARGO_HOME=/opt/cargo PATH=/opt/cargo/bin:$PATH
RUN pip install setuptools_rust &&\
    mkdir -m777 /opt/rust /opt/cargo &&\
    wget --https-only --secure-protocol=TLSv1_2 -O- https://sh.rustup.rs | sh /dev/stdin -y &&\
    rustup target add x86_64-unknown-freebsd &&\
    printf '#!/bin/sh\nexport CARGO_HOME=/opt/cargo\nexec /bin/sh "$@"\n' >/usr/local/bin/sh &&\
    chmod +x /usr/local/bin/sh

RUN apt-get clean &&\
    apt-get autoclean -y &&\
    apt-get autoremove -y &&\
    apt-get clean &&\
    rm -rf /tmp/* /var/tmp/* &&\
    rm -rf /var/lib/apt/lists/* &&\
    rm -f /etc/ssh/ssh_host_*

WORKDIR /root/app

COPY . .

RUN pip install --upgrade pip && \
    pip install -r requirements.txt --no-cache-dir

CMD ["bash"]

requirements.txt

numpy<1.24.0
scipy==1.10.1
pandas==1.5.3
transformers==4.25.1
spacy[transformers,cuda110]==3.4.4
ginza==5.1.2
ja-ginza-electra==5.1.2