How to Upload Files to Cvmfs Directory

Introduction [edit]

Compute Canada provides repositories of software and data via a file arrangement chosen CERN Virtual Motorcar File System (CVMFS). On Compute Canada systems, CVMFS is already set for you, so the repositories are automatically available for your apply. For more information on using the Compute Canada software surroundings, please refer to available software, using modules, Python, R and Installing software in your home directory pages.

The purpose of this folio is to describe how you can install and configure CVMFS on your computer or cluster, so that you lot can admission the same repositories (and software surround) on your system that are available on Compute Canada systems.

The software environment described on this folio has been presented at Practices and Experience in Advanced Enquiry Computing 2019 (PEARC 2019).

Earlier you offset [edit]

Please subscribe to announcements to remain informed of important changes regarding the Compute Canada software environment and CVMFS, and fill out the registration form. If use of our software surroundings contributes to your inquiry, please acknowledge information technology according to these guidelines. (We appreciate information technology if you also cite our paper).

Subscribe to announcements [edit]

Occasionally, changes will exist fabricated regarding CVMFS or the software or other content provided by Compute Canada CVMFS repositories, which may impact users or crave administrators to take action in order to ensure uninterrupted access to the Compute Canada CVMFS repositories. Subscribe to the cvmfs-announce@gw.alliancecan.ca mailing listing in social club to receive of import just infrequent notifications virtually these changes, by emailing cvmfs-announce+subscribe@gw.alliancecan.ca and then replying to the confirmation email you afterward receive. (Compute Canada staff can alternatively subscribe here.)

Terms of use and support [edit]

The CVMFS client software is provided by CERN. The Compute Canada CVMFS repositories are provided by Compute Canada without any warranty. Compute Canada reserves the right to limit or block your admission to the CVMFS repositories and software environs if yous violate applicable terms of use or at our discretion.

CVMFS requirements [edit]

For a unmarried organization [edit]

To install CVMFS on an individual organization, such as your laptop or desktop, y'all will need:

• A supported operating system (see installation).
• Back up for FUSE.
• Approximately l GB of available local storage, for the cache. (It will only be filled based on usage, and a larger or smaller cache may be suitable in different situations. For light use on a personal computer, just ~ five-10 GB may suffice. Come across cache settings for more details.)
• Outbound HTTP access to the internet.
  • Or at least outbound HTTP access to i or more local proxy servers.

If your system lacks FUSE support or local storage, or has limited network connectivity or other restrictions, you lot may be able to use some alternative approaches.

For multiple systems [edit]

If multiple CVMFS clients are deployed, for instance in a cluster, laboratory, campus or other site, each system must run into the to a higher place requirements, and the following considerations utilize as well:

• Nosotros recommend that yous deploy frontwards caching HTTP proxy servers at your site to amend functioning and bandwidth usage, specially if you have a big number of clients. Refer to setting up a local squid proxy.
  • Note that if y'all take only ane such proxy server it will exist a single point of failure for your site. Generally y'all should have at least two local proxies at your site, and potentially boosted nearby or regional proxies as backups.
• It is recommended to synchronize the identity of the cvmfs service business relationship beyond all customer nodes (east.g. using LDAP or other means).
  • This facilitates utilise of an conflicting cache and should be washed earlier CVMFS is installed. Even if yous do not anticipate using an alien cache at this time, it is easier to synchronize the accounts initially than to try to potentially change them after.

Software environment requirements [edit]

Minimal requirements [edit]

• Supported operating systems:
  • Linux: with a Kernel 2.6.32 or newer.
  • Windows: with Windows Subsystem for Linux version 2, with a distribution of Linux that matches the requirement in a higher place.
  • Mac OS: but through a virtual auto.
• CPU: x86 CPU supporting at to the lowest degree 1 of SSE3, AVX, AVX2 or AVX512 education sets.

Optimal requirements [edit]

• Scheduler: Slurm or Torque, for tight integration with OpenMPI applications.
• Network interconnect: Ethernet, InfiniBand or OmniPath, for parallel applications.
• GPU: NVidia GPU with CUDA drivers (seven.5 or newer) installed, for CUDA-enabled applications. (See below for caveats nearly CUDA.)
• Equally few Linux packages installed as possible (fewer packages reduce the odds of conflicts).

Installing CVMFS [edit]

If y'all wish to apply Ansible, a CVMFS client function is provided every bit-is, for basic configuration of a CVMFS client on an RPM-based system. Besides, some scripts may exist used to facilitate installing CVMFS on cloud instances. Otherwise, use the following instructions.

Pre-installation [edit]

Information technology is recommended that the local CVMFS enshroud (located at /var/lib/cvmfs by default, configurable via the CVMFS_CACHE_BASE setting) be on a dedicated filesystem so that the storage usage of CVMFS is non shared with that of other applications. Accordingly, yous should provision that filesystem earlier installing CVMFS.

Installation [edit]

Follow the instructions relative to your operating arrangement in order to install CVMFS. These instructions take been tested on the following distributions:

• CentOS half-dozen, CentOS 7, CentOS 8
• Fedora 29, Fedora 32
• Debian ix
• Ubuntu 18.04

When installing packages yous may be prompted to have some GPG keys. You should ensure that their fingerprints match these expected values:

• CernVM key: 70B9 8904 8820 8E31 5ED4 5208 230D 389D 8AE4 5CE7
• Compute Canada CVMFS fundamental one: C0C4 0F04 70A3 6AF2 7CC4 4D5A 3B9F C55A CF21 4CFC
• Compute Canada CVMFS primal 2: DDCD 3C84 ACDF 133F 4BEC FBFA 49DE 2015 FF55 B476

RedHat/CentOS Fedora Debian/Ubuntu SLES/openSuSE Windows

• Install the CERN YUM repository and GPG central:

[name@server ~]$ sudo yum install https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm

• Install the Compute Canada YUM repository and GPG keys:

[name@server ~]$ sudo yum install https://package.computecanada.ca/yum/cc-cvmfs-public/prod/RPM/computecanada-release-latest.noarch.rpm

• Install the CVMFS client and configuration packages from those YUM repositories:

[name@server ~]$ sudo yum install cvmfs cvmfs-config-default cvmfs-config-computecanada cvmfs-motorcar-setup

• Install the default configuration package:

[proper name@server ~]$ sudo dnf install https://ecsft.cern.ch/dist/cvmfs/cvmfs-config/cvmfs-config-default-latest.noarch.rpm

• Download the CVMFS client RPM for your operating arrangement from https://cernvm.cern.ch/portal/filesystem/downloads and install information technology with dnf (or yum).
  • Since a yum repository for CVMFS is not available for this operating system, you volition need to periodically bank check for updates to the CVMFS client and default configuration and install them manually.
• Apply the initial client setup:

[name@server ~]$ sudo cvmfs_config setup

• Install the Compute Canada YUM repository and GPG keys:

[proper noun@server ~]$ sudo dnf install https://package.computecanada.ca/yum/cc-cvmfs-public/prod/RPM/computecanada-release-latest.noarch.rpm

• Install the Compute Canada CVMFS configuration from that YUM repository:

[name@server ~]$ sudo dnf install cvmfs-config-computecanada

• Follow the instructions here to add the CERN apt repository:

wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb sudo dpkg -i cvmfs-release-latest_all.deb rm -f cvmfs-release-latest_all.deb sudo apt-get update              

• Install the CVMFS client from that repository:

sudo apt-get install cvmfs cvmfs-config-default              

• Apply the initial client setup:

sudo cvmfs_config setup              

• Download and install the Compute Canada CVMFS configuration package:

wget https://package.computecanada.ca/yum/cc-cvmfs-public/prod/other/cvmfs-config-computecanada-latest.all.deb sudo dpkg -i cvmfs-config-computecanada-latest.all.deb              

• Since an apt repository is not available for this parcel, make sure you are subscribed to be informed of updates.

As these operating systems are RPM-based, following the aforementioned instructions as for Fedora should piece of work.

• For Windows, you get-go need to take Windows Subsystem for Linux, version 2. Equally of this writing (July 2019), this is supported only in a developer version of Windows. The instructions for installing information technology are here [1].
• One time it is installed, install the Linux distribution of your option, and follow the appropriate instructions from one of the other tabs.
• Under WSL2, with Ubuntu, /dev/fuse is not usable by other users than root. This does not permit CVMFS to piece of work properly. To fix this, run

[name@server ~]$ chmod go+rw /dev/fuse

For more information refer to the quickstart guide.

Configuration [edit]

Simple setup Standard setup

On RPM-based systems, if yous want an easy way to go started and are not concerned with performance or disk usage, just do:

[name@server ~]$ sudo yum install cvmfs-quickstart-computecanada

If you encounter any issues, uninstall this packet and follow the standard setup instructions instead.

Do non create any CVMFS configuration files ending with .conf. In club to avoid collisions with upstream configuration sources, all locally-applied configuration must be in .local files. See structure of /etc/cvmfs for more information.

In particular, create the file /etc/cvmfs/default.local, with at least the following minimal configuration:

                CVMFS_REPOSITORIES="cvmfs-config.computecanada.ca,soft.computecanada.ca"                CVMFS_STRICT_MOUNT="yeah"                CVMFS_QUOTA_LIMIT=10000 # see below and adapt equally needed              

• CVMFS_REPOSITORIES is a comma-separated listing of the repositories to use.
• CVMFS_QUOTA_LIMIT is the corporeality of local cache space in MB for CVMFS to use; set it to under 85% the size of your local enshroud filesystem. It should be at to the lowest degree 50 GB for compute nodes in heavy use, while ~ 5-ten GB may suffice for light utilize.
• If you have proxy servers, specify them with CVMFS_HTTP_PROXY. Run into the documentation about this parameter, including syntax, examples, and use of load-balancing groups and round-robin DNS.

For more than information on client configuration see the quickstart guide and client parameters documentation.

Testing [edit]

• Validate the configuration:

[proper name@server ~]$ sudo cvmfs_config chksetup

• Brand sure to accost whatsoever warnings or errors that are reported.
• Cheque that the repositories are OK:

[name@server ~]$ cvmfs_config probe

If you encounter problems, this debugging guide may help.

Enabling our environment in your session [edit]

Once you have mounted the CVMFS repository, enabling our surroundings in your sessions is equally elementary as running the bash script /cvmfs/soft.computecanada.ca/config/profile/bash.sh. This volition load some default modules. If you want to mimic a specific cluster exactly, simply ascertain the environment variable CC_CLUSTER to one of beluga, cedar or graham before using the script, for instance:

[name@server ~]$ export CC_CLUSTER=beluga

[name@server ~]$ source /cvmfs/soft.computecanada.ca/config/contour/fustigate.sh

The in a higher place control will not run annihilation if your user ID is below thou. This is a safeguard, because you should not rely on our software environs for privileged functioning. If you lot even so want to enable our environment, you tin can first define the environment variable FORCE_CC_CVMFS=1, with the control

[name@server ~]$ export FORCE_CC_CVMFS=1

or you can create a file $HOME/.force_cc_cvmfs in your domicile binder if you lot want it to e'er exist active, with

[name@server ~]$ touch $Home/.force_cc_cvmfs

If, on the opposite, you desire to avoid enabling our environment, you can ascertain SKIP_CC_CVMFS=i or create the file $Habitation/.skip_cc_cvmfs to ensure that the environment is never enabled in a given business relationship.

Customizing your environment [edit]

By default, enabling our surroundings will automatically detect a number of features of your system, and load default modules. You lot tin control the default behaviour past defining specific surround variables prior to enabling the environment. These are described below.

Environment variables [edit]

CC_CLUSTER [edit]

This variable is used to place a cluster. It is used to ship some information to the system logs, also as ascertain behaviour relative to licensed software. By default, its value is computecanada. You may desire to prepare the value of this variable if y'all want to take system logs tailored to the name of your system.

RSNT_ARCH [edit]

This environment variable is used to place the set of CPU instructions supported by the system. By default, it will be automatically detected based on /proc/cpuinfo. However if you want to strength a specific ane to be used, you lot can ascertain it before enabling the environment. The supported education sets for our software environment are:

• sse3
• avx
• avx2
• avx512

RSNT_INTERCONNECT [edit]

This environment variable is used to place the type of interconnect supported past the organisation. By default, it volition be automatically detected based on the presence of /sys/module/opa_vnic (for Intel OmniPath) or /sys/module/ib_core (for InfiniBand). The fall-back value is ethernet. The supported values are

• omnipath
• infiniband
• ethernet

The value of this variable will trigger different options of send protocol to be used in OpenMPI.

RSNT_CUDA_DRIVER_VERSION [edit]

This environment variable is used to hibernate or show some versions of our CUDA modules, according to the required version of NVidia drivers, every bit documented [here]. If non divers, this is detected based on the files founds nether /usr/lib64/nvidia.

For backward compatibility reasons, if no library is constitute under /usr/lib64/nvidia, nosotros assume that the driver versions are enough for CUDA 10.2. This is considering this feature was introduced merely every bit CUDA 11.0 was released.

Defining RSNT_CUDA_DRIVER_VERSION=0.0 will hibernate all versions of CUDA.

RSNT_LOCAL_MODULEPATHS [edit]

This environment variable allows to define locations for local module copse, which will be automatically mesh into our fundamental tree. To use it, ascertain

[name@server ~]$ consign RSNT_LOCAL_MODULEPATHS=/opt/software/easybuild/modules

and then install your EasyBuild recipe using

[name@server ~]$ eb --installpath /opt/software/easybuild <your recipe>.eb

This will utilize our module naming scheme to install your recipe locally, and it volition be picked up by the module bureaucracy. For example, if this recipe was using the iompi,2018.3 toolchain, the module will become available subsequently loading the intel/2018.iii and the openmpi/iii.1.2 modules.

LMOD_SYSTEM_DEFAULT_MODULES [edit]

This surroundings variable defines which modules are loaded by default. If it is left undefined, our environment volition define information technology to load the StdEnv module, which volition load by default a version of the Intel compiler, and a version of OpenMPI.

MODULERCFILE [edit]

This is an environment variable used by Lmod to define the default version of modules and aliases. Y'all can ascertain your ain modulerc file and add information technology to the environment variable MODULERCFILE. This will take precedence over what is divers in our environment.

Organization paths [edit]

While our software environment strives to exist as contained from the host operating system as possible, there are a number of system paths that are taken into account by our environment to facilitate interaction with tools installed on the host operating organization. Beneath are some of these paths.

/opt/software/modulefiles [edit]

If this path exists, information technology will automatically be added to the default MODULEPATH. This allows the use of our software environment while also maintaining locally installed modules.

$HOME/modulefiles [edit]

If this path exists, information technology volition automatically be added to the default MODULEPATH. This allows the use of our software environment while also assuasive installation of modules inside of home directories.

/opt/software/slurm/bin, /opt/software/bin, /opt/slurm/bin [edit]

These paths are all automatically added to the default PATH. This allows your own executable to be added in the search path.

Installing software locally [edit]

Since June 2020, we support installing additional modules locally and have information technology discovered past our central hierarchy. This was discussed and implemented in this issue.

To do so, showtime place a path where you want to install local software. For example /opt/software/easybuild. Brand certain that binder exists. Then, consign the environment variable RSNT_LOCAL_MODULEPATHS:

[name@server ~]$ export RSNT_LOCAL_MODULEPATHS=/opt/software/easybuild/modules

If yous want this branch of the software hierarchy to be found by your users, we recommend you define this environment variable in the cluster's mutual profile. And then, install the software packages y'all want using EasyBuild:

[name@server ~]$ eb --installpath /opt/software/easybuild <some easyconfig recipe>

This will install the piece of software locally, using the hierarchical layout driven past our module naming scheme. It will also be automatically found when users load our compiler, MPI and Cuda modules.

Caveats [edit]

Utilise of software environment by organisation administrators [edit]

If you perform privileged system operations, or operations related to CVMFS, ensure that your session does not depend on the Compute Canada software environment when performing any such operations. For example, if you endeavor to update CVMFS using YUM while your session uses a Python module loaded from CVMFS, YUM may run using that module and lose admission to information technology during the update, and the update may become deadlocked. Similarly, if your environment depends on CVMFS and you reconfigure CVMFS in a way that temporarily interrupts access to CVMFS, your session may interfere with CVMFS operations, or hang. (When these precautions are taken, in most cases CVMFS can be updated and reconfigured without interrupting access to CVMFS for users, because the update or reconfiguration itself will consummate successfully without encountering a circular dependency.)

Compute Canada configuration repository [edit]

If y'all already accept CVMFS installed and configured in lodge to utilise other repositories (like CERN'due south repositories), and if your CVMFS client configuration relies on the use of a configuration repository, exist aware that the cvmfs-config-computecanada parcel sets upwardly and enables the cvmfs-config.computecanada.ca configuration repository, which may disharmonize with your utilize of any other configuration repository and potentially intermission your pre-existing CVMFS customer configuration, since clients tin can merely use a single configuration repository. (The Compute Canada CVMFS configuration repository is a central source of configuration that makes all other Compute Canada CVMFS repositories available. It provides all site-independent client configuration required for Compute Canada usage and allows client configuration updates to be automatically propagated. The contents can exist seen in /cvmfs/cvmfs-config.computecanada.ca/etc/cvmfs/ .)

Software packages that are not available [edit]

On Compute Canada systems, a number of commercial software packages are made bachelor to authorized users according to the terms of the license owners, but they are non bachelor outside of Compute Canada systems, and post-obit the instructions on this page volition not grant you admission to them. This includes for instance the Intel and Portland Group compilers. While the modules for the Intel and PGI compilers are available, you will just have access to the redistributable parts of these packages, usually the shared objects. These are sufficient to run software packages compiled with these compilers, but not to compile new software.

CUDA location [edit]

For CUDA-enabled software packages, our software surroundings relies on having commuter libraries installed in the path /usr/lib64/nvidia. However on some platforms, contempo NVidia drivers will install libraries in /usr/lib64 instead. Because information technology is not possible to add together /usr/lib64 to the LD_LIBRARY_PATH without also pulling in all system libraries (which may accept incompatibilities with our software environment), we recommend that you create symbolic links in /usr/lib64/nvidia pointing to the installed NVidia libraries. The script below volition install the drivers and create the symbolic links that are needed (adjust the commuter version that yous desire)

File : script.sh

NVIDIA_DRV_VER="410.48" nv_pkg=( "nvidia-driver" "nvidia-driver-libs" "nvidia-driver-cuda" "nvidia-commuter-cuda-libs" "nvidia-driver-NVML" "nvidia-commuter-NvFBCOpenGL" "nvidia-modprobe" ) yum -y install ${nv_pkg[@]/%/-${NVIDIA_DRV_VER}} for file in $(rpm -ql ${nv_pkg[@]}); do   [ "${file%/*}" = '/usr/lib64' ] && [ ! -d "${file}" ] && \    ln -snf "$file" "${file%/*}/nvidia/${file##*/}" done

LD_LIBRARY_PATH [edit]

Our software environs is designed to apply RUNPATH. Defining LD_LIBRARY_PATH is not recommended and can lead to the environment not working.

Missing libraries [edit]

Because nosotros do not define LD_LIBRARY_PATH, and because our libraries are not installed in default Linux locations, binary packages, such equally Anaconda, will often not observe libraries that they would ordinarily look. Delight run into our documentation on Installing binary packages.

dbus [edit]

For some applications, dbus needs to be installed. This needs to exist installed locally, on the host operating organisation.

grossmestans1945.blogspot.com

Source: https://docs.computecanada.ca/wiki/Accessing_CVMFS