Huggingface: Difference between revisions

m
(Created page with "[http://huggingface.co Hugging Face] is an organization that builds and maintains several popular open-source software packages widely used in Artificial Intelligence research. In this article, you will find information and tutorials on how to use packages from the Hugging Face ecosystem on our clusters. =Transformers= Transformers is a python package that provides APIs and tools to easily download and train state-of-the-art pre-trained models on various tasks in multi...")
 
Line 20: Line 20:
==Using git lfs==
==Using git lfs==


Pre-trained models are usually made up of fairly large binary files. The Hugging Face makes these files available for download via [https://git-lfs.com/ Git Large File Storage]. To download a model, load the <tt>git-lfs</tt> module and clone your chosen model repository from GitHub:
Pre-trained models are usually made up of fairly large binary files. The Hugging Face makes these files available for download via [https://git-lfs.com/ Git Large File Storage]. To download a model, load the <tt>git-lfs</tt> module and clone your chosen model repository from the model hub:
  module load git-lfs/3.3.0
  module load git-lfs/3.3.0
  git clone https://huggingface.co/bert-base-uncased
  git clone https://huggingface.co/bert-base-uncased
Line 28: Line 28:
  model = AutoModel.from_pretrained("/path/to/where/you/cloned/the/model", local_files_only=True)
  model = AutoModel.from_pretrained("/path/to/where/you/cloned/the/model", local_files_only=True)
  tokenizer = AutoTokenizer.from_pretrained("/path/to/where/you/cloned/the/model", local_files_only=True)
  tokenizer = AutoTokenizer.from_pretrained("/path/to/where/you/cloned/the/model", local_files_only=True)


==Using python==
==Using python==
cc_staff
282

edits