Arrow: Difference between revisions

<translate>

[https://arrow.apache.org/ Apache Arrow] is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication. Languages currently supported include C, C++, C#, Go, Java, JavaScript, MATLAB, Python, R, Ruby, and Rust.

Arrow is also available with CUDA.

{{Command|module load gcc/8.3.0 arrow/0.16.0 cuda/10.1}}

The module contains bindings for multiple python versions.  

To discover which are the compatible Python versions:

{{Command|module spider arrow/0.16.0}}

The Arrow Python bindings (also named ''PyArrow'') have first-class integration with NumPy, pandas, and built-in Python objects. They are based on the C++ implementation of Arrow.

1. Load the required modules:

{{Command|module load gcc/8.3.0 arrow/0.16.0 python/3.7 scipy-stack}}

2. Import PyArrow

{{Command|python -c "import pyarrow"}}

The command display nothing. You have successfully imported PyArrow.

For more information, see the [https://arrow.apache.org/docs/python/ Arrow Python] documentation.

The [http://parquet.apache.org/ Parquet] file format is available.  

To import it, execute previous steps for <tt>pyarrow</tt>, then :

{{Command|python -c "import pyarrow.parquet"}}

The command display nothing. You have successfully imported the Parquet module.

The arrow package exposes an interface to the Arrow C++ library to access many of its features in R. This includes support for analyzing large, multi-file datasets ([https://arrow.apache.org/docs/r/reference/open_dataset.html open_dataset()]), working with individual Parquet ([https://arrow.apache.org/docs/r/reference/read_parquet.html read_parquet()], [https://arrow.apache.org/docs/r/reference/write_parquet.html write_parquet()]) and Feather ([https://arrow.apache.org/docs/r/reference/read_feather.html read_feather()], [https://arrow.apache.org/docs/r/reference/write_feather.html write_feather()]) files, as well as lower-level access to Arrow memory and messages.

1. Load the required modules:

{{Command|module load gcc/8.3.0 arrow/0.16.0 r/3.6 boost/1.68.0}}

2. Specify the local installation directory:

{{Commands

}}

3. Export the required variables to ensure we are using the system installation:

{{Commands

}}

4. Install the bindings

{{Command|R -e 'install.packages("arrow", repos{{=}}"https://cloud.r-project.org/")'}}

Once installed, you can load the bindings.

1. Load the required modules:

{{Command|module load gcc/8.3.0 arrow/0.16.0 r/3.6}}

2. Load the library:

{{Command

}}

For more information on its usage, see [https://arrow.apache.org/docs/r/index.html Arrow R documentation]

</translate>