1
edit
Apache Spark: Difference between revisions
How to fix errors while running Pyspark
No edit summary |
(How to fix errors while running Pyspark) |
||
| Line 48: | Line 48: | ||
stop-master.sh | stop-master.sh | ||
}} | }} | ||
== Notes on pyspark_submit.sh == | |||
1. If you are encountering error <code><span style="color:#ff0000">Error: Cannot load main class from JAR file</span></code> try replacing the <code>--executor-memory ${SLURM_MEM_PER_NODE}M</code> (Line no. 23) with <code> --executor-memory=<size>M </code> | |||
2. If you are encountering error <code><span style="color:#ff0000">Error: Master must either be yarn or start with spark, mesos, local</span></code> increase the sleep time. Replace <code>sleep 1</code> (Line no. 16) with <code>sleep 5</code> | |||
== Java Jars == | == Java Jars == | ||
{{File | {{File | ||
|name= | |name=pyspark_java_submit.sh | ||
|lang="sh" | |lang="sh" | ||
|contents= | |contents= | ||