Solve the “zsh: no matches found: local[4]” issue when running a spark application.
Issue
When I was trying to submit a Spark streaming application with this script:
1 | $ ~/spark/bin/spark-submit \ |
I came across this error from zsh:
1 | $ zsh: no matches found: local[4] |
Solution
I replaced local[4] with local it seemed to work, but according to the usage of master URL:
| command | meaning |
|---|---|
| local | run Spark locally with one worker thread (i.e. no parallelism at all) |
| local[K] | run Spark locally with K worker threads (ideally, set K to the number of cores on your machine) |
I should not change the master URL because I wanted it to run in parallelism.
So I changed back to bash and run spark submit script again, interestingly it worked! Obviously it was an issue of zsh.
According this post Zsh says “no matches found” when trying to download video with youtube-dl, it said that zsh only accepts parameters by quoting them:
1 | $ ~/spark/bin/spark-submit \ |
It worked!
Conclusion
In zsh, by default, a failed expansion is an error, but in Bash it isn’t: the failed pattern is just left as an argument exactly as it was written. zsh’s behaviour is safer, in that you can’t write a command that secretly doesn’t do what you meant because a file was missing, but you can change it to have the Bash behaviour if you want:
1 | setopt nonomatch |
This will resolve the original use case. In general it will be better to quote arguments with special characters, though, in order to avoid any mistakes where a file happens to exist with a corresponding name, or doesn’t exist when you thought it did.
The NOMATCH option is on by default, and causes the errors I were seeing. If you disable it with setopt nonomatch then any failed glob expansions will be left intact on the command line, for example:
1 | $ echo foo?bar |