You forgot to mention the log4j.properties file must be accessible from all Spark driver and workers. You must use a shared file-system, or the Spark — files directive.