Spark python worker failed to connect back

Работнику Python не удалось подключиться обратно в Pyspark или spark версии 2.3.1

После установки anaconda3 и spark(2.3.2) я пытаюсь запустить пример кода pyspark.

Это всего лишь пример программы, которую я запускаю через Jupyter, я получаю сообщение об ошибке, например

Работнику Python не удалось подключиться обратно.

В соответствии с вопросом ниже при переполнении стека:

Я вижу такое решение, я получил ту же ошибку. Я решил это, установив предыдущую версию Spark (2.3 вместо 2.4). Теперь он работает отлично, возможно, это проблема последней версии pyspark.

Но я использую версию spark 2.3.1 и версию python 3.7.

Тем не менее, я столкнулся с этой проблемой. Пожалуйста, помогите мне решить эту ошибку

from pyspark.sql import SparkSession spark = SparkSession.builder.appName("mySparkApp").getOrCreate() testData=spark.sparkContext.parallelize([3,8,2,5]) testData.count() 

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1.0 (TID 6, localhost, executor driver): org.apache.spark.SparkException: Python worker failed to connect back. at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:170) at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:97) at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117) at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108) at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) 

Источник

[Solved] Jupyter Notebook Error: SparkException: Python worker failed to connect back

Remember to restart the computer after the configuration is completed!

Read More:

• [Solved] ERROR PythonRunner: Python worker exited unexpectedly (crashed)
• Jupyter notebook Failed to Switch to the Virual Environment: DLL load failed python.exe could not find the entry
• [Solved] jupyter notebook Error: ModuleNotFoundError: No module named jupyter_nbextensions_configurator
• How to Solve jupyter notebook Read CSV files Error
• [Solved] Jupyter notebook: TypeError: __init__() got an unexpected keyword argument ‘io_loop’
• [Solved] Jupyter Notebook Start Error: Fatal error in launcher: Unable to create process using
• [Solved] Jupyter notebook use pyLDAvis Error: modulenotfounderror: no module named ‘pyLDAvis’‘
• jupyter notebook Use pyLDAvis Error: AttributeError: module ‘pyLDAvis‘ has no attribute ‘gensim
• [Solved] jupyter notebook Error: 500 : Internal Server Error
• Python classes that connect to the database
• Invalid python sd, Fatal Python error: init_fs_encoding: failed to get the Python cod [How to Solve]
• Pychar can’t connect to Python console, but it can run. Py file, and Anaconda’s command line can run Python command
• Python Error aiohttp.client_exceptions.ClientConnectorCertificateError, Cannot connect to host:443
• [Solved] PyTorch Caught RuntimeError in DataLoader worker process 0和invalid argument 0: Sizes of tensors mus
• [Python] error in installing jupyter: defaulting to user installation because normal Requirement already satisfied
• [Solved] opencv-python: recipe for target ‘modules/python3/CMakeFiles/opencv_python3.dir/all‘ failed
• Python Connect database error: command listdatabases requires authentication
• [Solved] Yolov5 Deep Learning Error: RuntimeError: DataLoader worker (pid(s) 2516, 1768) exited unexpectedly
• Python Error: pip install mysql-connector-python failed
• How to Solve Python WARNING: Ignoring invalid distribution -ip (e:\python\python_dowmload\lib\site-packages)

Источник

SparkException: Python worker failed to connect back

Org.apache.spark.sparkexception: Python worker failed to connect back
tried various online methods, and then
solution:
put my computer – Management – advanced system settings – environment variables – system variables,
put spark_ Set home to the EXE file of python, as shown in the following figure:

it’s done
WIN10 Spark 3.1.2

Read More:

• warning failed to connect to lvmetad，falling back to device scanning.
• Worker failed to boot
• Gunicorn reported error worker failed to boot
• Failed to connect to Mir:Failed to connect to server socket:No such file or directory
• Importerror: no module named typing error reporting solution (python2 PIP needs to be backed back from 21)
• MongoNetworkError: failed to connect to server [localhost:27017] on first connect [Error: connect EC
• Summary of solutions to open flash back problem after Python packaging
• JDBC connect to Sql Server to connect to the database–The TCP/IP connection to the host localhost, port 1433 has failed
• Flash back problem of Python running
• RTMP_Connect0, failed to connect socket. 110 (Connection timed out)
• CentOS Liunx 7 uses rdesktop to remotely connect to the Alibaba Cloud server and it appears Failed to connect, CredSSP required by server.
• Tomcat — failed to start, flash back
• org.apache.spark.SparkException: Task not serializable
• The software of Final Cut Pro x 10.5 failed to open, and it flashed back directly.
• Failed to connect to remote VM. Connection reused. Connection reused: Connect
• The mongoDB service failed to start (exception: connect failed)
• Pod reports error “Back-off restarting failed container” solution
• Python module learning-Paramiko-Use python to throw an exception: Authentication failed.
• PING[ARC2]: Heartbeat failed to connect to standby ‘jzhddg’. Error is 16032.
• EBS R12 login error—R12.1 FRM-92050: FAILED TO CONNECT TO SERVER: /FORMS/LSERVLET

Источник