Managing Apache Spark

Managing Apache Spark#

Introduction#

Querona ships with a single, pre-configured instance of Apache Spark that starts automatically. Spark is a separate entitity from Querona and works independently in a dedicated Java process. However, Querona manages Spark configuration and lifecycle, and uses it for the following purposes:

a default federation engine,
a ready-made materialization target for end users, available for use via a pre-configured connection.

The Querona Spark driver#

Querona communicates with Spark through the Querona Spark driver, a gateway that carries requests and results in both directions between Querona and Spark. On a Spark cluster the driver runs on the cluster head nodes; it is resilient and follows the active head node when a new head-node election takes place.

When Spark calls back into Querona over a reverse connection — for example, while executing a distributed task — Querona spools the query results. This lets a Spark task be retried and obtain the same, stable result again.

Instance management#

Local Spark instance management options can be found under the Administer ‣ Local Spark Instances.

Viewing instance status#

Selecting the instance brings up the details blade with all configuration values in read-only mode.

Click the Refresh button to refresh the instance list and to ensure the State column shows the latest information. The view has not auto-refreshing functionality so it might be necessary to use it a few times during state transition.

Starting and stopping Spark instance#

The following buttons control the instance state:

Button	Action
Start	Starts the instance, if not already started.
Stop	Stops the instance, if not already stopped, by sending a “self-shutdown” command, allowing the instance to free all resources and close itself gracefully. This may take up from one minute to several minutes, depending on the load and instance size.
Kill	Kills the instance without waiting for a clean shutdown, by sending an OS-level kill signal which terminates the underlying JVM process (not recommended, unless a graceful shutdown fails).