Running interactive Databricks clusters that do not use the Photon engine

Photon trades a higher rate for less runtime

Photon is Databricks’ native vectorized query engine, processing data in columnar batches. It accelerates SQL, DataFrame API calls, ETL pipelines and stateless streaming, and it does best on longer queries over large data; queries that finish in under two seconds see little gain. Photon compute also consumes DBUs at a different rate from the standard engine.

That makes the economics conditional. If a job finishes in much less time, fewer DBU-hours at a higher rate can still come out cheaper. If the work is dominated by code Photon cannot run, the query falls back to Spark and you pay the different rate for no speed-up.

Seeing which engine each cluster runs

databricks clusters list --cluster-states RUNNING -o json \
  | jq -r '.[] | [.cluster_id, .cluster_name, (.runtime_engine // "unset"), .spark_version] | @tsv'

The runtime_engine field takes STANDARD, PHOTON or no value. Some runtime images carry Photon in their version string, which is why the version column matters too.

How ZopNight decides Photon is off

The cluster must be interactive (UI or API created), and not stopped or in error. Photon counts as on if the runtime engine is PHOTON or the Spark version string contains -photon-. If neither is true, the cluster is reported.

Clusters deliberately passed over

• Clusters on a Databricks Runtime ML image, identified by -ml- in the version. ML code often leans on UDFs and RDD APIs, which Photon does not support.
• Clusters with no recorded Spark version, since the engine cannot be judged without it.
• Job, pipeline, SQL warehouse and model serving clusters, which Databricks manages.

No saving is promised here

Because the DBU rate goes up while runtime may or may not come down, ZopNight cannot claim a saving without measuring your workload, and it does not try. The saving is zero and the category is performance. Treat the finding as a shortlist of clusters worth one controlled test.

Testing Photon before switching

1. Confirm the workload is mostly SQL or DataFrame code on Delta or Parquet. Photon does not run user-defined functions, RDD or Dataset APIs, or stateful streaming.
2. Clone the cluster and tick Use Photon Acceleration, or set runtime_engine to PHOTON through the Clusters API.
3. Run the same representative workload on both and compare DBUs consumed per run, not wall time.
4. Switch the original cluster only where the Photon run used fewer dollars end to end.