Weston Kimbro and Harrison Schumann
Apache Spark 4.2 dropped, and the release notes read like someone let the whole team submit a wishlist and just said yes to all of it. Governed metrics. Native vector search. Geospatial types. Change data capture. A Python path fast enough that you might finally stop apologizing for it. A streaming mode that promises to make your Flink cluster feel like a museum piece. It's a lot.
So we did what any reasonable engineer would do with a release like this: we made a tier list and started fighting with ourselves about it.
Here's the twist: We are writing this from two perspectives, a data engineer and an AI/ML engineer. Due to this dual perspective, we’ll have a few different ranks for different features. You’ll see our debate, then our final decision and combined rank. Some ranks will change, some will remain the same. Thanks to this, you get 3 tier-lists for the price of one: a DE list, an AI/ML list, and a combined list. You’re welcome!
One more thing: Some of these features were lumped together in buckets in the Spark release highlights. We keep a decent chunk of them the same then we take some and (dot)explode them out into more features.
We went feature by feature and asked ourselves if it actually changes our day-to-day, whether it's production ready now or a "cool, check back in 2027" situation, and whether it reduces the number of systems we have to operate. Then we added the question that forced us to rerank multiple features: is this actually new, or is it Databricks customers finally sharing their toys with the rest of the open source Spark playground?
Metric views introduce a native semantic layer to Spark SQL using CREATE VIEW ... WITH METRICS. They define dimensions and metrics as catalog objects stored within view definitions, allowing Spark to preserve aggregation semantics across SQL, BI tools, and external applications.
Metric Views is the feature that kicked off my whole investigation into this release, so let's talk about why it landed here instead of higher. Unity Catalog metric views have existed as a Databricks Public Preview feature for a while now, defined in YAML and registered in Unity Catalog just like any other view. Spark 4.2's version, created with CREATE VIEW ... WITH METRICS, is the same underlying idea, now catalog agnostic. Mechanically, the whole metric definition just lives in the view's stored text, so any catalog that can hold a view definition, Hive Metastore, Glue, whatever, can store one without any special effort. Here's the actual plot twist: for a team already on Databricks, this feature changes basically nothing today. For a team running Spark on EMR or Glue that's never had a governed semantic layer before, this is a genuinely big deal, once your platform gets around to shipping Spark 4.2. Yay for a good open source catch-up!
Metric Views are genuinely useful and provide a unified and governed surface that both AI agents and human analysts can interact with. By defining business metrics once, organizations can ensure consistency across their various workflows, including those involving agents. With the vast improvements in model reasoning capabilities and reliable SQL generation, many organizations have begun to rely more and more on agents to perform data analysis. However, without the right context or constraints, AI agents will rely on their parametric knowledge (information encoded within the model’s parameters from training) and create their own definitions that may differ from that of your organization and even other AI agents. Metric Views help by providing a governance layer for your business metrics, unifying them across BI and AI workflows. However, this feature won’t exactly be new to you if your organization already uses Databricks; it’s mostly just open source Spark catching up to the things we already know and love.
Spark Connect uses a gRPC and Arrow-based protocol to decouple client applications from the Spark server. The 4.2 updates add RDD API compatibility, DataFrame inputs to spark.read.*, expanded error propagation, and support for YARN cluster-mode execution.
Spark Connect improvements land in B tier with no drama attached. This is a properly Spark-native project, and the continued work on RDD compatibility, error handling, and YARN cluster mode support makes it genuinely more practical to call Spark as a remote service from an app or an agent.
While gaps like legacy RDD support and direct JVM access still remain, the improvements to Spark Connect represent nothing but positives. As someone who primarily interacts with Spark through Databricks notebooks, anything that makes Spark Connect faster and easier to use is a win in my book.
Spark 4.2 enables Arrow-optimized Python UDF execution by default for PySpark without requiring code modifications. It adds integration for the Arrow C Data Interface and PyCapsule protocol, enabling direct in-memory passing of DataFrames to engines like Polars and DuckDB without data copying or serialization.
Arrow-first Python execution is the closest thing in this release to a free lunch. Arrow-optimized UDFs are now the default, existing PySpark code gets faster with zero rewrite, and the new PyCapsule and Arrow C Data Interface support lets DataFrames slide into Polars or DuckDB without a serialization tax. No catch, no Databricks predecessor, just solid engineering doing its job quietly.
What really bumps up this feature in the ranking for me is that you don’t have to change a thing on your end to benefit from it. Once you upgrade, all those UDFs you made to perform feature engineering will have Arrow-optimized execution enabled by default, making your code run faster with zero rewrite.
This feature allows batch and streaming data sources to be written, registered, and executed in Python via standard Spark interfaces. Spark 4.2 adds execution profiling tools, including time and memory profiling, to inspect Python connector performance.
Python Data Sources with profiling is a nice, Spark-native improvement for the people building the plumbing behind custom connectors. If that's not you, feel free to keep scrolling.
Yup, that’s not me but it sounds useful… continues scrolling
Spark 4.2 introduces native GEOMETRY and GEOGRAPHY data types along with built-in ST_* functions in Spark SQL. The update includes support for Parquet storage, WKT/WKB format parsing, SRID tracking, and Python type conversion.
Native geospatial types (GEOMETRY/GEOGRAPHY, ST_* functions), straight into S tier. This is the rare feature in this release that is genuinely, unambiguously new to open source Spark. Databricks has had native H3 functions since Runtime 11.2 and a whole geospatial stack through the Databricks Labs Mosaic project, but that was Photon-only and locked to Databricks. Some third party geospatial tooling built for Databricks literally detects when it's running on open source Spark and turns itself off, which is about as blunt a confession as you'll find that this stuff didn't exist anywhere else before. Now it does, for everyone. If I've been bolting Sedona onto vanilla Spark to get spatial joins to behave, this is my sign to stop.
Native geospatial types are one of the features I was initially most excited about in this release. Although many people who never work with geospatial data may not care, the addition of GEOMETRY and GEOGRAPHY data types enables spark to be capable of handling geospatial workflows without the need for external tools to simply write/read geospatial values in a native way. However, when I looked for the ST_* functions that would be needed to actually build common feature engineering workflows, I found only a handful of functions that mainly return/parse WKB or return/set the SRID. There is nothing in this release that enables geospatial calculations and transformations — no ST_Distance, no ST_Contains, no spatial join, etc. You can store and move geospatial data natively now, but the second you want to do something more you will need to bring in other tools. Although the capabilities are still limited at this point, this does lay the foundation for working with geospatial data in spark and is a direction I am glad to see them head. With Parquet and Iceberg adding support for GEOMETRY and GEOGRAPHY types and popular data analytics systems like BigQuery, Redshift, PostgreSQL, and Snowflake already supporting geospatial workflows, this will help Spark begin to catch up.
Spark 4.2 adds SQL primitives for vector manipulation, including distance functions, similarity metrics, normalization, and aggregation. It also introduces the NEAREST BY clause, a top-K ranking join designed for distance-based matching across DataFrames.
Vector search (vector distance/similarity functions, NEAREST BY) also earns S tier from me, but I need to have a talk about the marketing. Yes, this is a genuinely new SQL surface, not a Databricks feature in a trench coat. But read the release notes closely and there's no mention of an index structure anywhere, which is a strong hint that this is brute force distance computation happening live at query time, not an approximate nearest neighbor index like HNSW. That's a real and useful capability. It is not the same thing as Databricks' own AI Search product, which builds actual ANN indexes with vector compression and a dedicated Rust serving engine for billion-scale retrieval. The "this could retire your vector database" headline floating around the internet is doing a lot of heavy lifting. For quick similarity lookups inside a bigger pipeline without standing up a whole separate service, though, this earns its spot on my list.
Apache Spark 4.2 adding native vector distance functions and the NEAREST BY join is a massive quality-of-life win for data science and ML engineering teams. By allowing top-K semantic retrieval directly within standard SQL and PySpark DataFrames, it eliminates the need to continuously syncing millions of embeddings into external vector databases just for batch recommendations, RAG candidate generation, or entity resolution. While dedicated vector DBs aren't going away anytime soon particularly for real-time online inference, bringing native APPROX and EXACT vector joins into the lakehouse drastically simplifies the offline ML stack—saving significant cloud expenditure, cutting down infrastructure sprawl, and letting us keep end-to-end feature pipelines where the data already resides.
This update adds SQL procedural and analytical capabilities, including SET PATH for schema name resolution, procedural script cursors (DECLARE, OPEN, FETCH, CLOSE), QUALIFY for window result filtering, time_bucket for time-series data, Tuple sketches, and IGNORE NULLS/RESPECT NULLS options for aggregate functions.
SET PATH and search paths, SQL script cursors, sketches, time_bucket, QUALIFY, and the new NULL handling options for aggregation functions are all genuinely useful. They're also exactly the kind of feature that's easy to lose in a release this stacked, and specific enough that most teams will implement one or two of these occasionally rather than reorganize a pipeline around them.
These additions represent nice quality-of-life improvements for Spark SQL users. Between enhancing procedural controls with script cursors and syntactic features like QUALIFY and time_bucket, Spark 4.2 cuts down on verbose CTE wrappers and forced context-switching into PySpark or Scala. This is a pragmatic, developer-focused upgrade that can potentially make your data engineering or data science work in Spark SQL cleaner, faster, and more accessible.
Auto CDC provides a declarative Python API (@create_auto_cdc_flow) within Spark Declarative Pipelines for processing Change Data Capture feeds. It automates deletion handling and resolves out-of-order change events to update target tables as Slowly Changing Dimension (SCD) Type 1 tables.
Auto CDC in Spark Declarative Pipelines is a genuine quality of life upgrade, low migration cost, high payoff. It is also, I should mention, not exactly a surprise if you've been paying attention to Databricks. This capability, down to a nearly identical name, has existed in Lakeflow Declarative Pipelines since 2022, first as APPLY CHANGES INTO and later rebranded AUTO CDC. Databricks' own documentation used to explicitly note that the AUTO CDC APIs were not supported in Apache Spark Declarative Pipelines, which is a very polite way of saying "we built this for ourselves first." Spark 4.2 finally closes that gap. If I've been hand rolling merge logic to survive out of order CDC records on non-Databricks Spark, that headache is now someone else's problem.
Spark 4.2’s Auto CDC is a win for data engineering teams and therefore a win for the downstream ML/AI teams that consume their data. This feature allows engineers to replace hundreds of lines of brittle MERGE INTO logic and complex streaming state management with a simple, declarative Python decorator (@create_auto_cdc_flow) that natively resolves out-of-order events. For ML teams, this is phenomenal because it keeps online feature stores accurate, eliminates silent feature poisoning in production serving layers, and enables quicker development of data pipelines.
Real-Time Mode (RTM) is a Structured Streaming execution engine designed for low-latency query processing. In Spark 4.2, PySpark support for RTM is restricted to stateless queries and excludes Python UDF execution.
Real-Time Mode for PySpark is doing its best impression of a rookie who's actually a ten year veteran on another team. RTM launched in public preview on Databricks back in August 2025 and hit general availability there in March 2026, already proven out at companies like DraftKings and Coinbase for fraud detection and live personalization. Spark 4.2 extending it to stateless PySpark queries is open source Spark catching up to something that already shipped and matured somewhere else first. It's also currently limited to stateless queries with no Python UDFs, with stateful support still on the roadmap. Keep an eye on it. I'm not building my whole real-time architecture around it just yet.
For ML engineers and data scientists maintaining separate Apache Flink clusters just to populate real-time feature stores, PySpark’s new Real-Time Mode (RTM) is a huge step forward. Spark 4.2 unlocks millisecond end-to-end latency within the familiar comfort of the Python ecosystem many of us prefer, opening the door for operational AI use cases like instant fraud detection and personalization. That said, Spark 4.2 restricts PySpark RTM to stateless queries without Python UDFs, meaning custom model scoring wrappers and stateful feature aggregations aren't ready yet. Thankfully, the upcoming Spark 4.x roadmap directly targets these pain points by introducing a dedicated streaming shuffle, concurrent stage scheduling, transformWithState, and native Python UDF execution. Once those pieces are delivered, PySpark will finally be a single, unified engine capable of handling everything from offline model training to sub-second real-time inference.
Spark 4.2 adds the CHANGES SQL clause paired with a Data Source V2 (DSv2) changelog API. This provides a unified SQL syntax and API interface for querying row-level change feeds across any DSv2-compliant storage format.
The CHANGES clause and CDC in DSv2 also land here. This is not a story about Delta having CDC while Iceberg waits its turn. Delta, Iceberg, and Hudi all already had their own way to get row-level changes before Spark 4.2 existed: Delta uses its Change Data Feed (table_changes()), Iceberg uses a create_changelog_view procedure that produces .changes views, and Hudi leans on custom incremental read options. Three formats, three different syntaxes, zero portability between them. That fragmentation is exactly what Spark 4.2's new CHANGES clause and DSv2 changelog API are trying to fix: one SQL clause, one API, and Spark's engine handles the shared post-processing instead of every connector reinventing it. There's still a real asterisk, and it is simple: your preferred table format must support DSv2. If it doesn’t this is a nothing-burger, if it does then we get a nice and seamless approach to CDC across multiple table formats.
This is not a feature that I typically deal with directly, but I'll give the side that matters to my typical work. Incremental training and feature refresh get dramatically easier to audit when you can ask a table "what changed between these two versions" and get a straight answer. Also, one CHANGES clause that behaves the same across connectors is the sort of standardization that makes data pipelines easier to build and more flexible. I give this feature a B not because it's unimportant but because, from the AI/ML seat, it's not likely to play a big part in my day-to-day.
Spark 4.2 expands Data Source V2 (DSv2) schema evolution by adding WITH SCHEMA EVOLUTION to SQL INSERT INTO statements and withSchemaEvolution() to the DataFrame writer API, enabling automatic column addition via name- or position-based resolution. Additionally, MERGE INTO query execution gains whole-stage code generation and detailed operational metrics.
DSv2 MERGE INTO and schema evolution improvements are solid engineering aimed at a narrow audience, and yes, another partial catch-up. Delta's mergeSchema and autoMerge options for MERGE INTO have been around since Delta 0.8.0 in 2021, just scoped to Delta tables only. Spark 4.2 extends schema evolution to INSERT INTO and generalizes the whole thing to the DSv2 API, so it's no longer a Delta exclusive in principle. The catch: this only helps if the connector I actually use has implemented the relevant interfaces since we all have the weird step-child data sink (we don’t).
While schema evolution is important (feature sets grow constantly throughout the data science process) and extending that to INSERT INTO and generalizing it past Delta is a nice convenience, this feature only pays off if the connector you are using implements the interfaces. And for an AI/ML engineer, the impact of this feature will be felt (if at all) through a pipeline someone else likely built. I’d say it's useful, but for a niche audience.
Platform updates in Spark 4.2 include support for JDK 25, Kubernetes integration updates, and a redesigned Web UI featuring dark mode and updated SQL query execution plan visualizers.
The Web UI overhaul with dark mode, Kubernetes improvements, and JDK 25 support are necessary, unglamorous, and correctly boring. Nobody is switching query engines because of a nicer plan visualizer. I will, however, quietly appreciate it the next time I'm debugging a stuck job at 1am.
Nobody is going to upgrade just because of dark mode. However, the SQL plan visualization upgrades are a nice-to-have when debugging feature-engineering jobs in the future and JDK 25 support plus the Kubernetes work are the boring improvements that keep everything running under the hood while flashier features get talked about.
As you can tell from our rankings, we agree on most of Spark 4.2's impact. Despite coming from different roles, Data and AI/ML Engineering share significant overlap. We frequently lean on each other for domain expertise, and cross-functional projects blur those lines even further. Where we differed, it usually came down to daily priorities or niche tools on opposite ends of our workflows. That said, we wanted to deliver a single, unified ranking that incorporates both perspectives—so we reconciled our differences and outlined our joint reasoning below.
This is a classic "if you know, you know" update. Not everyone gets deep into the weeds of building and debugging custom connectors, but if you do, you'll definitely appreciate this QoL upgrade.
We went with the higher of our two individual ratings here. Native geospatial support is a massive milestone, and we're excited to see Spark moving in this direction. While it's currently limited by a small set of native ST_* functions, we expect that library to expand over time.
While we appreciate their addition, SQL primitives sit lower on our list simply because their scope and day-to-day impact aren't quite as broad as other features in 4.2.
Millisecond latency in the Python ecosystem is a game-changer for operational AI, but current limitations—stateless-only execution and a lack of Python UDFs—hold it back for now. With stateful support on the 4.x roadmap, this is one we’re watching closely.
Extending schema evolution to INSERT INTO and beyond Delta is a nice convenience, but it only pays off if your specific connector implements the new interfaces. Useful, but aimed at a niche audience.
All in all, Spark 4.2 is a stacked release, and going through it feature by feature only made that clearer. The biggest wins are the features that either break genuinely new ground or speed up existing workflows for free—geospatial types, vector search, Arrow-first Python, and Auto CDC. The features lower on our list are mostly catch-ups, connector-dependent, or waiting on future roadmap items to unlock their full potential.
Where you land on any given feature depends heavily on the seat you're sitting in, but that's the beauty of a release this big: there's something here for everyone, and plenty to be excited about.
Read more about the latest and greatest work Rearc has been up to.
A data engineer and an AI/ML engineer tier-rank every major Spark 4.2 feature, argue about it, and reconcile their disagreements into one combined ranking.
LLM applications have a semi-infinite attack surface, and they are notoriously hard to secure without breaking the user experience.
Recently, articles surrounding the supposed dangers of an open source abliteration tool called Heretic, along with legal notice being served to its creator, have inspired me to speak out for two reasons.
A step-by-step guide to deploying a Databricks workspace with Private Service Connect (PSC) on GCP and common pitfalls to avoid.
Tell us more about your custom needs.
We’ll get back to you, really fast
We will evaluate your query and respond within 2 business days.
Kick-off meeting
We will schedule a quick meeting to further understand your use case and start working toward a solution together!
