[SPARK-58015][INFRA] Move dependency list from requirements.txt to pyproject.toml

[gaogaotiantian]

What changes were proposed in this pull request?

1. Moved the dependency list from requirements.txt to pyproject.toml (requirements.txt file is not deleted in this PR for compatibility, it will eventually be removed).
2. Update the docs, including AGENTS.md, to use pyprojec.toml instead of dev/requirements.txt.
3. Fixed some leftover reference to black where we already use ruff.
4. Added the explanation for both human and agents to use pip>=25.1 which supports the dependency groups.

Why are the changes needed?

This is just the first step of an important project - to put all of our python dependencies in a single place. To be more specific, dependencies for

1. development (currently requirements.txt)
2. CI (individually defined in docker files and workflow files)
3. packaging (defined in setup.py)

Every time we change a dependency, we need to change it all over our code base and we probably will miss some. What's even worse is that we had to pin dependencies for different release branches.

With the effort to decouple CI, we basically solve the release branch dependency issue (except for 4.2 which has been in rc stage for a while - I want to wait for its release to decouple it). We don't need to keep dependencies for different release branches anymore.

Now we want to move dependency pins from docker files and workflows to pyproject.toml - requirements.txt can't do it because the testing environment is complicated. They share some dependencies but also require different ones for each workflow. We need group structure to make it possible and that's where dependency group comes in. It's a relatively new feature PEP-735 which was supported by pip on April 2025 (more than a year ago). uv sync also supports it.

I believe with this approach, we can eliminate all the dependencies in the docker file and workflow file. We should be able to have a single source of truth of the Python dependency. And it will be more structured.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

This PR should not affect CI.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

Nice!

[dongjoon-hyun]

Please preserve the existing requirements.

spark/dev/requirements.txt

Line 8 in 6b9d033

scipy>=1.8.0

[dongjoon-hyun]

Ditto. If you want to change this, we should do it independently and consistently.

spark/dev/requirements.txt

Line 74 in 6b9d033

graphviz==0.20.3

[dongjoon-hyun]

ml_extra is missing. Without ml_extra, we cannot claim the following in the documentation change because those test cases will be ignored. Please clarify the reason why we exclude this and revise the developer documentation.

- pip install --upgrade -r dev/requirements.txt
+ pip install --upgrade --group dev

[gaogaotiantian]

It makes sense to keep this PR only about structure change so I can keep everything in dev.

One thing that's worth discussing is how many packages we want users to install when they try to work on pyspark. The reason I tried to rule out torch is because torch is very large. The wheel is 500M for Linux and it takes about 1.7G disk after unzip.

torch itself is only used for some features of ml and in a few tests. It's also listed as optional for pyspark installation (https://spark.apache.org/docs/latest/api/python/getting_started/install.html). I think it's a bit of a waste for users who don't deal with ml module to install torch.

However, keeping it exactly as before does not make it worse so we can definitely do that. What's your opinion on this?

[dongjoon-hyun]

The following will be the correct replace for python/docs/source/development/contributing.rst‎.

- pip install --upgrade -r dev/requirements.txt
+ pip install --upgrade --group all

[dongjoon-hyun]

I commented a few missing parts. Please correct them.

[dongjoon-hyun]

If we don't want to do that, let's not claim like that, @gaogaotiantian .

However, keeping it exactly as before does not make it worse so we can definitely do that. What's your opinion on this?

In other words, this PR should not claim a simple replacement or a simple move in the PR title and the PR description. This PR is doing something-else like removing (ml_extra) silently and changing black to ruff officially. I'm not arguing that the direction is wrong. The devil is in the details of transition.

Hence, I'd like to propose two steps to clarify this Python dependency framework improvement approach.

1. In Apache Spark 4.3.0, please introduce additional pyproject.toml based approach while keeping dev/requirements in all docs and CIs. Whatever you want, you can do that in the better way you think.
2. In Apache Spark 5.0.0, we can remove dev/requirements.txt from AGENTS.md, README.mds and contributing.rst.

The bottom line is clean: Let's be clear what we are doing instead of trying to achieve something else at the same time.

Otherwise, please proceed the independent ones first. For example, a recommendation change from black to ruff in docs? Splitting ml-extra from dev/requirements.txt to split some burdens? Those look orthogonal to pyproject.toml migration.

[gaogaotiantian]

Thanks for the suggestion.

There are three things here

1. ruff vs black - this is actually an out-of-date doc. We replaced black with ruff long time ago. For protobufs too. So if the users follow the current documentation, they will have a wrong result. That's more like a "fix" to the current documentation, instead of a "change". However, if you think it's better to have those in a separate PR, I can do that.
2. ml_extra to save some time - I'll do this in a separate PR so we can have proper discussion specific for that. But that can't be done with requirements.txt because of the limitation of that approach.
3. pyproject.toml vs requirements.txt - they should not exist together, otherwise we are introducing another source of dependency. The whole purpose of this refactoring is to have a single source of truth.

Notice that the existing CI does not rely on requirements.txt - it hard-coded all its dependencies in Dockerfile. This has caused us troubles because when we upgrade versions of a dependency, we often miss some in the Dockerfile. If we introduce pyproject.toml but do not change CI/docs and eventually remove requirements.txt, that wouldn't be helpful.

I would not touch cut branches because they are stable. We are not supposed to change the dependencies for branch-4.x, but I believe that's for the actual spark dependencies. We could potentially upgrade the version of devtools or CI 3rd party libraries. When we do such things, it would be really nice to have a synced version of branch-4.x and master. We would make less mistakes and backport would be much easier.

Do you think we can make this happen in both branch-4.x and master? We can keep requirements.txt for a while just to make sure we do not interrupt developers abruptly. But with this change, CIs can finally refer to a file for Python dependencies, instead of hard-coding.