Richard Si – Assorted ramblings

What's new in pip 26.1 - lockfiles and dependency cooldowns!

Sun, 26 Apr 2026 00:00:00 -0400

On April 26, 2026, we, the pip team, released pip 26.1.

As always, please consult the changelog to learn about all of the changes contained in this release.

Python 3.9 is no longer supported

pip 26.1 now targets Python 3.10 and higher.

Python 3.9 entered End-of-Life (EOL) status six months ago in October 2025. At this point in the year, many of pip’s vendored dependencies have stopped supporting Python 3.9, so it made sense for pip to drop support as well.

New features ✨

Experimental: installing from pylock files

After a year since the acceptance of PEP 751 which standardized pylock.toml lock files, pip 26.1 gains experimental (see below for more details on “experimental”) support for reading and installing from such lockfiles. 🎉

The -r <file>, --requirement <file> option now accepts pylock.toml files. You may pass a local lockfile or a remote lockfile as long it is named as expected: pylock.toml or pylock.<name>.toml.

$ pip install -r pylock.toml
WARNING: Using pylock.toml as a requirements source is an experimental feature. It may be removed/changed in a future release without prior warning.
Collecting six==1.17.0 (from pylock.toml)
 Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Collecting urllib3==2.6.3 (from pylock.toml)
 Downloading urllib3-2.6.3-py3-none-any.whl (131 kB)
Installing collected packages: urllib3, six
Successfully installed six-1.17.0 urllib3-2.6.3

pip install, pip download, and pip wheel all support reading lockfiles since they accept -r.

$ pip wheel -r pylock.toml -w ./wheelhouse
WARNING: Using pylock.toml as a requirements source is an experimental feature. It may be removed/changed in a future release without prior warning.
Collecting six==1.17.0 (from pylock.toml)
 Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Collecting urllib3==2.6.3 (from pylock.toml)
 Using cached urllib3-2.6.3-py3-none-any.whl (131 kB)
Saved ./wheelhouse/six-1.17.0-py2.py3-none-any.whl
Saved ./wheelhouse/urllib3-2.6.3-py3-none-any.whl

$ pip install wheelhouse/*.whl --no-deps --no-index
Processing ./wheelhouse/six-1.17.0-py2.py3-none-any.whl
Processing ./wheelhouse/urllib3-2.6.3-py3-none-any.whl
Installing collected packages: urllib3, six
Successfully installed six-1.17.0 urllib3-2.6.3

The intention of this feature is support “environment replication” from a lockfile. Ideally, pip install -r pylock.toml is executed within an empty environment, although this isn’t required. While we are interested in hearing about additional use-cases, the pip team reserves to declare such use-cases as out of scope.

While you may mix -r pylock.toml with any other normal requirement or constraint specifier or file, as long as the total set of requested packages do not conflict, this is strongly discouraged. This is a side-effect of the current implementation, and it is likely to change as the pip project stabilizes its lockfile support.

Limitations and gotchas

Extras and dependency groups currently cannot be selected from a (multi-use) pylock file¹
VCS and local directory entries currently cannot be mixed with external requirements that are hash-locked
The optional fallback to using packages.index to resolve broken URLs is unsupported
By default, pip will let requirements from lock files pull in additional non-locked requirements. Pass --no-deps if this is undesirable.
Archive file sizes are currently not validated
Only the format control options --only-binary and --no-binary will influence locked requirements. All other package selection options (such as --pre, --abi, etc.) will be ignored

Please note that requirements sourced from a lockfile are treated specially. They function like pinned requirements where the used archive or path is pre-constrained.

If hashes are available for a locked requirement, pip will validate hashes as expected.

What’s next?

As pip’s lockfile support is experimental, pip install -r pylock.toml and pip lock should not be considered production ready. In the next few releases, the pip team will refine the lockfile UI/UX in response to real-world experience and feedback.²³

Warning

We reserve the right to modify, deprecate, or remove experimental features without notice. pip’s support for reading pylock files via -r pylock.toml is intended as a transitional mechanism until pip can adopt expanded and better UI/UX.

We will be also working towards adding a pip sync command which will function similar to pip-sync and uv sync although in a way that makes sense for pip’s operating model. We intend for pip sync to be primary way in which pip interacts with lockfiles once available.

Thank you to Stéphane Bidoul who not only championed pip’s implementation, but upstreamed pylock.toml support into the packaging library so the whole Python ecosystem can benefit.

We need your feedback!

In the meanwhile, if you can, please test out -r pylock.toml. If you have any feedback, let us know in the following issues!

We’d especially appreciate comments on your use-case for lockfiles and how pip enables or does not enable your use-case.

Dependency cooldowns

--uploaded-prior-to now supports accepts a relative duration in the PnD format, where n is the number of days.⁴ This is meant to enable dependency cooldowns, which aim to reduce the impact of compromised upstream packages.

$ pip install --uploaded-prior-to=P3D pip --force-reinstall # exclude the recent pip release
Collecting pip
 Downloading pip-26.0.1-py3-none-any.whl.metadata (4.7 kB)
Downloading pip-26.0.1-py3-none-any.whl (1.8 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 2.3 MB/s 0:00:00
Installing collected packages: pip
 Attempting uninstall: pip
 Found existing installation: pip 26.1.dev0
 Uninstalling pip-26.1.dev0:
 Successfully uninstalled pip-26.1.dev0
Successfully installed pip-26.0.1

A dependency cooldown involves configuring a set period of time before a newly released package is eligible for installation, providing time for package registries and third-party security firms to discover and package authors to recover from a compromise.

It’s worth noting that while a cooldown can minimize the impact of supply chain attacks, it will also delay security fixes from reaching your environment. If you use this option, pair it with a vulnerability scanning tool such as Dependabot or pip-audit so that you are notified of security issues independently of your update schedule.

For more information, please consult William Woodruff’s excellent post “We should all be using dependency cooldowns”. Thank you also to Norbert Manthey for contributing this feature to pip!

Lifting several 2020 resolver limitations

The modern pip resolver (“the 2020 resolver”) is miles better than the legacy resolver, capable of resolving complex dependency trees without producing an invalid environment. However, pip has kept its legacy resolver available as a deprecated feature via --use-deprecated=legacy-resolver to support use-cases that are unsupported by the 2020 resolver.

Starting in pip 26.1, the 2020 resolver now supports most of these previously unsupported use-cases:

Fix a crash when installing a requirement with an extra that has a corresponding URL constraint
Properly apply URL constraints to requirements with extras
Allow unpinned requirements to use hashes from constraints. Constraints like {name}=={version} --hash=... feeds into hash verification for a corresponding requirement.

With a few remaining planned changes, pip will be in a position to fully remove the legacy resolver, which we would like to do sometime in 2027.

Security fixes

Avoid mistreating a .tar.gz archive as a zip file (CVE-2026-3219)

It’s possible for a .tar.gz archive to look like a zip file, i.e. when given to the stdlib zipfile.is_zipfile() function, it will return true. On pip 26.0 and older, pip will ignore the .tar.gz contents and use the zip contents instead. This can be abused to easily obfuscate malicious code.

pip 26.1 updates the logic used to disambiguate tar and zip archives. pip will now use the following signals, in decreasing priority:

Content type returned by the remote server
File extension
Python standard library magic detection functions if only they unambiguously agree

Thank you to Dmitrii Sutiagin for contributing a fix and to Caleb Brown for reporting this issue responsibly to the pip team.

Fix ACE caused by self check deferred imports (CVE-2026-6357)

pip will issue a notice if there is a newer version of pip available. For performance reasons this check occurs at the end of a command invocation, with the responsible code being imported when needed. If pip’s own modules are overwritten during a pip install, this may result in arbitrary code execution.

pip 26.1 fixes this vulnerability by performing the self check eagerly before any command runs, deferring only the emitting of the notice to the very end.

Thank you to Damian Shaw for discovering and fixing this issue.

Upgrade to urllib3 2.x

Now that pip requires Python 3.10, pip 26.1 ships with urllib3 2.6.3 instead of 1.26.20. Unlike the 1.x series, 2.x is actively supported by the urllib3 project.

This upgrade fixes a number of CVEs associated with the urllib3 library:

Please note that we consider these CVEs to be largely inconsequential for pip. The main benefit for end-users is that any downstream security scanners will stop complaining about pip’s vendored copy of urllib3.

Deprecations & upcoming removals

Here’s the current list of current deprecations with the release in which they are scheduled for removal. As always, any given removal may be pushed to a future release as needed.

The PIP_CONSTRAINT envvar will eventually stop taking effect for build dependencies. We’re making this change to accommodate the planned transition to installing build dependencies in-process. Affected users should use --build-constraint or PIP_BUILD_CONSTRAINT.

Acknowledgements

None so far. I did write this on a compressed timeline, however, so please forgive any typos.

--group will not do what you want. It will read from the local pyproject.toml. ↩︎
If you take a look at the linked issues, there many open design questions left to resolved. I have my own opinion and the other pip committers may have different opinions. We want to get the UI/UX right, so please forgive us as we cycle on pip’s lockfile support. ↩︎
In general, we don’t want pip to be an innovator when it comes to new features or new UX flows. We will aim to match pre-existing features from other package managers where feasible. ↩︎
We chose to only support the simple PnD ISO 8601 duration format to keep the parsing code as minimal as feasible. We did not want to vendor a human time parsing library or roll our own. ↩︎

What's new in pip 26.0 - prerelease and upload-time filtering!

Fri, 30 Jan 2026 00:00:00 -0500

On January 30, 2026, we, the pip team, released pip 26.0.

As always, please consult the changelog to learn about all of the changes contained in this release.

Also, I apologise for the lack of a release post for pip 25.3. Changes have occurred in my personal life that have resulted in me having very little free time. I was thus unable to find the time to write a post in time for pip 25.3. Please enjoy this post for 26.0.

New features ✨

Excluding distributions by upload time

The new --uploaded-prior-to option allows you to filter packages by their upload time to a package index, only considering packages that were uploaded before a specified datetime. This can be useful for debugging broken dependencies and ensuring reproducible builds by ensuring you only install packages that were available at a known (good) point in time.

A common situation is that you have a years old project (with no lockfile) and attempting installing the dependencies results in a broken install. With --uploaded-prior-to, you can exclude packages uploaded after the last time you know the install worked and reobtain the same set of working dependencies.

The option accepts ISO 8601 datetime strings in several formats:

2025-03-16 - Date in implicit local timezone
2025-03-16T12:30:00 - Datetime in implict local timezone
2025-03-16T12:30:00Z - Datetime in UTC
2025-03-16T12:30:00+05:00 - Datetime with local timezone offset

For consistency, the timezone should be explicitly provided, although it is not required.

$ pip install -r requirements.txt --uploaded-prior-to 2024-03-16T12:30:00+05:00

[...]
Installing collected packages: platformdirs, pathspec, packaging, mypy-extensions, click, black
Successfully installed black-24.3.0 click-8.1.7 mypy-extensions-1.0.0 packaging-24.0 pathspec-0.12.1 platformdirs-4.2.0

Warning: it is an exclusive upper bound!

While similar to uv’s --exclude-newer option, --uploaded-prior-to behaves a bit differently. Notably, --uploaded-prior-to is an exclusive upper bound for both datetimes and dates. In other words, --uploaded-prior-to 2025-01-01 is equivalent to --uploaded-prior-to 2025-01-01 00:00:00, not 2025-01-01 23:59:59 as with --exclude-newer.

More information can be found in the pip user guide. We thank uv for the inspiration with their --exclude-newer option.

Separate build constraints

This feature actually shipped in pip 25.3, but it’s noteworthy enough to mention in this post.

Build constraints can be set independently with --build-constraint. This allows constraining the versions of packages used during the build process (e.g., setuptools) without affecting the final installation.

This can be useful when a new release of a build backend is causing a package build to error. You can constrain (block) the problematic versions of the backend and continue with happily installing whatever you were installing earlier.

constraints.txt

setuptools != 80.0.0 # imagine this version broke your build

$ pip install myproject.tar.gz --build-constraint constraints.txt

Previously, your best option to constrain build dependencies was through setting the PIP_CONSTRAINT environment variable. Unlike the --constraint option, the envvar would be picked up by the pip subprocess invoked to install build dependencies. This works, but is quite obscure and has the side-effect of constraining the final installation as well.

At some point, we would like to stop calling pip in a subprocess to install build dependencies and switch to installing them in-process. This has a bunch of benefits, but would break the PIP_CONSTRAINT workaround. While we could update the --constraint option to also influence build dependencies , we are using this transition as an opportunity to clean up constraint handling.¹

As part of this transition, pip has deprecated using PIP_CONSTRAINT to constrain build dependencies. Affected users are encouraged to use --build-constraint or set PIP_BUILD_CONSTRAINT.

`--only-final` and `--all-releases`

By default, pip installs stable versions of packages, unless their specifier includes a pre-release version (e.g., SomePackage>=1.0a1) or if there are no stable² versions available that satisfy the requirement. The --all-releases and --only-final options provide per-package control over pre-release selection.

As their names imply, these new options allow you to tell pip to only consider stable or all versions when resolving packages. They function like their --no-binary and --only-binary cousins.

# accept the bleeding edge for a dependency
python -m pip install --all-releases=DependencyPackage SomePackage
# we only accept stable versions for this dependency
python -m pip install --only-final=DependencyPackage SomePackage

The pre-existing --pre flag is now equivalent to --all-releases :all:.

For more information, please consult the pip user guide.

Installing from inline script metadata (PEP 723)

This release has pip gain the --requirements-from-script option for installing dependencies declared in inline script metadata (PEP 723). Inline script metadata is useful for scripts that depend on external libraries, but are designed to be distributed as a single file.

Recently, I had to write a script for converting a Jekyll site into a form that could be imported by the Ghost CMS. Jekyll posts are stored as Markdown files with YAML frontmatter, which meant I had to pull in PyYAML and a Markdown parser. This is no problem. I can simply declare these dependencies using a script metadata comment. No separate requirements.txt needed.

"""
Archive Jekyll posts into a JSON file Ghost can import.
"""

# /// script
# requires-python = ">=3.11"
# dependencies = [
# "click",
# "rich",
# "python-frontmatter",
# "markdown-it-py[plugins]",
# "PyYaml",
# ]
# ///

import json
import subprocess
import zipfile

# and so on ...

To run this script, I can simply run pipx run ghost_export.py. pipx will see the script comment and auto-install the dependencies before executing the script. This is extremely convenient!³

Why a new option?

This is a good time to explain why this feature is its own option. While we could’ve overloaded the -r option as suggested by some, --requirements-from-script is not equivalent to -r. If requires-python is declared in a script metadata block, pip will validate that the Python version is compatible as well.

And while we think that using pip with inline script metadata is suboptimal (you should really be using a proper script runner), we recognise that there are situations where such script runners are unavailable or undesirable.

--requirements-from-script can also be given with pip download, pip lock, and pip wheel.

Thank you to @SnoopJ for contributing this feature to pip!

Experimental: In-process build dependencies

The experimental feature to install build dependencies in-process, originally slated for 25.3, has now landed in pip 26.0. You should read my pip 25.2 post for the rational for this change, but the benefits include:

Faster build environment provisioning
More reliable and less confusing option and flag inheritance
In the future: support for authentication prompts while installing build dependencies

It can be enabled with the --use-feature=inprocess-build-deps flag, although the plan is enable it by default in a future release once the feature stabilizes.

Deprecations & upcoming removals

Here’s the current list of current deprecations with the release in which they are scheduled for removal. As always, any given removal may be pushed to a future release as needed.

Acknowledgements

Thank you to Damian Shaw for reviewing and correcting some glaring mistakes. Any remaining mistakes are my own.

This also has the nice benefit of aligning our constraint options with uv which already has --build-constraint. ↩︎
What is a stable version? Any package whose version does NOT contain a development segment or pre-release segment (.alpha, .beta, .rc, .dev). ↩︎
You can use uv run as well. ↩︎

What's new in pip 25.2 ... and a sneak peek of 25.3

Wed, 30 Jul 2025 00:00:00 -0400

On July 30, 2025, we, the pip team released pip 25.2.

Relative to the large 25.1 release, this release is tiny. It’s honestly so small that it barely warrants a post, but it wouldn’t be a proper release without a post, so enjoy whatever this is.

As always, please consult the changelog to learn about all of the changes contained in this release.

Key changes

Python 3.14 is now supported, with fixes to file URL/path handling included.¹
Automatic download resumption is now stable and enabled by default.🎉
- By default, pip will resume a download up to five times. If more attempts are desired, you may configure a higher limit with --resume-retries.
- In addition, caching support was added to resumed downloads so pip does not need to redownload files as expected.
The download ETA is replaced by the elapsed time once the download has completed. While ETA 0:00:00 is technically correct, it is not useful information 😉

The truststore and proxy features can be used together without conflicting with each other.
Sub-commands are included in TAB completion.

Deprecations & upcoming removals

Here’s the current list of current deprecations with the release in which they are scheduled for removal. As always, any given removal may be pushed to a future release as needed.

See the pip 25.0 post for more details. This got pushed to pip 25.3 as the support for the Direct URL syntax for installing editable VCS projects did not land in this release, sadly.

Please read the deprecation issue for more details and advice. This was scheduled for removal in 25.0 and then 25.1, but it got pushed back (again) to coincide with the deprecation of setup.py bdist_wheel installs.

Please read the deprecation issue for more details and advice, however, the summary is that pip will stop running setup.py bdist_wheel directly to build a wheel for installation. This is a compatibility fallback for old environments that do not support the modern PEP 517 interface.

This is a continuation of the old deprecation as the original deprecation did not catch all non-standard wheel filenames. See the pip 25.0 post for more details.

Sneak peek: in-process build dependencies

Since this release post is so small, I thought I’d take the time to share what I’ve been working on in addition to my typical maintainer duties.

If you’ve ever installed a project from source, you’ve likely noticed that pip takes a while to build the project.

$ time pip install ./dev/oss/pip
Processing /home/ichard26/dev/oss/pip
 Installing build dependencies ... done
 Getting requirements to build wheel ... done
 Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: pip
 Building wheel for pip (pyproject.toml) ... done
 Created wheel for pip: filename=pip-25.2.dev0-py3-none-any.whl size=1856601 sha256=8d02c1146bbeaa023421ac298240575fc2e96400ac3a7f86209426a8254dc0f8
 Stored in directory: /tmp-ram/pip-ephem-wheel-cache-4ctznmto/wheels/de/a6/21/441242ad27acc3bddc1079ff202cc70bad47d31432987f2bf0
Successfully built pip
Installing collected packages: pip
Successfully installed pip-25.2.dev0

real 0m2.986s
user 0m2.132s
sys 0m0.527s

While pip is generally not fast, PEP 517 builds—which is the modern/default mechanism for building projects—are particularly heavy. If I disable PEP 517 processing and force pip to use the legacy setuptools-specific build mechanism, the install is noticeably faster.

$ time pip install ./dev/oss/pip --no-use-pep517
[...]

real 0m1.919s
user 0m1.486s
sys 0m0.413s

The overhead of PEP 517 processing comes from two main sources:

The PEP 517 interface itself. To build a project in the PEP 517 way, pip must call Python hooks provided by the configured build backend. For example, PEP 517 backends must supply a build_wheel function which when called builds the project and returns the path to the resultant wheel. All of these calls are mandated to occur in fresh subprocesses which each incur a Python startup/shutdown penalty. Yikes!
Build isolation. In PEP 517 mode, pip by default sets up a temporary isolated environment to perform the build in. The build backend and any additional build dependencies are installed to this temporary environment on the fly.

In practice, pip has to call effectively the same hooks even under the legacy mechanism so the design is not actually any slower.² The additional overhead is from build isolation. Creating the temporary environment is cheap, but installing build dependencies is very much not.

A major part of the install time is spent on accessing the network. Yes, pip is checking PyPI (or the configured index) to determine the most up to date versions of build dependencies every time it sets up a build environment. This is why building from source often requires active Internet access with modern pip.³

However, another major source of the overhead is from invoking a pip in a subprocess to install build dependencies. Large Python applications are rather slow to start up and pip is no exception. For example, pip install --help takes ~220ms to execute on my system.

Starting a new pip process also comes with other cons, including numerous bugs related to configuration not being passed down and abysmal error reporting.

Consequently, we’ve wanted to replace this mechanism and switch to installing build dependencies within the same root pip process for years.

The good news is that this work is finally starting to happen!

I’ve published a draft PR that would add an alternative mechanism which installs build dependencies in-process. There will be behavioural differences compared to the current subprocess installation method, but those will be covered in a future post when the feature is generally available for testing. The good news is that it is already demonstrating performance improvements.

$ time pip install . --use-feature=inprocess-build-deps
[...]

real 0m2.672s
user 0m1.836s
sys 0m0.483s

In comparison with the initial example, 300ms has been saved which is in line with the startup overhead of the pip install command.⁴ The configuration inheritance story related to build dependencies will be cleaned up as part of this effort, too.

There remains considerable work (notably around error handling and adding more tests) before the PR is ready for review, but if everything goes to plan, it should be available in pip 25.3 as an experimental feature.

Well, it should be supported, anyway. There were lots of test failures on Python 3.14 due to URL<->path conversion issues. They were caused by a mix of upstream changes in pathname2url() and url2pathname() and then undiscovered bugs in pip’s URL<->path utilities. It was hell, and frankly, I’m not 100% sure everything is right as I am barely an expert on file URLs (even after reading RFCs for an hour). ↩︎
Although the design does serve to limit the lowest achievable build times even with well-optimized tools like uv. ↩︎
We recognize that it’d be nice if pip supported an offline mode where pip fell back to using whatever packages were already stored in the HTTP or wheel caches, but unfortunately, this would require a major refactoring of the caching stack. If you’d like to help us out and suggest a proposal that’s feasible with the limited resources the pip project has, you are more than welcome to do so. ↩︎
Remember that there is even more overhead beyond initial import times. A new pip process also needs to configure the network stack (which can be quite slow on some systems), load configuration, and shut down properly. ↩︎

What's new in pip 25.1 - Dependency groups!

Fri, 25 Apr 2025 00:00:00 -0400

On April 26, 2025, we, the pip team released pip 25.1.

Compared to previous releases, this release is a large one. The changelog is quite long. Among the changes, there are numerous new features, including an install progress bar and support for Dependency Groups!

I won’t cover every single change here, so as always, please refer to the changelog for the full list of changes.

Key features ✨

Dependency groups (PEP 735)

Support for installing Dependency Groups (PEP 735) is now available with the --group option.

What are Dependency Groups?

Dependency Groups are the modern standardized replacement for requirements.txt files and extras used for development workflows. Dependency Groups are NOT published nor installable by the end user, unlike extras.

They are meant to replace files like test-requirements.txt or extras like [test], [dev], or [doc]. They are defined in a pyproject.toml file under the dependency-groups table.¹

First, let’s assume there is a pyproject.toml file in the current directory that defines test and lint dependency groups:

pyproject.toml

[dependency-groups]
test = ["pytest", "pytest-xdist"]
lint = ["mypy", "isort"]
# Dependency Groups can include other groups! ✨
dev = [ {include-group = "test"}, {include-group = "lint"} ]

To install the test group, one can simply pass --group test.

$ pip install --group test

You can repeat the option to install multiple groups. You can also use --group with -r, -e, and any other dependency specifier. Want to install the local project, two groups, and a requirements.txt? You bet you can.

$ pip install . --group lint --group test -r my-old-requirements.txt

To install a dependency group from a pyproject.toml file that exists in a different directory, state the path before the group separated with a colon.

$ pip install --group ./dev/vem/pyproject.toml:dev

Warning

pip will NOT look in parent directories for a pyproject.toml file to install dependency groups from. pip does not have the concept of a project workflow.²

Additionally, Dependency Groups are a standardized feature, thus pip-specific options are NOT supported. If you need to pass --no-deps or some other pip flag, you should continue to use a requirements.txt file.³

Please read the pip user guide for more information. You can also read the motivation and rationale sections of PEP 735 if you are curious about why this feature came about.

Thank you to Stephen Rosen for writing and shepherding PEP 735 and contributing the pip implementation. Congratulations! 🎉

Package installation progress bar

A progress bar has been added to pip install that tracks the installation step.

Previously, pip would provide no feedback on how many packages have been installed. Is pip slowly installing a large package (e.g., torch) or is pip simply stuck? Who knows! The only time you do know is when pip first needs to uninstall an existing package to install a new version, emitting Attempting uninstall: mypackage as it goes.

Now, you can install every single Home Assistant dependency and know much progress pip has made after dependency resolution and downloads.

Resumable downloads

Support for automatic download retrying is available as an experimental feature starting with pip 25.1. The download retry limit can be configured using the --resume-retries option.

Important

Download retrying (--resume-retries) is separate from connection retrying (--retries).

When a download terminates early, pip will attempt to resume the download from where it left off. If the remote server does not support download resumption, pip will fall back to restarting the download. If the download is still incomplete after pip has run out of retries, a diagnostic error will be raised.

It has been a long-standing complaint that pip is unreliable on poor/flaky connections. Any downloads of a sufficiently large package would fail midway through, forcing the user to retry the entire pip command, only for it to fail again. For users on a slow or metered connection, pip is likely unusable. And if hash-checking was enabled, then a confusing and misleading hash mismatch error would be raised.

If you use the feature, please let us know how it goes! We want to make sure it works well. Assuming nothing is fundamentally broken, --resume-retries will be changed in the next release to allow a few download retries by default.

Thank you to George Margaritis for contributing this feature!

Experimental lockfile (PEP 751) generation: `pip lock`

pylock.toml (PEP 751) is the recently accepted standard for Python lockfiles. Lockfiles are alternatives to the classic requirements.txt format, designed to enable reproducible installation in a Python environment.

pip 25.1 makes the first step at supporting the new lockfile format. The pip lock command has been added to create a pylock.toml from a set of requirements.

Here’s an example locking the pip project in editable mode and six.

$ pip lock -e ./pip six --output - -qq
lock-version = "1.0"
created-by = "pip"

[[packages]]
name = "pip"

[packages.directory]
path = "pip"
editable = true

[[packages]]
name = "six"
version = "1.17.0"

[[packages.wheels]]
name = "six-1.17.0-py2.py3-none-any.whl"
url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl"

[packages.wheels.hashes]
sha256 = "4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274"

The produced lockfile is specific to the Python version and platform pip is invoked under. In other words, the lockfile is NOT a universal lockfile.

The command is experimental, meant to enable basic locked installation scenarios. It is expected to undergo further development once the lockfile standard sees more widespread adoption and matures.

Installing from a lockfile is unsupported, but it is on the roadmap.

Congratulations to Brett Cannon for getting a lockfile standardized after many years(!) of PEP drafting and several rounds of discussion. I look forward to seeing pylock.toml mature!

`pip index versions` is stable

The pip index versions command is no longer experimental. It is the modern replacement for the old pip install mypackage== hack which used to be an easy way to get the list of all available versions.

$ pip index versions six
six (1.17.0)
Available versions: 1.17.0, 1.16.0, 1.15.0, 1.14.0, 1.13.0, 1.12.0, 1.11.0, 1.10.0, 1.9.0, 1.8.0, 1.7.3, 1.7.2, 1.7.1, 1.7.0, 1.6.1, 1.6.0, 1.5.2, 1.5.1, 1.5.0, 1.4.1, 1.4.0, 1.3.0, 1.2.0, 1.1.0, 1.0.0, 0.9.2, 0.9.1, 0.9.0
 INSTALLED: 1.17.0
 LATEST: 1.17.0

In addition, you can now ask for JSON output using the --json flag.

$ pip index versions six --json
{"name": "six", "versions": ["1.17.0", "1.16.0", "1.15.0", "1.14.0", "1.13.0", "1.12.0", "1.11.0", "1.10.0", "1.9.0", "1.8.0", "1.7.3", "1.7.2", "1.7.1", "1.7.0", "1.6.1", "1.6.0", "1.5.2", "1.5.1", "1.5.0", "1.4.1", "1.4.0", "1.3.0", "1.2.0", "1.1.0", "1.0.0", "0.9.2", "0.9.1", "0.9.0"], "latest": "1.17.0", "installed_version": "1.17.0"}

If you use pip index versions programmatically, you are encouraged to switch to the --json output. Not only is it easier to parse, but you will avoid breakage if the human-friendly format ever changes later.

Thank you to Krishan Bhasin for stabilizing pip index versions and adding JSON support!

Key bugfixes

Dependency resolution improvements

Dependency resolution is not my area of expertise, so I’m going to shamelessly steal the changelog entries that Damian Shaw wrote:

Speed up resolution by first only considering the “priorities” of candidates that must be required to complete the resolution.
Improved heuristics for determining the order of dependency resolution by preferring direct URL candidates and requirements with upper bounds.⁴
The upgrade to ResolveLib 1.1.0 fixes a known issue where pip would report a ResolutionImpossible error even though there is a valid solution. However, some very complex scenarios that previously resolved may resolve slower or fail with an ResolutionTooDeep error.
The ResolutionTooDeep error has been converted into a more user-friendly diagnostic error. If you have any real-world examples that result in a ResolutionTooDeep error, we would like to hear them!

Legacy `.egg` distributions are only detected once

There are several formats that an installed package can be recorded in. Today, virtually every package is installed using the .dist-info format. However, legacy .egg distributions are still kicking around.

Support for .egg distributions has been deprecated since pip 23.2. In pip 25.1, this deprecation was finalized… but wait, pip still supports .egg distributions! It turns out that importlib.metadata, the library used to discover installed packages on Python 3.11+, supports .egg distributions just fine.

This meant that .egg distributions were being discovered several times. The importlib.metadata backend would find it, and then the pkg_resources backend, leading to this confusing pip list output.

$ pip list
Package Version
----------------------------- ---------------------
mypackage 0.7.16
mypackage 0.7.16

In pip 25.1, the pkg_resources backend is no longer used to discover .egg distributions and this duplication will not occur. Warnings about the .egg deprecation will no longer be printed.

Deprecations & upcoming removals

Starting with this release, the pip project no longer supports Python 3.8.

In addition, here is a list of current deprecations and the release they will be removed. As always, any given removal may be pushed to a future release as needed.

See the pip 25.0 post for more details.

This is a continuation of the old deprecation as the original deprecation did not catch all non-standard wheel filenames. See the pip 25.0 post for more details.

Acknowledgements

Thank you to Damian Shaw, Emma Smith, Hugo van Kemenade, and Stéphane Bidoul reviewing the draft of this post. Any typos or glaring mistakes are my own.

They are not defined under the project table because Dependency Groups are meant to be a general feature that work regardless of whether you have an actual Python package to install or not. ↩︎
There has been extensive discussion about pip’s fundamental design and whether it should change. There is too much to easily summarize, but broadly, pip has always been designed as a package manager, and not as a project workflow tool. There are many other tools that compete in the project management space. The consensus during review is that pip is valuable as a foundational tool that simply manages your installed Python packages and does what you want without too much fuss. More practically, shifting to a project-based model would constitute a major redesign of the pip UI and UX. ↩︎
This is a limitation of the specification. However, if sufficient consensus builds around certain tool-specific flags, they can be standardized so they can be included in Dependency Groups in the future. ↩︎
Please don’t add upper bounds unless you know for a fact your codebase is incompatible with the newer versions. Pre-emptive upper-bounds will not help dependency resolution and are generally considered harmful. ↩︎

Speeding up pip's Windows CI by setting TEMP to the D: drive

Mon, 10 Mar 2025 00:00:00 -0400

TL;DR

If your GHA Windows jobs are I/O heavy, double check whether the I/O work is happening on the C: drive (OS) or D: drive (working space). If the C: drive is being used, you will likely benefit by moving that work to the D: drive. If you’re feeling more adventurous, you should also consider trying a ReFS/Dev Drive.

A few weeks ago, I was waiting for pip’s CI to turn 🍏 so I could merge a PR. I’d just spent an hour on review and had pushed a few final tweaks. Waiting an additional 20 minutes was the last thing I wanted to do.¹

I was sufficiently annoyed with checking and re-checking the CI status, so I enabled auto-merging on the repository so I could at least focus my attention on something entirely different while waiting for CI to pass. That smoothed over the pain, but it got me wondering, can pip’s tests (and CI in particular) be faster?

It turns out for the Windows CI jobs, the answer was: yes, they could be much faster.

How slow was pip’s Windows CI?

At the time of writing², pip’s Github Actions CI workflow consists of 28 jobs. 23 run the test suite across Ubuntu, macOS, Windows, and CPython 3.8 through 3.13.

7 on the ubuntu-latest runners
6 on the macos-latest (ARM) runners
6 on the macos-13 (Intel) runners
4 on the windows-latest runners

The extra Ubuntu job runs the functional test suite against a zipapp version of pip.

On the other hand, the Windows jobs have historically been so slow that we don’t even test pip across the entire supported Python version matrix. We only test the oldest and newest version, 3.8 and 3.13, on Windows. Plus, the test suite is sharded over two jobs per version because—as you can probably guess—the Windows jobs are that much slower 🐌.

I wrote a script that queries the GitHub API and calculates the average job duration over the last X successful runs. As a baseline, this is what pip’s CI looked like:

 Last 50 CI runs on main

 Job Mean Minimum Stdev
 ─────────────────────────────────────────────────────────────
 tests / 3.10 / macos-latest 6:25 5:44 0:34 (8%)
 tests / 3.11 / macos-latest 6:40 6:03 0:39 (9%)
 tests / 3.9 / macos-latest 6:57 6:03 0:55 (13%)
 tests / 3.13 / macos-latest 6:59 6:15 0:37 (8%)
 tests / 3.12 / macos-latest 7:09 6:14 1:01 (14%)
 tests / 3.8 / macos-latest 7:11 6:32 0:31 (7%)
 tests / 3.11 / ubuntu-latest 9:33 9:08 0:10 (1%)
 tests / 3.10 / ubuntu-latest 10:04 9:43 0:14 (2%)
 tests / 3.12 / ubuntu-latest 10:34 10:16 0:14 (2%)
 tests / 3.13 / ubuntu-latest 10:35 10:10 0:12 (2%)
 tests / 3.13 / Windows / 2 10:39 9:55 0:20 (3%)
 tests / 3.8 / ubuntu-latest 11:02 10:41 0:14 (2%)
 tests / 3.13 / Windows / 1 11:13 10:29 0:24 (3%)
 tests / 3.9 / ubuntu-latest 11:44 11:20 0:19 (2%)
 tests / 3.10 / macos-13 12:10 8:53 2:16 (18%)
 tests / 3.11 / macos-13 12:59 9:09 2:34 (19%)
 tests / 3.13 / macos-13 12:59 9:54 2:41 (20%)
 tests / 3.9 / macos-13 13:10 10:09 2:42 (20%)
 tests / 3.8 / macos-13 13:34 10:09 2:30 (18%)
 tests / 3.12 / macos-13 13:55 9:54 3:23 (24%)
 tests / 3.8 / Windows / 2 14:58 14:04 0:26 (2%)
 tests / zipapp 16:53 16:27 0:15 (1%)
 tests / 3.8 / Windows / 1 17:37 16:40 0:30 (2%)

Among the slowest three jobs, the Windows 3.8 jobs hold the 3rd and 1st places for duration. Yikes!

Windows file I/O is poor on GitHub Actions

It’s a known issue that file I/O on the Windows runners is noticeably slower than the Ubuntu and macOS runners.

As a package installer, pip’s test suite is especially I/O heavy. Most functional tests require at least setting a scratch environment and installing something into it. Countless files and directories are created, opened, and deleted during a run.

Moving TEMP to the D: drive

According to Steve Dower, the GHA Windows workers have a slow (but read-optimized) C: system drive and a faster D: drive for working space.

Warning

Only the standard Windows runners have a D: drive. The larger (paid) Window runners do NOT have a D: drive. For such situations, ReFS/Dev Drives should be considered.

For many projects, this is probably fine. The repository is still checked out on the D: drive. Any files you create in the checkout will be on the D: drive. For pip’s test suite, however, all of the temporary files are created under the system temporary directory. Where is this path set to by default? On the C: drive at C:/Users/runneradmin/AppData/Local/Temp/.³

Thus, pip’s CI was being slowed down by reading and writing primarily to the non-performant disk.

The good news is that this can be overridden by setting the TEMP environment variable which Python’s tempfile will respect.

 - name: Set TEMP to D:/Temp
 run: |
 mkdir "D:\\Temp"
 echo "TEMP=D:\\Temp" >> $env:GITHUB_ENV

This simple switch to D:/Temp translated into the Windows CI time decreasing by up to 30%. 🎉

Job	Before	After	% decrease
Python 3.8 (1)	17m 37s	12m 41s	28%
Python 3.8 (2)	14m 58s	10m 44s	28%
Python 3.13 (1)	11m 13s	9m 53s	11%
Python 3.13 (2)	10m 39s	9m 32s	10%

Or in graph form (note: the data shown here is slightly more recent than the statistics).

What about ReFS or Dev Drives?

When I was investigating ways to improve file I/O performance on Windows, I’d discovered Dev Drives. Now, I am no Windows expert. I stopped using the OS on a daily basis many years ago, but the TL;DR is they are a new kind of storage volume, built on the modern ReFS (Resilient File System) filesystem and designed with developer workloads in mind.

The uv project has had good success with using ReFS to speed up their Windows GHA jobs, thus I actually started with a ReFS drive⁴ while optimising pip’s Windows CI. Unfortunately, using a ReFS drive was noticeably slower than simply moving TEMP to the D: drive. ReFS did save ~a minute for Python 3.8 (1) but that pales in comparison with the nearly five minutes improvement from TEMP=D:/Temp.

The performance improvements of ReFS/Dev Drives are primarily realized by deferring malware scans (Dev Drive only) and the inherent optimizations of the newer ReFS filesystem. Presumably the malware scans are already disabled on GitHub Actions, which would explain why pip’s CI did not see as big of an improvement as I was expecting.

Given uv’s success with ReFS, I can’t universally recommend using the D: drive over ReFS/Dev Drives, and vice versa. Which one will be faster is likely workload dependent. If you would like to experiment with ReFS/Dev Drives, my Powershell script to create such a drive can be found here. To invoke it, simply pass the desired drive letter and volume size: devdrive.ps1 -Drive R -Size 4GB.

Happy experimenting!

Before you kill me for complaining, let me provide some context… When I used to maintain Black, I never really noticed how CI took. Most of the jobs would finish in 3-6 minutes. Short enough that I could look over my changes one last time, add notes as comments, and then come back to merge.

I did notice that Black’s test suite was slow, taking a good few minutes on the slow laptop I had at that time. I took a look at the test logic, trimmed redundant work, and added pytest-xdist for parallelized testing. Afterwards, local test suite runs took less a minute!

I was spoiled by Black’s fast tests and CI, admittedly. ↩︎
I wrote the majority of this post in January… ↩︎
We’ve actually set it to C:/Temp to work around filename too long errors. Also, the exact default system temporary directory may vary from worker to worker. I don’t feel like double checking this. ↩︎
When I did my experiments, the Windows GHA runners did not support Dev Drives. Only ReFS was supported. ↩︎

What's new in pip 25.0

Fri, 31 Jan 2025 00:00:00 -0500

On January 26, 2025, the pip team released pip 25.0.

Before delving into the key changes, I’d like to mention that the pip project has two new maintainers: Damian Shaw and Richard Si (that’s me!). I want to thank the pip core team for entrusting me with the commit bit on such a foundational project like pip.

Also, this is the first release cut from a GitHub Actions CI/CD workflow that implements Trusted Publishing and bundles PEP 740 digital attestations.

As always, please read the changelog for the full list of changes.

Key features

PEP 639: SPDX License Expressions

About 5 months ago, PEP 639 – Improving License Clarity with Better Package Metadata was approved.

It added the License-Expression core metadata field, which is mandated to contain a valid SPDX expression. In addition, there is now a multi-use License-File field so packages can declare the path to their licensing-related files.

Warning

The old free-form License field and license classifiers are deprecated, although there are no current plans to remove them from the metadata specification.

It took a few months, but it is now generally supported by build backends, publishing tools, and PyPI. With this release, pip show will display the License-Expression over the License field when available.¹

$ pip install prettytable==3.13.0
$ pip show prettytable
Name: prettytable
Version: 3.13.0
Summary: A simple Python library for easily displaying tabular data in a visually appealing ASCII table format
Home-page: https://github.com/prettytable/prettytable
Author:
Author-email: Luke Maurits <luke@maurits.id.au>
License-Expression: BSD-3-Clause
Location: /home/ichard26/.local/lib/python3.12/site-packages
Requires: wcwidth
Required-by:

Furthermore, the License-Expression and License-File fields are included in the JSON report emitted by pip install --report and pip inspect.

Thank you Philippe Ombredanne, C.A.M. Gerlach, and Karolina Surma for authoring the PEP! Doubly so to Karolina who drove the PEP to completion and contributed part of pip’s implementation.

Network cache inherits permissions from root cache directory

It’s easiest to explain this change by quoting the original issue:

On a shared Linux system, we want to share pip’s cache between multiple users, so packages are not downloaded 50 times when 50 users install the same package.

Problem is that pip would set the permissions for network cache entries so that the user running pip is the sole user able to access them. This wasn’t intentional, but a byproduct of tempfile.NamedTemporaryFile.

Now, pip will update the entries’ permissions to inherit the read/write permissions set on the root cache directory. Thus, if the root cache directory is accessible by other users, those users will be able to use any entries added later.² Thank you to Justin van Heek for contributing this improvement!

Key bugfixes

First off, here’s a rapid-fire list of smaller, but still noteworthy bugfixes:

Fix a security bug allowing a specially crafted wheel to execute code during installation.

You should read this issue for all of the details, but essentially, a deferred import of pip’s self check could be abused to run arbitrary code at the end of the same pip install. Reported and fixed by Caleb Brown.
The inclusion of packaging 24.2 changes how pre-release specifiers with < and > behave.

Including a pre-release version with these specifiers now implies accepting pre-releases (e.g., <2.0dev can include 1.0rc1). To avoid implying pre-releases, avoid specifying them (e.g., use <2.0). The exception is !=, which never implies pre-releases.

Certificate and proxy options are used for build dependencies

When pip needs to build a project for installation or metadata³, pip by default sets up an isolated environment to install the build backend and its dependencies in. These build dependencies are installed by calling pip recursively in a subprocess.

Unfortunately, the --cert, --client-cert, and --proxy options were not passed down to the pip subprocesses. However, --index always⁴ has been, resulting in the common problem where pip tries to install build dependencies from a private index that uses a custom certificate… that it can’t verify.

Before pip 25.0, the workaround was to use environment variables (e.g., PIP_CERT) as those are passed down to the subprocess by the OS. This is no longer necessary.

The --proxy patch was authored in part by Luis Carlos. Thanks!

Truststore regression

While fixing this, truststore was inadvertently disabled while installing build dependencies. Truststore enables system CAs to work without any configuration. This will be fixed in an upcoming pip 25.0.1 bugfix release.

“that seems silly!”

Admittedly, the whole “pip calling itself” logic is not ideal. There is a performance penalty to starting a new pip process, numerous bugs, unexpected behaviour caused by forgetting to pass enough state (i.e. pip flags), and the error reporting is abysmal.

We would like to pay down the debt that has accumulated here, but installing build dependencies in-process is likely to require substantial refactoring as the codebase was not designed w/ repeated resolve-download-install cycles in mind.

Faster package candidate collection

When pip initially fetches the list of distributions available for each package, it needs to parse the HTML or JSON returned by the index and filter out the distributions that obviously aren’t supported by the current platform.

This release eliminated redundant URL parsing and similar processing, especially for indices which serve absolute URLs to their distribution files (e.g., PyPI). Furthermore, requires-python checks are cached.

The URL processing and requires-python checks are fast individually, but they add up once you start to use packages with hundreds of releases and thousands of distributions. A good example is setuptools. It has ~1500 different distributions that could be eligible for installation, leading to a fair bit of overhead:

`pip install setuptools --dry-run`	pip 24.3.1	pip 25.0
Index page parsing	90ms	45ms
Candidate evaluation	130ms	30ms

Obviously the network overhead usually dominates in practice, but this is still a good improvement! And since setuptools is a common backend, this makes the build environment setup phase of installing from source just a bit faster for many projects. 🏎️

Upcoming removals

Finally, here is a list of deprecations currently scheduled for removal in pip 25.1 (Q2 2025). As always, this schedule is subject to change. Any given removal may be pushed to a future release as needed.

Legacy (setup.py) editable installs (Deprecated since pip 24.2): I’ve already said enough on this deprecation, so I’m going to simply ask that you read the deprecation issue for information and advice. This was originally scheduled for removal in 25.0, but it got pushed back as the removal wasn’t finished in time.
Legacy installed .egg distributions (Deprecated since pip 23.2): (#12330) On Python 3.11 or later, support for detecting and uninstalling installed .egg distributions is deprecated. Eggs are the legacy format used to record installed distributions. It has been long since replaced by .dist-info. Today, they’re most often the product of using setup.py install. The recommended solution is to reinstall the distribution using a recent pip. FWIW, this removal has been pushed back several times.
Non-bare project name in egg fragment (Deprecated since pip 23.0): (#13157) If you’re ever seen #egg=mypkg at the end of an URL requirement, that’s called an egg fragment. It’s the old syntax used to tell pip what project the URL is for. They’ve been largely replaced by the standardized Direct URLs references. When egg fragments were initially added, they were meant to only contain the project name. You can technically put a version specifier in the fragment, but pip won’t respect it. To avoid confusion, anything but a bare project name is deprecated. However, the removal is blocked on adding support for requesting extras via the Direct URL syntax for VCS references.
Non-standard wheel filenames (Deprecated since pip 24.3): (#12938) For historical reasons, pip does its own parsing of wheel filenames. We’d like to replace this with the reference parser in packaging. The side-effect is that wheels that violate the wheel filename specification are now rejected. Unless you use old packaging tooling or play shenanigans with your project name or wheel filenames, you shouldn’t be affected.
–no-python-version-warning (Deprecated since pip 25.0): (#13154) The flag has long done nothing since Python 2 support was removed in pip 21.0. pip has also never warned about deprecations of any other Python version as the flag’s help suggests.

As long as the package’s metadata version is 2.4 or higher. There was a bit of a kerfuffle as PEP 639 was implemented by certain backends before being accepted, thus they generated distributions without metadata 2.4 declared (because it didn’t exist before approval). This is technically invalid, and pip wishes to enforce specification compliance. ↩︎
Of course, this doesn’t help with old cache entries with the overly restrictive permissions. Plus, everyone must be using pip 25.0 or else there will be a mix of entries with the right and overly restrictive permissions, rendering the shared cache useless. ↩︎
For example, during dependency resolution, pip may need to build metadata so it knows what requirements mypkg needs. Building a wheel and inspecting its metadata is a valid approach, but PEP 517 allows pip to ask a backend for only metadata. ↩︎
Maybe not always, but definitely for a long time. ↩︎

What's new in pip 24.3

Thu, 14 Nov 2024 00:00:00 -0500

On October 27, 2024, the pip team released pip 24.3. This release was a significantly smaller release with only a handful of minor fixes and improvements.

This write-up also includes the hotfixes included in pip 24.3.1 released on the same day.

Thank you to Damian Shaw, Hugo van Kemenade, and Stéphane Bidoul for reviewing the draft of this write-up.

PSA: Legacy editable installs are deprecated

TL;DR, don’t panic. The -e option is not deprecated, but the way it works under the hood will change, potentially necessitating changes to how your project is packaged.

If you aren’t aware, since pip 24.2, legacy editable installs are deprecated and support is scheduled for removal in pip 25.0 in the new year (at the time of writing).

$ pip install -e temp/pip-test-package
Obtaining file:///home/ichard26/dev/oss/pip/temp/pip-test-package
 Preparing metadata (setup.py) ... done
Installing collected packages: version_pkg
 DEPRECATION: Legacy editable install of version_pkg==0.1 from file:///home/ichard26/dev/oss/pip/temp/pip-test-package (setup.py develop) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457
 Running setup.py develop for version_pkg
Successfully installed version_pkg-0.1

Editable installs are not deprecated, but pip will stop running the setuptools specific setup.py file under the hood. Instead, pip is transitioning to exclusively using the standardized mechanism to request an editable install from the project’s backend defined in PEP 660.

If you’re confused, don’t worry. The implementation details are admittedly a bit complex, but all you need to know is that:

You will see this deprecation notice if the package you are trying to install has setuptools as its build backend AND:
- EITHER this package does not have a valid build system declaration, thus pip will opt-out of the modern mechanisms and run setup.py directly as required
- OR this package’s build system declaration explicitly requests a version setuptools older than version 64 which lacks support for the modern editable installation mechanism (PEP 660)
The easiest way to address the deprecation is to add a pyproject.toml file that declares your package’s build system to be setuptools (see the guide “How to modernize a setup.py based project?” for more details)
You can keep your setup.py file. It’s a configuration file for setuptools, and setuptools still supports it. pip simply won’t be running it directly anymore, delegating to setuptools (see the discussion “Is setup.py deprecated?” for more details).
- If you wish, you can migrate your packaging setup to pyproject.toml entirely, but this isn’t necessary

For more advice, please see the issue tracking this deprecation and this comment. You can also read my write-up on pip 24.2 which discusses the context behind the deprecation.

QoL improvements

Recursive requirement files are now detected

Imagine you had a requirement file that recursively included itself:

-r requirements.txt

If you ran pip install -r requirements.txt, you’d get a nasty crash as pip kept following the loop until it reached the recursion limit:

ERROR: Exception:
Traceback (most recent call last):
[trimmed...]
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/cli/req_command.py", line 255, in get_requirements
 for parsed_req in parse_requirements(
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 151, in parse_requirements
 for parsed_line in parser.parse(filename, constraint):
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 332, in parse
 yield from self._parse_and_recurse(filename, constraint)
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 361, in _parse_and_recurse
 yield from self._parse_and_recurse(req_path, nested_constraint)
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 361, in _parse_and_recurse
 yield from self._parse_and_recurse(req_path, nested_constraint)
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 361, in _parse_and_recurse
 yield from self._parse_and_recurse(req_path, nested_constraint)
 [Previous line repeated 974 more times]
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/req/req_file.py", line 337, in _parse_and_recurse
 for line in self._parse_file(filename, constraint):
[trimmed...]
RecursionError: maximum recursion depth exceeded

This example is admittedly a bit contrived, but it can happen if you’re referencing requirement files in a chain and forget what you’ve already included. Now, pip will detect this loop and raise a proper error.

$ pip install -r test.txt
ERROR: /home/ichard26/dev/oss/pip/test.txt recursively references itself in test.txt

This was contributed by Kuntal Majumder (@hellozee) in PR #12877. ✨

(There was a bug where the error would be raised too often, even when no actual recursion was present. This was promptly addressed in pip 24.3.1.)

Installed packages with broken dependency metadata now raise a proper error

Since pip 24.1, non standard-compliant versions or dependency specifiers are unsupported. While pip will ignore packages with this legacy invalid metadata when possible, in other cases, the offending package had to be uninstalled to allow pip to function.

Unfortunately, similar to the issue above, pip would just crash and leave the user a horrible traceback in these cases.

$ pip install celery
Requirement already satisfied: celery in ./venv/lib/python3.12/site-packages (4.4.7)
ERROR: Exception:
[trimmed...]
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/resolution/resolvelib/candidates.py", line 401, in iter_dependencies
 for r in self.dist.iter_dependencies():
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/metadata/importlib/_dists.py", line 215, in iter_dependencies
 req = get_requirement(req_string.strip())
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/ichard26/dev/oss/pip/src/pip/_internal/utils/packaging.py", line 45, in get_requirement
 return Requirement(req_string)
 ^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/ichard26/dev/oss/pip/src/pip/_vendor/packaging/requirements.py", line 38, in __init__
 raise InvalidRequirement(str(e)) from e
pip._vendor.packaging.requirements.InvalidRequirement: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
 pytz (>dev)
 ~^

To address this, a new diagnostic error has been added when encountering a broken pre-installed package.

$ pip install celery
Requirement already satisfied: celery in ./venv/lib/python3.12/site-packages (4.4.7)
error: invalid-installed-package

× Cannot process installed package celery 4.4.7 in '/home/ichard26/dev/oss/pip/venv/lib/python3.12/site-packages' because it has an invalid requirement:
│ Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
│ pytz (>dev)
│ ^
╰─> Starting with pip 24.1, packages with invalid requirements can not be processed.

hint: To proceed this package must be uninstalled.

Admittedly, the error is a bit hard to read, but with time, pip’s errors should improve. In the interim, you can quickly tell what package is causing the problem (celery) and how to fix it (uninstall it).

macOS 10.12 is resupported

If you’re part of the tiny group that still uses modern pip on macOS 10.12, you would’ve learned first-hand that truststore did not support macOS 10.12, breaking pip. Good news though, pip 24.3 restores support by vendoring in an updated version of truststore that uses older macOS APIs when needed.

This brings the pip project one step closer to being able to use system HTTPS certificates when possible, all without blowing up in odd situations. There are still some known issues with private certificates, however.

Invalid wheel filenames are deprecated

Note: Unless you use ancient packages before the advent of modern packaging tooling AND are incredibly unlucky, this deprecation will NOT affect you. This is mostly for those interested in the packaging nitty gritty.

If you have any sort of familiarity with Python packaging, you’d have heard of wheels. Wheels are a binary distribution format for Python packages. They are pre-built, which makes installing them particularly easy: just unzip them and copy the contents to the site-packages directory.¹

Something you might not know is that their filenames follow a certain structure. This enables installers to extract essential metadata without inspecting the wheel’s contents. As an example, let’s take a look at the newest wheel for pip.

pip-24.3.1-py3-none-any.whl

Here, we know three pieces of information immediately:

pip is the package name
24.3.1 is the version
py3-none-any is the compatibility tag. I went over compatibility tags in my previous pip write-up, but essentially, py3-none-any is code for this package supports “any Python 3 version”, “any Python ABI”, and “any architecture.”

Unfortunately, some wheels out in the wild have improper names, such as:

translate_toolkit-1.9.0_pinterest3-py27-none-any.whl
j2cli-0.3.0_1-py2-none-any.whl
password_strength-0.0.1_1-py2-none-any.whl

According to the version specifier standard, underscores can be used to separate the release segment from any pre/post/development release segments. However, they cannot be used to separate an implicit post-release.

In other words, 1.2.0_post1 (explicit) is valid, but 1.2.0_1 (implicit) is not.²

For historical reasons, pip has accepted wheels with this error. This is inconsistent and confusing, thus starting with pip 24.3, these wheels are no longer supported.

Frankly, virtually no packages today have this issue, so you can forget this deprecation even exists³, but it is a perfect opportunity to learn about the complexity found in Python packaging. Many things may look easy in packaging, but often they are not :)

Other changes

iOS wheels are provisionally supported: pip will now recognize and accept iOS tagged wheels, although whether pip properly supports them is something I don’t know.
Installing pip in editable mode in a virtual environment on Windows doesn’t result in a data race crash anymore
The --target option is ignored while pip prepares a build environment during installation, which often caused confusing crashes when it was configured globally in a pip configuration file or via the PIP_TARGET environment variable.

Of course, it’s not this simple in reality, but there is no “build step” involved which makes them trivial compared to source distributions (typically .tar.gz). ↩︎
To declare a post-release without the post prefix, you must use the - separator, e.g. 1.2.0-1. Naturally, this raises the question of what to do when this version is placed in the wheel filename as dashes are reserved as wheel filename separators. The answer is that the version should be normalized when placed in a wheel filename, and fortunately for us, 1.2.0-1 normalizes to 1.2.0.post1. ↩︎
While reviewing the deprecation PR, I scraped the index pages for the top 8000 PyPI packages and could only find 4 ancient wheels that have this quirk in their filename. ↩︎

What's new in pip 24.2 — or why legacy editable installs are deprecated

Mon, 26 Aug 2024 00:00:00 -0400

On July 28, 2024, the pip team released pip 24.2. As a recent addition to the pip core team, I’ll walk you through the noteworthy or interesting changes from pip 24.2. Let’s go!

PSA, PSA, PSA 📣
If you’re using setuptools, do not have a pyproject.toml, and use pip’s -e flag, you may be impacted by the deprecation of legacy editable installs.

If you’re an user of setuptools and came to this post after seeing the legacy editable install deprecation warning, you should first read the deprecation issue for solutions, not here.

You can come back and read this after for more historical context on the deprecation, however! It’s a good read, I promise.

Here’s a link to the 24.2 changelog if you’d like the full list of changes.

Legacy editable installs are deprecated

TL;DR, don’t panic. The -e option is not deprecated, but the way it works under the hood will change, potentially necessitating changes to how your project is packaged.

Over the last decade, there has been a major transition towards standardized mechanisms for packaging Python projects:

PEP 517: introduced an interface where frontends (e.g. the pip installer) can interact with a build backend (e.g. Poetry¹), including asking it to build a wheel for later installation
PEP 518: introduced pyproject.toml and defined a standard format for declaring build-time dependencies using this file
PEP 621: defined a standard format for declaring project metadata in pyproject.toml
PEP 660: augmented the PEP 517 interface to define a way to ask a build backend to prepare an editable install

These PEPs are what enable the use of alternative backends like Poetry, Hatch, and scikit-build-core without needing to add specific support for each backend in every frontend (e.g. pip, tox, uv). Instead of running setup.py bdist_wheel to ask setuptools to build a wheel for installation, pip can simply call the build_wheel hook² from PEP 517 which all backends are guaranteed to have.

As each standard matured, pip has progressively deprecated and removed support for legacy (and often setuptools-specific) mechanisms to reduce technical debt and ensure that pip can continue to evolve.

Editable installs—the thing you get from pip install -e <path>—used to be a setuptools only feature, requiring pip to call the project’s setup.py with the develop sub-command. However, this stopped being the case under PEP 660. As long as your project’s backend supports PEP 660, pip install -e will continue to work. No setuptools required. (Although modern setuptools works fine as well.)

OK, so what now?

This release, pip has deprecated support for the setup.py develop fallback which is used when a project lacks support for modern editable installs. It will be removed in pip 25.0 (Q1 2025) after which projects MUST support PEP 660 to perform an editable install. Affected projects will see this deprecation warning:

pip install -e temp/pip-test-package
Obtaining file:///home/ichard26/dev/oss/pip/temp/pip-test-package
 Preparing metadata (setup.py) ... done
Installing collected packages: version_pkg
 DEPRECATION: Legacy editable install of version_pkg==0.1 from file:///home/ichard26/dev/oss/pip/temp/pip-test-package (setup.py develop) is deprecated. pip 25.0 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457
 Running setup.py develop for version_pkg
Successfully installed version_pkg-0.1

In practice, this usually means one of two things:

The pyproject.toml file doesn’t exist at all, thus pip will opt-out of the PEP 517 interface, running setup.py as required
The declared setuptools version (in the [build-system].requires field) is too old and doesn’t support PEP 660, i.e. anything older than setuptools 64.0.0

For affected projects, the best solution is likely to use a modern version of setuptools and declare this via pyproject.toml. You can place your project metadata in this file as well, but it’s not required to stop the legacy editable deprecation warnings.

# pyproject.toml
[build-system]
requires = ["setuptools >= 64"]
build-backend = "setuptools.build_meta"

Tip: if you have other dependencies needed to run setup.py or otherwise build your package, you should add them to the ["setuptools >= 64"] list. This field is equivalent to setuptools’ setup_requires parameter.

Alternatively, projects can pass the --enable-pep517 flag to force pip to use the PEP 517 interface, including the editable install extensions from PEP 660. If no backend is declared, pip will assume that the project uses setuptools and ensure it’s available in the isolated build environment.³ As long as setuptools 64+ still supports your Python version, a modern editable install will be performed. Once the legacy mechanism is removed, --use-pep517 will have no effect and will essentially be enabled by default.

Finally, if you don’t like either of those solutions, you can always throw away setuptools and switch to a different backend entirely. I’ve mentioned a few earlier, but there are even more options to choose from if you’re willing to do some research and play around. I’d start with with the Python Packaging User Guide’s list of the commonly used backends if you want to replace setuptools.

If you stick with setuptools, one potential snag is that setuptools has introduced a new kind of editable installs while rolling out PEP 660 by default:

If a package uses the “src” layout or otherwise places the source code in a separate top-level directory, setuptools will create a static .pth file to extend the Python import path with that directory
If a package uses the “flat” layout, setuptools will install a custom import hook which intercepts imports and “redirects” them to the original modules (so auxiliary top-level files like noxfile.py or setup.py or a tests package aren’t importable)

If the legacy behaviour is desired, one must pass --config-settings editable_mode=compat. There is also one more method: strict editable installs. They behave closer to a normal installation, but are implemented in an entirely different way from setup.py or the default method described above.⁴

(Update: a setuptools maintainer reached out to strict editable installs are not enabled by default. It turns out that the situation behind potential breakage is a bit more nuanced. Sorry!)

Warning: Static analysis tools including mypy, pyright, and pylint may not function properly when setuptools uses an import hook to implement an editable install. This is a known issue. The recommended workaround is to pass --config-settings editable_mode=compat.

For more details using setuptools with pyproject.toml, you should read their documentation:

⭐ User guide to using pyproject.toml ⭐
⭐ setuptools’ editable install docs ⭐
setuptools’ build system docs

If you’d like to learn more, especially about the history of why and how these standards came to exist, I strongly recommend reading Paul Ganssle’s excellent “Why you shouldn’t invoke setup.py directly” article. It’s long, but packs way more information than I could ever hope to cover.

System HTTPS certificates by default(*)

If you’ve ever used pip in a corporate environment, there is a good chance that you’ve encountered a network or SSL verification error because the remote index server (PyPI or a private index) returned a HTTPS certificate that couldn’t be verified.

pip install toto
Looking in indexes: https://repos.company/api/pypi/python_pypi/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)': /api/pypi/python_pypi/simple/toto/
Could not fetch URL https://repos.company/api/pypi/python_pypi/simple/toto/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host= 'repos.company', port=443): Max retries exceeded with url: /api/pypi/python_pypi/simple/toto/ (Caused by SSLError(SSLError(0, 'unknown error (_ssl.c:3630)'),)) - skipping
ERROR: Could not find a version that satisfies the requirement toto (from versions: none)
ERROR: No matching distribution found for toto

The problem is that while your system may be configured with the required certificate authorities, pip had no ability to use those system CAs. This resulted in a confusing user experience, because while your browser was likely able to access the index server, pip blew up with a non-obvious error.

To be able to verify HTTPS certificates, since the early days, pip has shipped with the certifi CA bundle, which is simply a Python repackaging of the Mozilla CA bundle. To get pip to verify HTTPS certificates not trusted by certifi, you had to pass your own bundle via the --cert option which is clunky.

In pip 22.2, pip gained the ability to use truststore for HTTPS certificate validation, enabling pip to use the system certificate store instead of solely relying on certifi. The integration was experimental; users had to have truststore already installed and opt-in via --use-feature=truststore, but it was a step forward.⁵ The plan was to enable truststore by default once the feature matured.

Two years later, in this pip release, truststore is now enabled by default.⁶ Thus, assuming your system is configured properly with the required CAs, pip should simply “just work” even in many corporate environments. 🎉

There is one major problem though—you saw that asterisk, right? Truststore only works on Python 3.10 or higher. pip will continue to exclusively use its certifi bundle on Python 3.9 or 3.8. Additionally, any Python implementation that does not implement the ssl module APIs needed by truststore will not be able to use the system trust store either. In these situations, you will need to continue using --cert.

Major thanks goes to @sethmlarson for co-authoring truststore and writing patches improving and fixing pip’s truststore integration!

Performance optimizations, duh!

In this release, several optimizations of the environment inspection, file download, dependency resolution, and installation logic landed. While individually they are minor optimizations, together they provide a noticeable performance uplift in a variety of workloads!

Faster discovery of installed packages

First of all, on Python 3.11 or higher, pip is significantly faster to discover installed packages (especially if there are a lot of ’em). This not only makes pip list faster, but also makes pip generally faster as it has to figure out what packages are already installed quite frequently throughout its operation.

These are the before and after runtimes of pip list in my pip development environment which has 75 packages installed:

# before: pip list (24.1)
real 0m0.227s
user 0m0.192s
sys 0m0.035s

# after: pip list (24.2)
real 0m0.207s
user 0m0.170s
sys 0m0.036s

On slower systems, the performance improvement will be greater.⁷ This uplift was achieved by optimizing pip’s importlib based metadata backend to read package⁸ names and versions from the installed metadata directory names.⁹ Previously, pip would read the name and version from the METADATA file, which is slow as it involves an extra file read and invoking an email header parser.

Other optimizations

Faster downloads PR #12810 @morotti: Source distributions and wheels are downloaded in 256 KiB chunks instead of 10 KiB chunks, which significantly reduces overhead and enables faster download speeds. A sizable chunk (🥁) of the performance improvement comes from reducing the number of updates of the download progress bar.
Faster dependency resolution PR #12663 @notatallshaw: … of highly complex or pathological dependency trees. As pip’s dependency resolver explores a dependency tree, it has to parse requirements (e.g. pip==24.2) into a format it understands. In complex trees, the same requirements are often encountered many, many times. There was some caching in-place to minimize redundant work, but now requirements are consistently cached.
Faster installation PR #12803 @morotti: Wheels, which are really zip archives, are extracted using a larger block size (either 1MiB or the file size if it’s smaller than a MiB). Also, the zip decompressor is no longer invoked while copying empty files, eliminating pointless CPU cycles for files like __init__.py.
Faster reinstalls PR #12755 yours truly!: The full list of platform compatibility tags is now only generated once and reused across all look-ups of the wheel cache, speeding up cached reinstalls.

`pip check` just got a bit stricter

The check pip command now verifies whether installed packages are declared to support the current platform, issuing a warning if a package is incompatible.

pip check
catboost 1.1.1 is not supported on this platform
ninja 1.11.1.1 is not supported on this platform
xgboost 1.6.1 is not supported on this platform
frozendict 2.3.8 is not supported on this platform

Under the hood, for every package, pip reads the WHEEL metadata file that’s installed alongside the package, and checks whether at least one of the compatibility tags is supported on the platform.

For example, this is the WHEEL file from pip 24.2’s wheel:

# pip-24.2.dist-info/WHEEL
Wheel-Version: 1.0
Generator: setuptools (71.1.0)
Root-Is-Purelib: true
Tag: py3-none-any

py3-none-any is a platform compatibility tag. What’s a compatibility tag? They’re essentially markers for what systems a package supports. You can think of them like languages. They include information on the architecture, Python ABI, and Python version. Every system has a set of compatibility tags it supports. If a package and the system it’s being installed to share a common tag—or with the analogy, speak a shared language—the package is considered compatible.

In this case, py3-none-any is code for “any Python 3 version”, “any Python ABI”, and “any architecture.” This is supported by any modern Python installation, thus this wheel would be almost always eligible for installation (although it may be rejected for other reasons, like requires-python restrictions).

Now, pip install already uses compatibility tags to ensure it only installs compatible packages. This extra check in pip check is designed to surface packages that were compatible at the time of installation, but are no longer supported due to a system upgrade. This is possible, especially as Apple transitions from the x86-64 architecture (Intel) to arm64 (Apple Silicon).

(Un)fortunately, this has had the side-effect of revealing numerous packages with inaccurate or outright malformed metadata, leading to false positives. This is annoying, but regarded as an intentional side-effect as the pip project has been moving towards enforcing standards compliance so pip follows the standards instead of de facto writing them.¹⁰

This was kindly contributed by @q0w.

Oh yeah, `--require-virtualenv` is a thing

You can configure pip to only function when running in an activated virtual environment via the --require-virtualenv flag. You can set this via a system-side or user-side configuration file; and it’s great for ensuring you don’t scribble all over your system or user Python environment.

pip install package-that-shouldnt-be-installed-system-wide
ERROR: Could not find an activated virtualenv (required).

Anyway, it’s supposed to only apply to commands that modify the environment, like pip install or pip uninstall. Read-only commands, such as pip list, are unaffected by the flag and will function even if a virtual environment is not activated.

This release extends this QoL feature to pip check and pip index. You can now discover unsupported packages whenever you want. Thanks @branchvincent!

Summary

pip 24.2 delivers considerable performance and QoL improvements. It also continues the pip project’s goal of following and enforcing the use of and compliance with the interoperatability standards that have shaped modern Python packaging.

I realize that this post is long, but this release contained some cool changes and I believed they deserved more attention than a brief entry in the changelog. I make no promises that I’ll continue this series for future pip releases, but this sure was fun to write!

Finally, I’d like to thank Bartosz Sokorski, Bradley Reynolds, Hugo van Kemenade, and Paul Moore for reviewing this post in draft form and suggesting improvements. Any mistakes are of course my own.

Poetry is the all encompassing user-facing tool for packaging and dependency management, while poetry-core is its backend. The same nitpick exists for all other “backends” I mention. ↩︎
In reality, pip calls other hooks as well to query additional build dependencies and generate metadata, but this isn’t the post where I explain pip’s entire PEP 517 implementation. ↩︎
--no-build-isolation may be needed if the project has build-time requirements beyond setuptools and wheel. By passing this flag, you are responsible for making sure your environment already has the required dependencies to build your package. ↩︎
As of writing, the TL;DR is that strict editables are implemented as a tree of file links to the original source files in an auxiliary directory which is then added to sys.path, while legacy “compat mode” editables are implemented by placing the entire root of your package on sys.path. The new method is designed to ensure only files present at installation are exposed, mimicking a normal installation. It’s rather neat that setuptools has support for all three methods described for implementing an editable install in PEP 660. ↩︎
This was recognized as a problem in 2014, and pip gained the ability to use system certificates on Unix in 2015, but this change was later reverted because it turned out none of the major distributions had a functional OpenSSL configuration at the time. ↩︎
pip continues to use truststore in tandem with certifi. This is necessary for platforms that do not support truststore, but also provides a fallback if the system trust store is broken or otherwise inaccessible for some reason. There are currently no plans for pip to switch to exclusively use truststore. ↩︎
Also, it’s worth mentioning that 3/4 of the 200ms is taken by Python’s own startup overhead and importing everything pip list needs. ↩︎
OK, so technically I should be saying “distribution” to avoid ambiguity as the term “package” means something different under the Python import system, but distribution sounds weird which is why I’m saying “package” still. I’m horribly inconsistent about this though. ↩︎
If you dig into your site-packages directory, you’ll notice that there is a bunch of <name>-<version>.dist-info directories. These contain package metadata, and there is one for each package installed. They are included in wheels and are copied during installation. ↩︎
Historically, as pip was the only installer in town, various bits of pip’s design or behaviour has been become de facto standards. Even if there is a standard, “pip supports X” or “pip does X” can be regularly seen in the issue trackers of other packaging tools. While this is expected and unavoidable, it’s unideal. We don’t want to be in the business of telling what everyone else should be doing, especially as pip is old and has quirks and design flaws that frankly shouldn’t exist, let alone be copied by other tools. By gradually enforcing standards compliance, the Python packaging ecosystem becomes more and more defined by common specifications, allowing other packaging tools to function without reimplementing a bunch of pip bugs or whatever. ↩︎

What's new in Black 23.1.0

Tue, 31 Jan 2023 00:00:00 -0500

Today we released Black 23.1.0 🎉.

While Black does not follow SemVar, it is indeed a major release as it ships with the 2023 stable style. Many changes made to the preview style over the past year have been finally promoted to the stable style.

You can install it by running this command:

python -m pip install black==23.1.0

Below I’ll cover the important changes worth mentioning. If you’d like the full changelog for 23.1.0, please refer to the link above.

Code style

The 2023 stable style

We didn’t make any major changes to the 2023 stable style draft since 23.1a1.

So for the major parts of the new stable style, please refer to the 23.1a1 post. Everything under the “promoted changes” header were promoted officially in 23.1.0. The TL;DR is:

Improved empty line handling (mostly removing unnecessary ones)
Removal of redundant parentheses in several contexts

We did also fix quite a few bugs and crashes, including these ones reporting by users testing out 23.1a1:

For the rest, please refer to the changelog.

The 2023 preview style

Conditional expressions are parenthesized if needed now

Issue: #2248 ~ PR: #2278

If a conditional expression is too long, it will be wrapped in parentheses instead of whatever nonsense Black did before.

I’ll just let this diff speak for itself :)

--- pyanalyze/typeshed.py 2021-05-30 03:06:13.755715 +0000
+++ pyanalyze/typeshed.py 2021-05-30 03:06:32.534930 +0000
@@ -448,13 +448,13 @@
 arg = arg.replace(kind=SigParameter.POSITIONAL_OR_KEYWORD)
 cleaned_arguments.append(arg)
 return Signature.make(
 cleaned_arguments,
 callable=obj,
- return_annotation=GenericValue(Awaitable, [return_value])
- if is_async_fn
- else return_value,
+ return_annotation=(
+ GenericValue(Awaitable, [return_value]) if is_async_fn else return_value
+ ),
 )

 def _parse_param_list(
 self,
 args: Iterable[ast3.arg],

There are certain cases where it might be overkill, like this one, but overall, I consider it a major improvement.

 normalized = [
- (source, source)
- if source == "-"
- else (normalize_path_maybe_ignore(Path(source), root), source)
+ (
+ (source, source)
+ if source == "-"
+ else (normalize_path_maybe_ignore(Path(source), root), source)
+ )
 for source in src
 ]

Wrap multiple context managers in parentheses if targeting Python 3.9+

Issues: #3486 + #664 ~ PR: #3489

Since Python 3.9¹, you can break long with statements using parentheses. Thanks to work done by @yilei, Black will now use this style as long as the lowest targeted version is 3.9.

Source:

with make_context_manager1() as cm1, make_context_manager2() as cm2, make_context_manager3() as cm3, make_context_manager4() as cm4:
 pass

Before: unchanged!!

After:

with (
 make_context_manager1() as cm1,
 make_context_manager2() as cm2,
 make_context_manager3() as cm3,
 make_context_manager4() as cm4,
):
 pass

Black still doesn’t have a good story for formatting such with statements when Python <3.9 support is requested. All of the gory details can be found in issue #664. (The summary is that no one has stepped up to implement the proposed style.)

Packaging

mypyc wheels are available for all platforms (once again)

Long story short, recent versions of packaging and hatchling interacted with ways that led to errors when trying to build macOS mypyc wheels. The essence is that the wrong platform tag was being chosen which made the wheels uninstallable in certain situations (including on the same machine that built wheels >.<).

Anyway that’s all fixed now and 23.1.0 ships with macOS wheels, bringing us back to the full set of compiled wheels.

This affected the 22.10.0, 22.12.0, and 23.1a1 releases.

mypycified Black can be now built on armv7

We upgraded mypy/c from 0.971 which 0.991 which allows mypycified Black to be built on armv7.

Conveniently, this also fixes some segfaults caused by mypyc:

Segfault with black 22.1.0 (#2845)
black running on address sanitized Python interpreter crashes (#2867)
- same underlying issue as the one above

Output

No longer incorrectly claim that symbolic links exist when formatting from outside a project

Issue: #3384 ~ PR: #3385

When trying to format a project from the outside, the verbose output shows says that there are symbolic links that points outside of the project, but displays the wrong project path. This behavior was triggered when Black is executed from outside the project’s root.

Consider the following tree:

.
└── home/
└── project/
├── .git
└── dir/
└── main.py

When trying to format a folder from home, this is the output:

$ black ./project --check --verbose
Identified `/path/to/home/project` as project root containing a .git directory.
Sources to be formatted: "."
project/.git ignored: is a symbolic link that points outside /path/to/home/project/project
project/.git ignored: matches the --exclude regular expression
project/dir ignored: is a symbolic link that points outside /path/to/home/project/project
project/dir/file.py ignored: is a symbolic link that points outside /path/to/home/project/project
would reformat project/dir/file.py

Oh no! 💥 💔 💥
1 file would be reformatted.

Notice that the message $FILEPATH ignored: is a symbolic link that points outside $PROJECTPATH displays a wrong $PROJECTPATH (should be /path/to/home/project/ instead of /path/to/home/project/project).

Anyway, Black will no longer emit these superfluous and incorrect messages. All thanks goes to @aaossa.

Verbose output now shows the full loaded configuration

Issue: #3386 ~ PR: #3392

It’s easier to just copy and paste part of the issue to explain this feature:

Sometimes when dealing with multiple environment settings, IDEs, and pyproject.toml files etc. etc. is beneficial for a project team to have a confirmation about the actual applied black-settings (“line length”) in the CI CD pipeline.

black -v will show a line like this: Using configuration from project root.. However, it doesn’t tell us what the settings are […]

… so we changed Black to show the loaded configuration if -v / --verbose is passed.

$ black src/black/parsing.py --verbose
Identified `/home/ichard26/programming/oss/black` as project root containing a .git directory.
Sources to be formatted: "src/black/parsing.py"
Using configuration from project root.
line_length: 88
target_version: ['py37', 'py38']
include: \.pyi?$
extend_exclude: /(
 # The following are specific to Black, you probably don't want those.
 | blib2to3
 | tests/data
 | profiling
)/

preview: True
src/black/parsing.py already well formatted, good job.

All done! ✨ 🍰 ✨
1 file left unchanged.

Some cleaning up can definitely be done to the verbose output, but at least it’s there now.

Configuration

Better default for –target-version if PEP 621 python-requires is available

Issue: #3124 ~ PR: #3219

Black will now infer a default set of target versions using the project.requires-python field in pyproject.toml if possible.

For example, these are the inferred default for --target-version for three different requires-python values:

3.8.5 becomes ["py38"]
>3.6,<3.11 becomes ["py37", "py38", "py39", "py310"]
<3.7 becomes ["py33", "py34", "py35", "py36"]
- Python 3.3 is the minimal supported version for Black

If you’re curious what Black infers for your project, make sure to comment out any --target-version configuration you have set and run Black with --verbose. Thanks to the change mentioned previously, there should be a line showing the (inferred) target-version config value.

If --target-version is explicitly configured on the command line or in pyproject.toml, it will trump the inferred configuration. And if Black encounters any error inferring a better default, it will simply fallback to per-file autodetection (like it did before).

To make this work, Black now requires packaging (version 22 and up). It’s a tiny dependency and often installed with other development tools so it shouldn’t be a big deal.

In my opinion, the general hope is that the vast majority of projects will eventually use Black’s inference capabilities instead of setting --target-version separately. It’s less work (no need to update it!) and should be less error prone.

At the bare minimum, new projects shouldn’t have to configure --target-version anymore as long as they define their project metadata in the standardized way.

This was contributed by the lovely @stinodego.

A few final words

We tried to do some community outreach in an attempt to iron out issues before cutting 23.1.0 and make sure the 2023 stable style wouldn’t be a failure. We got some helpful feedback, but I realize we could’ve done more.

If you have any feedback for this release or suggestions for next time, please let us know! Feel free to open an issue, email me, message us on Discord, whatever.

Anyway, thanks for reading this. On behalf of the maintainer team, I hope you enjoy the new release!

Well officially speaking, this is only supported starting in Python 3.10 and higher, but GvR and co. might have snuck it into 3.9 :P ↩︎

Black 23.1a1 - please help us test the 2023 stable style!

Sun, 18 Dec 2022 00:00:00 -0500

As the post summary says, earlier today we released Black 23.1a1. You might be wondering why we released an alpha… it’s because the upcoming Black 23.1.0 release will see many preview code style changes promoted to the stable style.

In other words, Black 23.1.0 will format your code differently out of the box.

If you’re surprised we’re doing this, our stability policy allows us to update the stable style in the new year. This minimises disruption while keeping the door for improvements open.

23.1a1 contains a draft of the 2023 stable style that we, the maintainers, initially decided on.

However, we need your help in finalizing what goes into the 2023 stable style. If you can, please try 23.1a1 out on your codebase(s). You can install it by running this command:

python -m pip install black==23.1a1

If you have any feedback, concerns, or run into any issues, please drop us a comment in this issue (https://github.com/psf/black/issues/3407).

Note: Please read the start of the issue description before commenting. It will make the lives of everyone easier.

What follows is brief rundown of all of the changes in the preview style that were promoted to stable in 23.1a1.

Promoted changes

Bugfixes

`--skip-string-normalization` now prevents docstring prefixes from being normalized

-S / --skip-string-normalization stops Black from normalizing string prefixes and quotes, except when it doesn’t…

Sometime ago I accidentally added a bug where Black would normalize docstring prefixes even if you told it not to, leading to this:

 def add(a, b):
- F"""Add two numbers and return the result."""
+ f"""Add two numbers and return the result."""
 return a + b

This has been fixed in the preview style since 22.8.0 (leaving this example unchanged).

`--skip-magic-trailing-comma` now removes useless trailing commas in subscripts

Black has the concept of “magic trailing commas.” You can read about it here, but in essence, Black treats trailing commas as a sign that collections, calls, or function signatures should always be exploded.

# The trailing comma will stop Black from collapsing this collection into one line.
TRANSLATIONS = {
 "en_us": "English (US)",
 "pl_pl": "polski",
}

This can be disabled passing the -C / --skip-magic-trailing-comma flag.¹

However, Black didn’t remove unnecessary trailing commas in all situations. In the following example, the trailing comma in the function signature was removed as expected, but not the subscript.

 Point = Tuple[int, int,]

-def f(a: Point, b: Point,): ...
+def f(a: Point, b: Point): ...

This was fixed in the preview style since 22.8.0, which removes both commas as expected.

-Point = Tuple[int, int,]
+Point = Tuple[int, int]

-def f(a: Point, b: Point,): ...
+def f(a: Point, b: Point): ...

Correctly handle trailing commas that are inside a line’s leading non-nested parens

There are actually two problems in the “before” output:

The zero(one,).two(three,).four(five,) chain should be fully exploded
The magic trailing comma in the dictionary in refresh_token() is being ignored

Source:

zero(one,).two(three,).four(five,)


def refresh_token(self, device_family, refresh_token, api_key):
 return self.get(
 data={
 "refreshToken": refresh_token,
 },
 api_key=api_key,
 )["sdk"]["token"]

Before:

zero(one,).two(three,).four(
 five,
)


def refresh_token(self, device_family, refresh_token, api_key):
 return self.get(data={"refreshToken": refresh_token,}, api_key=api_key,)[
 "sdk"
 ]["token"]

After:

zero(
 one,
).two(
 three,
).four(
 five,
)


def refresh_token(self, device_family, refresh_token, api_key):
 return self.get(
 data={
 "refreshToken": refresh_token,
 },
 api_key=api_key,
 )[
 "sdk"
 ]["token"]

Both these issues have been fixed in the preview style since 22.12.0.

End quotes that violate the line length limit in multiline docstrings will be moved

1
2
3
4
5
6
7
8
9


def process_and_count(text):
 """
 Remove punctuation and return cleaned string, in addition to its length in tokens."""
 pass


def example_2(text):
 """Remove punctuation and return cleaned string in addition to its length in tokens."""
 pass

Here, the third line violates the line length limit (90 > 88) because of the end quotes. Earlier versions of Black would leave this be which wasn’t ideal, so now they are moved onto their own line.

 def process_and_count(text):
 """
- Remove punctuation and return cleaned string, in addition to its length in tokens."""
+ Remove punctuation and return cleaned string, in addition to its length in tokens.
+ """
 pass

However, Black won’t move quotes of single line docstrings (such as with example_2()) since that would look ugly.

This was added to the preview style in 22.8.0 and then tweaked in 23.1a1.

Better management of parentheses

Here’s a set of changes that together improve Black’s handling of parentheses in various situations.

Out of these five, “Remove redundant (outermost) parentheses in for statements” will probably be the most impactful.

Return type annotations

Source:

def foo() -> (int):
 return 2


def foo() -> intsdfsafafafdfdsasdfsfsdfasdfafdsafdfdsfasdskdsdsfdsafdsafsdfdasfffsfdsfdsafafhdskfhdsfjdslkfdlfsdkjhsdfjkdshfkljds:
 return 2


def frobnicate() -> ThisIsTrulyUnreasonablyExtremelyLongClassName | list[ThisIsTrulyUnreasonablyExtremelyLongClassName]:
 pass

Before:

def foo() -> (int):
 return 2


def foo() -> intsdfsafafafdfdsasdfsfsdfasdfafdsafdfdsfasdskdsdsfdsafdsafsdfdasfffsfdsfdsafafhdskfhdsfjdslkfdlfsdkjhsdfjkdshfkljds:
 return 2


def frobnicate() -> ThisIsTrulyUnreasonablyExtremelyLongClassName | list[
 ThisIsTrulyUnreasonablyExtremelyLongClassName
]:
 pass

After (22.6.0+):

def foo() -> int:
 return 2


def foo() -> (
 intsdfsafafafdfdsasdfsfsdfasdfafdsafdfdsfasdskdsdsfdsafdsafsdfdasfffsfdsfdsafafhdskfhdsfjdslkfdlfsdkjhsdfjkdshfkljds
):
 return 2


def frobnicate() -> (
 ThisIsTrulyUnreasonablyExtremelyLongClassName
 | list[ThisIsTrulyUnreasonablyExtremelyLongClassName]
):
 pass

Remove redundant parentheses around awaited objects

Source:

async def main():
 await (asyncio.sleep(1))
 await (set_of_tasks | other_set)

Before: unchanged.

After (22.6.0+):

async def main():
 await asyncio.sleep(1)
 await (set_of_tasks | other_set)

Remove redundant parentheses in with statements

Source:

with (open("bla.txt")):
 pass

with (open("bla.txt")), (open("bla.txt")):
 pass

with (open("bla.txt") as f):
 pass

with (open("bla.txt")) as f:
 pass

Before: unchanged.

After (22.6.0+):

with open("bla.txt"):
 pass

with open("bla.txt"), open("bla.txt"):
 pass

with open("bla.txt") as f:
 pass

with open("bla.txt") as f:
 pass

Remove redundant parentheses from except clauses

Source:

try:
 a.something
except (AttributeError) as err:
 raise err

Before: unchanged.

After (22.3.0+):

try:
 a.something
except AttributeError as err:
 raise err

Remove redundant (outermost) parentheses in for statements

Source:

for (k, v) in d.items():
 print(k, v)

Before: unchanged.

After (22.3.0+):

for k, v in d.items():
 print(k, v)

Unfortunately, nested parentheses are still left untouched. I have an open PR to fix that, but it isn’t ready yet.

Remove blank lines after code block open

Here’s another change that is likely to impact your codebase.

Note: This new feature will be applied to all code blocks: def, class, if, for , while, with, case and match.

Source:

def foo():



 print("All the newlines above me should be deleted!")


if condition:

 print("No newline above me!")

 print("There is a newline above me, and that's OK!")


class Point:

 x: int
 y: int

 def as_tuple(self) -> tuple[int, int]:
 return (self.x, self.y)

Before:

def foo():

 print("All the newlines above me should be deleted!")


if condition:

 print("No newline above me!")

 print("There is a newline above me, and that's OK!")


class Point:

 x: int
 y: int

 def as_tuple(self) -> tuple[int, int]:
 return (self.x, self.y)

After (22.6.0+):

def foo():
 print("All the newlines above me should be deleted!")


if condition:
 print("No newline above me!")

 print("There is a newline above me, and that's OK!")


class Point:
 x: int
 y: int

 def as_tuple(self) -> tuple[int, int]:
 return (self.x, self.y)

Ignore magic trailing comma for single-element subscripts

Black exempts single-element tuple literals from the usual handling of magic trailing commas: (1,) will not have a newline added (as would happen for lists, sets, etc.).

However, if you wrote tuple[int,] (to make it more visually distinctive to list[int]), Black would explode it, which doesn’t look great.

Source:

shape: tuple[int,]
shape = (1,)

Before:

shape: tuple[
 int,
]
shape = (1,)

After (22.3.0+): unchanged.

Enforce empty lines before classes/functions with sticky leading comments

The old (“before”) behaviour here caused flake8 to raise E302.

Source:

some_var = 1
# Leading sticky comment
def some_func():
 ...


def get_search_results(self, request, queryset, search_term):
 """
 Return a tuple containing a queryset to implement the search
 and a boolean indicating if the results may contain duplicates.
 """
 # Apply keyword searches.
 def construct_search(field_name):
 ...

 ...

Before: unchanged.

After (22.12.0+):

some_var = 1


# Leading sticky comment
def some_func():
 ...


def get_search_results(self, request, queryset, search_term):
 """
 Return a tuple containing a queryset to implement the search
 and a boolean indicating if the results may contain duplicates.
 """

 # Apply keyword searches.
 def construct_search(field_name):
 ...

 ...

Empty and whitespace-only files are now normalized

Currently under the stable style, empty and whitespace-only files are not modified. In some discussions (issues and pull requests), the consesus was to reformat whitespace-only files to empty or single-character files, preserving the original line endings (LR/CRLR) when possible.

In the end, we settled on these rules:

Empty files are left as is
Whitespace-only files (no newline) reformat into empty files
Whitespace-only files (1 or more newlines) reformat into a single newline character

This was added to the preview style in 22.12.0.

Spyder cell separator comments (`#%%`) are now normalized

Black adds a space after the pound character in comments as per PEP 8. This caused trouble as many editors have special comments for various functionality (eg. denoting regions in the file) which could not contain any spaces.

To play with other tools, Black leaves Spyder’s special #%% comments (among other special comments) alone.

Sometime later, Spyder started to recognize # %% alongside #%%. So in Black 22.3.0, the preview style was changed to start adding spaces in #%% comments.

What is not being promoted?

Ideally I would’ve provided examples for these changes too, but I need to get this post out quickly and I’m getting tired. I’ve added links for further reading if you’re curious.

Experimental string processing (aka ESP) is not being promoted to the 2023 stable preview. There’s a lot of reasons why, but Jelle summarized it well in this comment:

[…] At this point there’s still half a dozen stability bugs with experimental string processing, so that alone is enough to block it from going into the 2023 stable style. However, I think we should consider dropping (most of) the feature entirely, instead of leaving it behind a flag forever, since there are so many stability bugs and often it arguably makes for worse output. That should be discussed in #2188, though.

And these three depend on ESP being promoted:

Additionally, two other changes are currently not slated for promotion and will remain in the preview style until 2024, mostly since they were landed too late in the year to allow for enough testing. These are:

For assignment statements, prefer splitting the right hand side if the left hand side fits on a single line (#3368) ~ available in 22.12.0+
Improve long values in dict literals (#3440) ~ available on main and in 23.1a1 only

“I want a new change to the stable style”

If you want a change that’s not covered by something in the preview style already (and can’t be achieved by simply tweaking a pre-existing style change), check the issue tracker for a issue that covers what you want. If you don’t find one, file an issue here or drop a comment here.

Although, if you’re hoping to squeeze in a new major style change into the 2023 stable style, that probably won’t be possible being so late in the year. It can always be promoted to stable for 2024.

There’s some interesting history to why we added this flag, but I don’t have time to get into it right now. ↩︎

What's new in Black 22.10.0

Fri, 14 Oct 2022 00:00:00 -0400

On October 6th, 2022, we released Black 22.10.0. It’s a smaller release by the size of the changelog, but hey that means I don’t have to write as much!

This is meant to be an accompanying blog post for 22.10.0. I won’t discuss every change, just the ones worth highlighting. Instead, I’ll go into more detail, providing context and explanations (and probably some behind of scenes stuff too – this isn’t official anyway). I know this is a bit late, but I hope this is still insightful.

If you’d like the full changelog for 22.10.0, please refer to the link above.

Packaging

Goodbye Python 3.6 (and hello Hatchling!)

It’s official: you can no longer run Black on Python 3.6 starting with 22.10.0.¹

This is actually the first time Black has ever dropped runtime support since 3.6+ was required from the start.²

We dropped Python 3.6 because we wanted to switch build backends from Setuptools to Hatchling which only supports 3.7+. Setuptools has served us well as a simple and reliable backend, but we wanted to modernize. While Setuptools does finally now support most packaging standards (PEP 517, PEP 621, PEP 660 and more) our needs are a bit unique.

We use mypyc to compile Black for a significant performance boost. Before Hatchling we hand-rolled our own integration using mypyc’s basic Setuptools build API. It worked well enough, but it meant we would be stuck with setup.py forever until a Setuptools plugin for mypyc is available. (one does not exist as of writing)

All of the reasons behind the switch can be found in the PR, but in essence, Hatchling allowed us to move to static metadata in pyproject.toml while providing us with a nicer mypyc integration thanks to its hatch-mypyc plugin.

Switching to Hatchling hasn’t been as simple as I would’ve wished: we had to track down this confusing linker error that only appears when using build isolation (among other situations) and now our macOS builds are producing mislabelled wheels, ugh. It happens.

Compiled wheels for 3.11

We couldn’t build mypyc wheels for CPython 3.11 since we had been using an older manylinux docker image to avoid the linker error mentioned earlier. It was old enough that 3.11 wasn’t pre-installed!

Once the linker error was addressed by @zsol (thank you, I had no chance of figuring it out), it was pretty straightforward to configure cibuildwheel to build CPython 3.11 wheels. Just a CIBW upgrade, a type error fix, and a workaround so that aiohttp doesn’t fail to install on 3.11.

Code style

Failing to put `fmt: on` at the same indent level as `fmt: off` no longer crashes

Previously Black produced invalid code when # fmt: on is placed on a different indent level than the # fmt: off it’s paired with. Black failed similarly when # fmt: on just didn’t exist.

def x():
 # fmt: off
 return

def y():
 return

$ black issue-569.py
error: cannot format issue-569.py: INTERNAL ERROR: Black produced code that is not equivalent to the source. Please report a bug on https://github.com/psf/black/issues. This diff might be helpful: /tmp/blk_wv8p1p5k.log

Oh no! 💥 💔 💥
1 file failed to reformat.

If you disable the safety checks with --fast, you’ll quickly realize that Black badly messed up the indentation.

def x():
 # fmt: off
 return

 def y():
 return

This annoying bug was often hit. If you look at the original issue’s timeline, you’ll notice that this bug has been reported many, many times over. Yet, this bug was left unfixed for almost three years until @yilei opened GH-3281 to finally fix it.

Black still requires # fmt: off and # fmt: on to be used at the same indent level, but now, the code at or above the initial # fmt: off’s block level will be left untouched, when # fmt: on is used on a different level or there is no # fmt: on.

In other words, you can now use a single # fmt: off to turn off formatting by block level.

def ignore_this_function(condition):
 # fmt: off
 if condition:
 return 'hello, world'


def format_this_function():
 def except_this_block_level():
 # fmt: off
 return 'hello, world'

 return except_this_block_level ()

$ black file.py --diff --color
--- file.py 2022-10-15 18:30:53.887339 +0000
+++ file.py 2022-10-15 18:30:54.780816 +0000
@@ -7,6 +7,6 @@
 def format_this_function():
 def except_this_block_level():
 # fmt: off
 return 'hello, world'

- return except_this_block_level ()
+ return except_this_block_level()

Anyway, I’m so happy this major bug has finally been fixed. Thank you so much @yilei!

Preview style: fixed a crash related to string keys in dictionaries

Parentheses are added around implicitly concatenated strings in various situations under the preview style since 22.8.0. Unfortunately, this introduced a bug where long string keys would be wrapped in extra parentheses along with the value.

v = {
 ('a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_'
 'a_very_long_str_a_very_long_str_a_very_long_str_'): {
 'key':
 'value',
 },
}

 v = {
- ('a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_'
- 'a_very_long_str_a_very_long_str_a_very_long_str_'): {
- 'key':
- 'value',
- },
+ (
+ "a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_a_very_long_str_": {
+ "key": "value",
+ }
+ ),
 }

Now Black no longer adds these invalid parentheses, although it just merges the strings together violating the line length which doesn’t seem right…

GitHub Action improvements

The GitHub Action can now format Jupyter Notebooks if you set jupyter: true:

- uses: psf/black@stable
 with:
 src: "./src"
 jupyter: true
 version: "22.10.0"

Under the hood, it will install Black with the jupyter extra enabling Black’s optional support for .ipynb files.

Additionally, the version key now supports PEP 440 version specifiers. This is a great QOL improvement because if you want to match versions covered by Black’s stability policy, you can use the compatible release operator (~=) to do that now:

- uses: psf/black@stable
 with:
 src: "./src"
 version: "~= 22.0"

By restricting to 2022 releases, the latest version possible will be used without sudden changes (breaking your CI) since releases within the same year are guaranteed to not change the stable style.

This was a backwards-compatible change, you can still specify a single version if you’d like.

`-x` / `--skip-source-first-line`

Python has a flag (-x) to skip the first line of files given. I didn’t know this existed, but it turns out it’s commonly used with Python WASM pages. They shove the HTML code in a single line at the start as a non-standard “shebang” (since browsers generally require HTML files to start with <) so it can run in the browser and locally.

For example, this browser pong game uses pygame via pygbag (a WASM version of pygame). The start of the HTML file looks like this:

<html><head><meta charset="utf-8"></head><script src="https://pygame-web.github.io/archives/0.3/pythons.js" type=module id="__main__" data-src="gui" async defer>#<!--
import sys
import os
import asyncio

Since all the HTML code is at the top in a single line, you can run it as any other Python script by simply passing -x.

wget https://pygame-web.github.io/showroom/pygame-scripts/org.pygame.touchpong.html
# assuming pygame is installed
python -x org.pygame.touchpong.html

I was hesitant to add a similar option to Black, but it was a convincing use-case and the issue got a surprising amount of upvotes.

This option is also available for Blackd.

… and that’s all. Thanks for reading! ❀

We weren’t planning to remove 3.6 support in 22.10.0. We believed that Python 3.6 was too popular to stop supporting. The earliest agreed-upon removal date was 2023. But after even more internal discussion, we eventually decided to drop 3.6 earlier. ↩︎
Łukasz couldn’t live without his f-strings :) ↩︎

Compiling Black with mypyc, Pt. 3 - Deployment

Tue, 31 May 2022 16:27:00 -0400

This is part of the “Compiling Black with mypyc” series.

Pt. 1 - Initial Steps
Pt. 2 - Optimization
Pt. 3 - Deployment (you’re reading this one)

Building compiled wheels with GitHub Actions

With the mypyc branch functional and pretty fast, it was time to automate building the wheels. Not only does this make the release process easier, but it also means I can easily build wheels for platforms I don’t have access to, like MacOS.

To make the CI configuration process easier, I finally took a look at cibuildwheel. I won’t go into my exact process since it was basically trial and error + my many dumb mistakes 😅 ¹ but here are some noteworthy takeaways:

If you’re using setuptools-scm, you might have to do a full clone on CI so the tag history is still available by build time
Read cibuildwheel’s documentation before writing any configuration! Seriously, if I did I wouldn’t have to make this commit adding {project} to the test command
Set CIBW_BUILD_VERBOSITY to at least 1 because it will make debugging build errors (and trust me you will get some!) so much nicer
If wheel sizes are an issue then passing debug_level=0 to mypyc.build.mypycify should help by stripping all debug information. Not great if you hit a bunch of segfaults or similar though, so it’s a tradeoff

So yeah, 20-ish commits later I had a basic but functional setup which could compile wheels for x86-64 Windows, MacOS, and Linux from CPython 3.6 to CPython 3.10. Universial2 and ARM variants are also supported for the shiny M1 platform. The workflow is a bit slow, but that’s expected and not a big deal.

If you’re curious, you can find the workflow here: ichard26/black-mypyc-wheels

I don’t recommend using my workflow to build your own wheels since it’s a bit hacky and in need of a cleanup. I’d instead recommend taking a look at the examples in cibuildwheel’s docs.

I was going to mention the community field testing campaign I carried out that had its own package index, but honestly it was for the most part a failure as it reached no one. In hindsight, I did not promote it enough so yeah this one is on me :)

Stable release prep and shipping the wheels

Black for the longest time ever wasn’t stable (see GH-517). The team initially aimed to stabilize Black in late 2018, but that didn’t happen for reasons. I won’t go into it too much since it’d be a whole other story, but suffice to say when we made our newest “commitment” (at that point we explicitly worded our intentions to not be promises) we were very motivated to get it done and hopefully right.

We drafted a stability policy, dropped Python 2 support, introduced the --preview flag, and so much more ².

In this time I rewrote diff-shades into what it is today, a reliable enough tool used to provide direct feedback on PRs. I tried to integrate it into the wheel build workflow as part of the test step, but that turned out be very painful and I backed out of it since I had a stable release to manage and publish!

The TL;DR of it is as follows:

Building Black with mypyc usually requires disabling pip’s build isolation to work properly as mypyc is not a standard build dependency of Black. This messes up the installation of diff-shades which is packaged using flit. So I hacked up a script to edit the [build-system].requires field in pyproject.toml to include mypyc pre-build, but the isolation somehow broke the linker search path or something.

clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g0 -fPIC -I/tmp/pip-build-env-9bfrmy6j/overlay/lib/python3.7/site-packages/mypyc/lib-rt -Ibuild -I/opt/python/cp37-cp37m/include/python3.7m -c build/__native_f2d4935fd652bc9ef29d.c -o build/temp.linux-x86_64-3.7/build/__native_f2d4935fd652bc9ef29d.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option
 clang -shared -g0 build/temp.linux-x86_64-3.7/build/__native_f2d4935fd652bc9ef29d.o -o build/lib.linux-x86_64-3.7/f2d4935fd652bc9ef29d__mypyc.cpython-37m-x86_64-linux-gnu.so
 /usr/bin/ld: cannot find crtbeginS.o: No such file or directory
 /usr/bin/ld: cannot find -lgcc
 /usr/bin/ld: cannot find -lgcc_s
 clang: error: linker command failed with exit code 1 (use -v to see invocation)
 error: command '/usr/bin/clang' failed with exit code 1

I honestly still have no idea what’s wrong.

Anyway since I was like an hour or more into build errors, I just decided to stop and fall back to the basic testing that was already working.³ I triggered the last workflow run of the day, downloaded the artifacts, tested one of them locally to make sure nothing was on fire, and pushed ’em to PyPI and that’s how release 22.1.0 was born 🎉.

The core team chat was quite lively for the next hour, to say the least; after all, we just finished a major milestone! Also yeah, we may have done a lot of publicizing for this release which is why it was everywhere for a while :wink:

Post release calm

To be honest I haven’t really seen anyone comment on the fact Black is now compiled with mypyc yet short of this one mini blog post⁴ … which is both disappointing since I spent so much time on this effort, but also calming since usually buzz is sparked by bugs and whatnot.

Having mentioned bugs and crashes, so far we’ve received two reports, one bug probably from mypyc and another due to an unintentional restriction of Black’s unofficial APIs.

Version 22.1.0 still unintentionally broke quite a few integrations, though they weren’t mypyc related. Turns out lots of people depend on black.files.find_project_root returning a pathlib.Path and nothing else! I’ve seen at least five issues / PRs on GitHub fixing crashes related to our change making it return a tuple instead.

For this reason we, the core team, want to define a stable API (GH-779) soon. I can’t promise anything as Black’s development is volunteer-based, but if you were curious to what’s next for psf/black, there’s this.

Results & final thoughts

Black is now overall 2 times faster⁵. And as a bonus, startup time is down too (at most by 15%). You can read the whole report here. Please note these numbers were gathered quite a long time ago and they are probably a bit outdated, especially with the somewhat recent blib2to3 changes made to support 3.10 syntax. If you’d like to see what all of this work looked in PR form, here ya go.

From my experience, I believe mypyc will make for an interesting, but viable solution to speeding up Python code going forward. It’s far from perfect and still has a bunch of bugs, but today it is already making an impact. I don’t recommend using mypyc unless you are willing to feel a bit like a beta tester and have an exhaustive test suite, but hey I managed to do all of this without knowing much about programming in C⁶, which says a lot.

Going forward these open issues remain:

pre-commit is still slow: since pre-commit downloads hooks from source, the compiled wheels won’t be used. This is a shame since Black’s speed is noticeable during git commits. I suspect we will have to introduce a shim like mypy’s pre-commit/mirrors-mypy, but that will be a painful transition.
Wheel building is slow: this is because how mypyc’s setuptools integration works with PEP 517 builds. The source code ends up being parsed, type checked, and transcompiled twice during a build. pypa/hatch might be a good way to fix this assuming its mypyc plugin doesn’t suffer the same flaw.
We’re stuck on mypyc 0.920: a change to newer versions broke our usage of dataclasses and it still hasn’t been fixed …

I’m optimistic that it’s only a matter of time before all of these issues are resolved.

I hope to see the mypyc project gain more contributors and succeed. Along with Cython, PyPy, and the faster-cpython project, perhaps Python will finally have a solid speed story.

And yes, I did indeed learn a ton embarking on this project. Made mistakes along the way, but they turned out alright ❀

Congrats on reaching the end of this multi-part blog series!

By the way, Glyph has also written an excellent piece on mypyc where he tries to convince you to start using mypyc. You should totally read it.

If you would like to chat with me, my contact details are here.

Acknowledgements

I’d like to thank @msullivan for his original work on integrating mypyc into Black, spawning this summer project and eventually this blog series :)

I’d like to also thank @dawnofmidnight, @kosayoda, Jelle Zijlstra, and Łukasz Langa for their extensive feedback and help while writing this series. Any errors are my own, not theirs.

If you are in need of some quality entertainment, look at this commit history: https://github.com/ichard26/black-mypyc-wheels/commits/main ↩︎
I would link to the 22.1.0 heading, but currently the HTML IDs are uhhh … terrible and don’t persist. I’ve tried fixing this with a custom sphinx extension but it failed. I have other ideas left to try, but yeah :( ↩︎
I was frustrated and had a headache thanks to the stress 🙃 ↩︎
In the time it has taken me to edit this series, Łukasz Langa has done a talk about mypyc and Black’s usage of it. ↩︎
Originally when I first landed the relevant PR it was an overall 2x improvement, but once Jelle added a stability hotfix the effective speedup for files that are changed is 50%. If you’re formatting a bunch of already well formatted files, the speedup is still 2x ↩︎
Learning C (and Rust) is now one of my future goals thanks to this project ↩︎

Compiling Black with mypyc, Pt. 2 - Optimization

Tue, 31 May 2022 16:26:00 -0400

This is part of the “Compiling Black with mypyc” series.

Pt. 1 - Initial Steps
Pt. 2 - Optimization (you’re reading this one)
Pt. 3 - Deployment

Optimizing for mypyc

Having compiled Black successfully without it blowing up, it was time to try to optimize Black for mypyc. While mypyc is designed to handle all sorts of statically typed code, changing up the code even a little bit can allow mypyc to perform additional optimizations, ultimately helping performance a bunch.

And while I did go into this hoping I could spot some potential architectural or data structure optimizations, I wasn’t able to so the optimizations shared will be mypyc specific.

Getting started

First off, profiling! It’s hard to optimize code well if you don’t know where time is being spent.

To start off, I profiled Black over some of its own source code with cProfile to get a birds eye view:

$ python -m cProfile -o profile.pstats -m black src/black/__init__.py --check
All done! ✨ 🍰 ✨
1 file would be left unchanged.

$ gprof2dot profile.pstats | dot -Tsvg -o profile.svg

With the help of gprof2dot (well, actually the yelp-gprof2dot fork), I converted the profiling data into a nice SVG graph which I could then open in my web browser.

Tip: please open this massive SVG in a new tab because viewing it here will probably be painful :)

I tried using Scalene as I’ve heard good things about it, but it didn’t work sadly. It wasn’t that bad as I still had py-spy, line_profiler, and good ol’ cProfile. py-spy in particular was invaluable since it can profile (well err sample) C extensions which cProfile cannot. line_profiler was used exclusively for micro-optimizations :p

Anyway, I repeated this process quite a few times, making sure to try different files to get a general feel where time is going regardless of the input. Here are the main takeaways:

Initial parsing with blib2to3 takes up 30-50% of formatting runtime!
The AST equivalent safety check is cheap at 5-10% (obviously only when changes were made, otherwise it’s zero 🙂)
The actual formatting logic’s runtime is mostly spent in the CST visitor usually taking up 75%. The rest went to the transformers which handle line breaks, string literals, and some special cases.

I was quite surprised how much time blib2to3 related functions were eating up. Just look how concentrated the hotspots really are!

These hotspots make this part of the codebase way easier to optimize, so I optimized blib2to3 first.

It’s been so long since I first looked into this, so I don’t remember what optimizations I tried initially, but I do remember them having no effect :(

I tried other things and they actually helped 🎉 … probably since I took into account how mypyc works. Ultimately, many different optimizations were done over three rounds.

Tightening up existing type annotations

The stricter the type annotations are in your codebase, the more invariants mypyc will be able to infer. It’ll then use this information to write type-specific code that is faster. This code won’t work if it gets an object of a different type, but that’s why we use and enforce type annotations, so mypyc can safely assume it’s going to be the right type. In essence, the stricter the type annotations, the faster code mypyc can generate.

This meant reading through the code’s control flow, checking whether certain states are impossible. Blib2to3 is a legacy codebase, so type annotations were added to unblock other work. The goal was to make blib2to3 type check, and not write perfectly typed code. So naturally, there were a few permissive (parameter) type annotations I could make stricter:

diff --git a/src/blib2to3/pgen2/parse.py b/src/blib2to3/pgen2/parse.py
index 47c8f02..6b03188 100644
--- a/src/blib2to3/pgen2/parse.py
+++ b/src/blib2to3/pgen2/parse.py
@@ -138,7 +138,7 @@ class Parser(object):
 self.rootnode: Optional[NL] = None
 self.used_names: Set[str] = set()

- def addtoken(self, type: int, value: Optional[Text], context: Context) -> bool:
+ def addtoken(self, type: int, value: Text, context: Context) -> bool:
 """Add a token; return True iff this is the end of the program."""
 # Map from token to label
 ilabel = self.classify(type, value, context)
@@ -185,11 +185,10 @@ class Parser(object):
 # No success finding a transition
 raise ParseError("bad input", type, value, context)

- def classify(self, type: int, value: Optional[Text], context: Context) -> int:
+ def classify(self, type: int, value: Text, context: Context) -> int:
 """Turn a token into a label. (Internal)"""
 if type == token.NAME:
 # Keep a listing of all used names
- assert value is not None
 self.used_names.add(value)
 # Check for reserved words
 ilabel = self.grammar.keywords.get(value)
@@ -201,12 +200,10 @@ class Parser(object):
 return ilabel

 def shift(
- self, type: int, value: Optional[Text], newstate: int, context: Context
+ self, type: int, value: Text, newstate: int, context: Context
 ) -> None:
 """Shift a token. (Internal)"""
 dfa, state, node = self.stack[-1]
- assert value is not None
- assert context is not None
 rawnode: RawNode = (type, value, context, None)
 newnode = self.convert(self.grammar, rawnode)
 if newnode is not None:

Since mypyc injects runtime type checks, simplifying Optional[Text] to Text reduces function call overhead. value only needs to pass (in equivalent C code) isinstance(value, str). This also has the neat side-effect of allowing me to remove some asserts.

Tightening up type annotations involving typing.Any can be particularly worthwhile as it forces the use of generic C code that can handle any kind of object. I could only make this change once, but it’s better than nothing.

@@ -54,14 +56,14 @@ class Driver(object):
 self.logger = logger
 self.convert = convert

- def parse_tokens(self, tokens: Iterable[Any], debug: bool = False) -> NL:
+ def parse_tokens(self, tokens: Iterable[GoodTokenInfo], debug: bool = False) -> NL:
 """Parse a series of tokens and return the syntax tree."""
 # XXX Move the prefix computation into a wrapper around tokenize.
 p = parse.Parser(self.grammar, self.convert)
 p.setup()
 lineno = 1
 column = 0

Marking everything Final

Avoiding calculating the same value over and over again is one of the most common optimizations out there, and it’s for good reason, it’s usually easy to fix. BUT, with mypyc we can take this further by using typing.Final. Final variables can often be injected at lookup sites at compile time, skipping the lookups at runtime!

Let me show an example, let’s take this code and see what adding a single Final does.

SCALE = 5

def calculate_y(x):
 return x * SCALE

Compiling it with mypyc, the C code for the calculate_y function is .. well .. quite long! Notice how much work has to be done to safely look up the global value at runtime.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


PyObject *CPyDef_calculate_y(PyObject *cpy_r_x) {
 PyObject *cpy_r_r0;
 PyObject *cpy_r_r1;
 PyObject *cpy_r_r2;
 CPyTagged cpy_r_r3;
 PyObject *cpy_r_r4;
 PyObject *cpy_r_r5;
 PyObject *cpy_r_r6;
CPyL0: ;
 cpy_r_r0 = CPyStatic_globals;
 cpy_r_r1 = CPyStatics[3]; /* 'SCALE' */
 cpy_r_r2 = CPyDict_GetItem(cpy_r_r0, cpy_r_r1);
 if (unlikely(cpy_r_r2 == NULL)) {
 CPy_AddTraceback("final.py", "calculate_y", 6, CPyStatic_globals);
 goto CPyL4;
 }
CPyL1: ;
 if (likely(PyLong_Check(cpy_r_r2)))
 cpy_r_r3 = CPyTagged_FromObject(cpy_r_r2);
 else {
 CPy_TypeError("int", cpy_r_r2); cpy_r_r3 = CPY_INT_TAG;
 }
 CPy_DECREF(cpy_r_r2);
 if (unlikely(cpy_r_r3 == CPY_INT_TAG)) {
 CPy_AddTraceback("final.py", "calculate_y", 6, CPyStatic_globals);
 goto CPyL4;
 }
CPyL2: ;
 cpy_r_r4 = CPyTagged_StealAsObject(cpy_r_r3);
 cpy_r_r5 = PyNumber_Multiply(cpy_r_x, cpy_r_r4);
 CPy_DECREF(cpy_r_r4);
 if (unlikely(cpy_r_r5 == NULL)) {
 CPy_AddTraceback("final.py", "calculate_y", 6, CPyStatic_globals);
 goto CPyL4;
 }
CPyL3: ;
 return cpy_r_r5;
CPyL4: ;
 cpy_r_r6 = NULL;
 return cpy_r_r6;
}

If we mark the SCALE variable as Final, mypyc will notice that and inline the value.

from typing import Final

SCALE: Final = 5

def calculate_y(x):
 return x * SCALE

Just look at how much shorter this is!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


PyObject *CPyDef_calculate_y(PyObject *cpy_r_x) {
 PyObject *cpy_r_r0;
 PyObject *cpy_r_r1;
 PyObject *cpy_r_r2;
CPyL0: ;
 cpy_r_r0 = CPyTagged_StealAsObject(10);
 cpy_r_r1 = PyNumber_Multiply(cpy_r_x, cpy_r_r0);
 CPy_DECREF(cpy_r_r0);
 if (unlikely(cpy_r_r1 == NULL)) {
 CPy_AddTraceback("final.py", "calculate_y", 6, CPyStatic_globals);
 goto CPyL2;
 }
CPyL1: ;
 return cpy_r_r1;
CPyL2: ;
 cpy_r_r2 = NULL;
 return cpy_r_r2;
}

If the value can’t be inlined, say because it involves a function call then mypyc will just replace the globals dictionary lookup and instead use a C static (which is still faster).

Applying this optimization to Black yields changes like the following to black.comments and black.parsing¹ respectively:

@@ -12,11 +18,10 @@ from black.nodes import STANDALONE_COMMENT, WHITESPACE
 # types
 LN = Union[Leaf, Node]

-
-FMT_OFF = {"# fmt: off", "# fmt:off", "# yapf: disable"}
-FMT_SKIP = {"# fmt: skip", "# fmt:skip"}
-FMT_PASS = {*FMT_OFF, *FMT_SKIP}
-FMT_ON = {"# fmt: on", "# fmt:on", "# yapf: enable"}
+FMT_OFF: Final = {"# fmt: off", "# fmt:off", "# yapf: disable"}
+FMT_SKIP: Final = {"# fmt: skip", "# fmt:skip"}
+FMT_PASS: Final = {*FMT_OFF, *FMT_SKIP}
+FMT_ON: Final = {"# fmt: on", "# fmt:on", "# yapf: enable"}


 @dataclass

@@ -36,6 +36,11 @@ except ImportError:
 else:
 ast3 = ast27 = ast

+if sys.version_info >= (3, 8):
+ TYPE_IGNORE_CLASSES: Final = (ast3.TypeIgnore, ast27.TypeIgnore, ast.TypeIgnore)
+else:
+ TYPE_IGNORE_CLASSES: Final = (ast3.TypeIgnore, ast27.TypeIgnore)
+

 class InvalidInput(ValueError):
 """Raised when input source code fails all parse attempts."""
@@ -160,10 +165,7 @@ def stringify_ast(

 for field in sorted(node._fields): # noqa: F402
 # TypeIgnore has only one field 'lineno' which breaks this comparison
- type_ignore_classes: Tuple[Type, ...] = (ast3.TypeIgnore, ast27.TypeIgnore)
- if sys.version_info >= (3, 8):
- type_ignore_classes += (ast.TypeIgnore,)
- if isinstance(node, type_ignore_classes):
+ if isinstance(node, TYPE_IGNORE_CLASSES):
 break

 try:

In total, this was the most common optimization I applied throughout this whole project. It’s simple but effective!

Taking advantage of early binding

Final is so fast because of early binding. What’s great is you can take advantage of early binding with function calls too, assuming your code is static enough.

Time for another example:

from typing import Callable, List

def tag(item: object) -> str:
 return "<tag>" + repr(item)

def process_items(func: Callable[[object], object], items: List[object]) -> List[object]:
 return [func(i) for i in items]

process_items(tag, ["1", "2", "3"])

The function used to process the items isn’t known until call time, forcing mypyc to fall back to the standard Python calling convention (albeit it does use the faster vectorcall convention available for C functions). If I instead hardcode tag in process_items, mypyc can call the C function directly which involves way less work.

from typing import List

def tag(item: object) -> str:
 return "<tag>" + repr(item)

def process_items(List[object]) -> List[object]:
 return [tag(i) for i in items]

process_items(["1", "2", "3"])

 CPyL3: ;
 cpy_r_r8 = CPyList_GetItemUnsafe(cpy_r_items, cpy_r_r3);
 cpy_r_i = cpy_r_r8;
- PyObject *cpy_r_r9[1] = {cpy_r_i};
- cpy_r_r10 = (PyObject **)&cpy_r_r9;
- cpy_r_r11 = _PyObject_Vectorcall(cpy_r_func, cpy_r_r10, 1, 0);
- if (unlikely(cpy_r_r11 == NULL)) {
+ cpy_r_r9 = CPyDef_convert(cpy_r_i);
+ CPy_DECREF(cpy_r_i);
+ if (unlikely(cpy_r_r9 == NULL)) {
 CPy_AddTraceback("early_binding.py", "process_items", 7, CPyStatic_globals);
 goto CPyL8;
 }

Most of the time this isn’t possible because if your function calls are dynamic, it’s probably because the code depends on it … but keep it in mind.

By sheer luck, I was able to replace two dynamic function calls with static calls in blib2to3.pgen2.parse.Parser, the very hot parser code!

diff --git a/src/blib2to3/pgen2/parse.py b/src/blib2to3/pgen2/parse.py
index 6b03188..b5da4fa 100644
--- a/src/blib2to3/pgen2/parse.py
+++ b/src/blib2to3/pgen2/parse.py
@@ -23,7 +23,7 @@ from typing import (
 Set,
 )
 from blib2to3.pgen2.grammar import Grammar
-from blib2to3.pytree import NL, Context, RawNode, Leaf, Node
+from blib2to3.pytree import convert, NL, Context, RawNode, Leaf, Node

@@ -199,16 +206,13 @@ class Parser(object):
 raise ParseError("bad token", type, value, context)
 return ilabel

 def shift(self, type: int, value: Text, newstate: int, context: Context) -> None:
 """Shift a token. (Internal)"""
 dfa, state, node = self.stack[-1]
 rawnode: RawNode = (type, value, context, None)
- newnode = self.convert(self.grammar, rawnode)
- if newnode is not None:
- assert node[-1] is not None
- node[-1].append(newnode)
+ newnode = convert(self.grammar, rawnode)
+ assert node[-1] is not None
+ node[-1].append(newnode)
 self.stack[-1] = (dfa, newstate, node)

 def push(self, type: int, newdfa: DFAS, newstate: int, context: Context) -> None:
 @@ -221,12 +225,11 @@ class Parser(object):
 def pop(self) -> None:
 """Pop a nonterminal. (Internal)"""
 popdfa, popstate, popnode = self.stack.pop()
- newnode = self.convert(self.grammar, popnode)
- if newnode is not None:
- if self.stack:
- dfa, state, node = self.stack[-1]
- assert node[-1] is not None
- node[-1].append(newnode)
- else:
- self.rootnode = newnode
- self.rootnode.used_names = self.used_names
+ newnode = convert(self.grammar, popnode)
+ if self.stack:
+ dfa, state, node = self.stack[-1]
+ assert node[-1] is not None
+ node[-1].append(newnode)
+ else:
+ self.rootnode = newnode
+ self.rootnode.used_names = self.used_names

Also since blib2to3.pytree.convert is guaranteed to return a non-None value, I could drop a branch which was neat :)

Detour: let’s build developer tooling!

If you’ve worked with me before you’ll know that I love to write developer tooling to make life easier. (Un)fortunately this project needed two bits of tooling that simply didn’t exist at the time:

A benchmark suite
A tool to compare two builds of Black behaviourally

The first one is pretty self-explanatory, I needed a good benchmark suite to make sure this project would actually improve performance, and to also quantify the gains (important when weighing optimizations).

The second one is less clear, I effectively wanted mypy-primer, but for Black:

mypy-primer comment on mypy PR #12064.

So I got to work creating blackbench and (the original) diff-shades. In hindsight, blackbench sucks and needs a rewrite so I don’t want to go into too much detail about it, but the summary is that it came with benchmarks for the following tasks:

Formatting with safety checks
Formatting without safety checks
Blib2to3 parsing

across quite a few inputs. The story is similar for the original diff-shades, it worked and made it possible to verify mypyc didn’t change formatting, but its code was horrible (and not to mention unmaintainable). It was bad enough that I rewrote the tool later on.

The TL;DR version of what diff-shades does is that it clones a bunch of projects, runs Black on ’em while recording the results. Then you use its other commands to analyze and compare recordings.

Detour: does GCC help?

At this point I had found a workaround to the GCC array subscript 1 is above array bounds error and was curious to whether using GCC would produce faster binaries. The answer turned out to be very much no.

Anyway, other than getting distracted by diff-shades, I looked into the gcc issue to find a potential workaround. I found one, but it was useless 🙂 Collecting numbers for both gcc-10 and clang showed that gcc fails in basically all departments:

takes almost twice as long to compile Black AND definitely uses 200%+ more memory

had no meaningful difference in generated binary size all while being far more picky about the C code it’s given

oh and of course produced binaries that were around 8% slower 🙃

I’m honestly happy that GCC failed to compile on that parser setup code, clang (for Linux) is so much better.

Results

Quick note
I also did some micro-optimizations like reordering if checks to hit the common case first or replacing x = x + 1 with x += 1, but to this day I don’t know whether they actually had an impact.

Additionally, I haven’t discussed the last optimization round I did for src/black, but there’s nothing in that which I haven’t covered yet.

Anyhow, these optimizations bumped the parsing speedup over interpreted from ~1.73x to ~1.9x. You can see the changes made here and also here.

For more detail, see the tables at the bottom for the compiled-mypyc-preopt and compiled-mypyc columns in this report I compiled. It took a long time to compile this report by the way, setting up a properly configured benchmark setup and gathering multiple data samples is very time consuming!

In the end, I managed to increase compiled performance by an additional 10-15% which is pretty nice! I was aiming for 25%, but in hindsight I might have been hoping for too much 🙂. Also yes, they were virtually useless when interpreted.

We’re nearly there, only Pt. 3 - Deployment remains. It’s shorter, believe me.

In the end, I had to revert this optimization so Black wouldn’t crash under PyPy, can’t remember what the error was though ↩︎

Compiling Black with mypyc, Pt. 1 - Initial Steps

Tue, 31 May 2022 16:25:00 -0400

Release 22.1.0 of Black was special, not only was it the first stable version of Black, it was also the first release to ship with mypyc compiled wheels, doubling performance¹ 🎉.

The journey getting here took over two years, the creation of two new dev tools, and lots and lots of headscratching. Let’s go down memory lane, shall we?

This is a long story so it’s broken up into three parts. Collectively this is the Compiling Black with mypyc series.

Pt. 1 - Initial Steps (you’re reading this one)
Pt. 2 - Optimization
Pt. 3 - Deployment

While I’d love it if you would read this end to end, I totally understand if this ends up in the “to read later” section of your bookmarks. I just hope you enjoy whatever you do read!

… and yes it took me almost 6 months after deploying mypyc to write this.

Introduction to mypyc

Before I dig into the story of how I integrated mypyc into Black, I want to explain what mypyc is and show an example using it. mypyc is a transcompiler converting typed Python code into fast C extensions. Being built on top of the mypy type checker, it leverages standard type annotations² (unlike Cython).

It has been used to compile mypy (and itself since mypyc comes with mypy) since 2019, giving it a 4x performance boost over interpreted Python. According to the mypyc project:

Existing code with type annotations is often 1.5x to 5x faster when compiled. Code tuned for mypyc can be 5x to 10x faster.³

It achieves these impressive speed ups by:

Using the C-API directly, avoiding the CPython interpreter overhead.
Leveraging early binding, resolving called functions and names at compile-time, skipping the expensive dictionary lookup at runtime.
Using optimized, type-specific primitives for many built-in functions and methods.
Using custom memory-efficient, unboxed representations for integers and booleans.
Compiling regular classes to C extension classes. C extension classes use vtables for fast method calls and attribute access.

… and other optimizations I’ve left out for the sake of brevity.

Time for an example

Interested? I sure hope so since it’s time for an example! To get started, create a throwaway directory somewhere, a virtual environment, and install mypy. As of writing I’m using mypy 0.931 on Ubuntu 20.04.03 with CPython 3.8.5.

Next, let’s take a look at the code⁴ we’ll soon compile. It’s a simple program that bruteforces its way to find all of the factor pairs for a given product. I mostly chose this example because it’s very dear to my heart. It was the first “serious” program I wrote while learning Python⁵. I originally wrote the same logic in code.org’s app lab many years ago.

import time
from typing import List, Tuple

def compute(product: int) -> List[Tuple[int, int]]:
 answers = []
 for factor in range(1, product + 1):
 if not product % factor:
 factor2 = int(product / factor)
 answers.append((factor, factor2))
 return answers

t0 = time.perf_counter()
compute(27_000_000)
elapsed = time.perf_counter() - t0
print(f"compute() took {elapsed * 1000:.1f} milliseconds")

Once saved, run it noting the time it takes:

$ python coefficient_finder.py
compute() took 1880.7 milliseconds

Alright, it’s a bit slow, let’s see if mypyc can change that:

$ mypyc coefficient_finder.py
running build_ext
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/build
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/ichard26/programming/webdev/blog/testing-grounds/venv/lib/python3.8/site-packages/mypyc/lib-rt -I/home/ichard26/programming/webdev/blog/testing-grounds/venv/include -I/opt/python3.8.5/include/python3.8 -c build/__native.c -o build/temp.linux-x86_64-3.8/build/__native.o -O3 -g1 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable
creating build/lib.linux-x86_64-3.8
gcc -pthread -shared build/temp.linux-x86_64-3.8/build/__native.o -o build/lib.linux-x86_64-3.8/coefficient_finder.cpython-38-x86_64-linux-gnu.so

$ python -c "import coefficient_finder"
compute() took 331.6 milliseconds

And would you look at that, mypyc gave us a near 6x improvement over CPython! What’s neat is that I didn’t even need to type all of the variables thanks to mypy’s type inference.

And actually, since the mypyc script is a simple wrapper that generates a setuptools project and builds it in-place, you can peek into the build directory and look at the generated C code (should be in __native.c).

Feel free to edit the program and play around with mypyc, seeing what little scripts it can speed up. Pro-tip, to restore back to the interpreted version of your program, you can delete the .so file on Linux & MacOS or the .pyd file on Windows. In this example I got coefficient_finder.cpython-38-x86_64-linux-gnu.so.

It’s not a silver bullet

There’s several downsides to using mypyc. While the project aims for excellent compatibility with CPython, there are some major deviations worth noting:

Types are enforced at runtime and violations will cause a TypeError (this makes using typing.Any quite dangerous).
Compiled modules can’t be run directly, you have to import ’em instead.
Monkeypatching anything compiled is unlikely to work as compiled code skips many of the runtime __dict__ lookups normal Python does.
Assignments to class and instance namespaces will either error or do nothing, in particular you can’t add previously undeclared attributes later on.
Standard multiple inheritance isn’t supported.
Profilers (e.g. CProfile), debuggers (e.g. pdb), and tracing hooks (e.g. coverage) won’t work as the C code doesn’t trigger the relevant hooks.

There are more details available in the official mypyc documentation here and also here.

black + mypyc: Initial steps

I wasn’t the first person to integrate mypyc into Black; way back in September 2019, @msullivan opened a PR getting Black ready for compilation. Unfortunately, as is typical in open source projects, no one took the half-completed work and pushed it to completion … for almost two years.

You might be wondering why performance even matters, well clearly it mattered a lot since GH-366 was opened in June 2018! The TL;DR is that for environments where Black is ran on save automatically, the more responsive it is the better as less time is spent in a laggy editor window.

Start up time is important too given imports are costly (it turns out mypyc can reduce import time too!), but this issue was explicitly about formatting throughput.

Given I first publicly announced my work finishing up the project on July 4th 2021, I sadly don’t remember why I decided to pick it up. All I remember was fighting mypyc compile-time type errors and fighting an outdated gcc… well OK I do remember I was looking to learn more about type systems and C development in general, but that’s it, I promise!

First off, as of writing mypyc has a bug (mypyc#885) where its boxing & unboxing code triggers array subscript 1 is above array bounds on blib2to3.pgen2.parse.Parser.setup. I simplified the reproduction case to the following:

from typing import Optional, Tuple

Context = Tuple[str, Tuple[int, int]]
RawNode = Tuple[int, Optional[str], Optional[Context]]

def setup() -> None:
 # This is what ticks off gcc.
 newnode: RawNode = (7, None, None)

… yeah, I know this makes mypyc look fragile. While it’s not entirely stupid to use it in production (because I did so), you’d be right in pointing out it’s alpha quality and needs careful testing. And to be fair, poor past me didn’t know just how many issues I’d soon face. All I can hope is that this work will help mypyc improve ❀

Anyway I switched to clang and this issue disappeared …

Trying to compile black and blib2to3⁶ initially seemed like a small task with just these type errors:

$ python setup.py --use-mypyc bdist_wheel
Parsed and typechecked in 8.566s
Compiled to C in 0.000s
src/black/parsing.py:30:9: error: Cannot assign multiple modules to name "ast3" without explicit "types.ModuleType" annotation
src/black/parsing.py:30:9: error: Cannot assign multiple modules to name "ast27" without explicit "types.ModuleType" annotation
src/black/parsing.py:146:13: error: Incompatible types in assignment (expression has type "Tuple[Type[typed_ast.ast3.TypeIgnore], Type[typed_ast.ast27.TypeIgnore], Type[_ast.TypeIgnore]]", variable has type "Tuple[Type[typed_ast.ast3.TypeIgnore], Type[typed_ast.ast27.TypeIgnore]]")

The ones about Cannot assign multiple modules to name ... are valid if annoying, back then the code in black.parsing was pretty dynamic. The other one was just that mypy under “mypyc super-strict mode” won’t infer a variable to be a variable-length tuple even if it’s extended right after:

def stringify_ast(
 node: Union[ast.AST, ast3.AST, ast27.AST], depth: int = 0
) -> Iterator[str]:
 """Simple visitor generating strings to compare ASTs by content."""

 node = fixup_ast_constants(node)

 yield f"{' ' * depth}{node.__class__.__name__}("

 for field in sorted(node._fields): # noqa: F402
 # TypeIgnore has only one field 'lineno' which breaks this comparison
 type_ignore_classes = (ast3.TypeIgnore, ast27.TypeIgnore)
 if sys.version_info >= (3, 8):
 type_ignore_classes += (ast.TypeIgnore,)

The fix was to add Tuple[Type, ...].

type_ignore_classes: Tuple[Type, ...] = (ast3.TypeIgnore, ast27.TypeIgnore)

Now, the codebase was able to type check even under “mypyc super-strict mode,” but obviously that just meant I had another class of fires to deal with, *compiler crashes*!

$ python setup.py --use-mypyc bdist_wheel
Parsed and typechecked in 8.929s
Compiling cache, strings, black, debug, parsing, brackets, lines, files, literals, conv, trans, linegen, numerics, nodes, const, pgen, pygram, output, token, concurrency, parse, report, mode, driver, grammar, tokenize, pytree, rusty, comments
Traceback (most recent call last):
 File "mypyc/irbuild/builder.py", line 169, in accept
 File "mypy/nodes.py", line 950, in accept
 File "mypyc/irbuild/visitor.py", line 104, in visit_class_def
 File "mypyc/irbuild/classdef.py", line 137, in transform_class_def
 File "mypyc/irbuild/classdef.py", line 388, in generate_attr_defaults
 File "mypyc/ir/class_ir.py", line 174, in attr_type
 File "mypyc/ir/class_ir.py", line 171, in attr_details
src/black/trans.py:187: KeyError: "'CustomSplitMapMixin' has no attribute '_Key'"

This issue was just due to missing ClassVar declarations for attributes that are functionally class-only attributes. The fix was pretty easy :)

+@trait
 class CustomSplitMapMixin:
 """
 This mixin class is used to map merged strings to a sequence of
@@ -191,8 +204,10 @@ class CustomSplitMapMixin:
 the resultant substrings go over the configured max line length.
 """

- _Key = Tuple[StringID, str]
- _CUSTOM_SPLIT_MAP: Dict[_Key, Tuple[CustomSplit, ...]] = defaultdict(tuple)
+ _Key: ClassVar = Tuple[StringID, str]
+ _CUSTOM_SPLIT_MAP: ClassVar[Dict[_Key, Tuple[CustomSplit, ...]]] = defaultdict(
+ tuple
+ )

Do you see that @trait addition? well … mypyc does in fact support a limited form of multiple inheritance called traits. They’re basically mixins. As noted in the mypyc documentation, they shouldn’t be instantiated or subclass non-traits.

And as usual, fixing this crash revealed *even* *more* *issues*, albeit mypyc specific this time:

$ python setup.py --use-mypyc bdist_wheel
Parsed and typechecked in 8.122s
Compiling pgen, parse, parsing, mode, pytree, token, debug, const, lines, numerics, rusty, conv, comments, files, trans, cache, black, grammar, output, brackets, tokenize, linegen, pygram, report, literals, nodes, strings, concurrency, driver
Compiled to C in 0.970s
src/black/trans.py:250: error: Non-trait bases must appear first in parent list
src/black/trans.py:934: error: Non-trait bases must appear first in parent list
src/black/trans.py:1433: error: Non-trait bases must appear first in parent list
src/black/brackets.py:52: error: Inheriting from most builtin types is unimplemented
src/black/files.py:52: warning: Treating generator comprehension as list
src/black/trans.py:746: warning: Unsupported default attribute value
src/black/trans.py:1831: warning: Unsupported default attribute value
src/blib2to3/pgen2/tokenize.py:430: error: Local variable "stashed" has inferred type None; add an annotation
src/black/nodes.py:442: error: Local variable "stop_after" has inferred type None; add an annotation
src/black/nodes.py:443: error: Local variable "last" has inferred type None; add an annotation

These were pretty straightforward to fix and sometimes even improved code clarity, forcing me to add type annotations in complex code.

More changes were necessary than expected

All of this was only to get the codebase to type check and not crash the compiler. Getting the built binary to not crash at runtime needed more work. Some of these changes were improvements, like this one which involved an incorrect type annotation:

diff --git a/src/black/__init__.py b/src/black/__init__.py
index 8e2123d..6998c1e 100644
--- a/src/black/__init__.py
+++ b/src/black/__init__.py
@@ -359,7 +359,7 @@ def main(
 experimental_string_processing: bool,
 quiet: bool,
 verbose: bool,
- required_version: str,
+ required_version: Optional[str],
 include: Pattern,
 exclude: Optional[Pattern],
 extend_exclude: Optional[Pattern],

This goes to show that while the strict runtime type checks can be very annoying, they do have value beyond memory safety.

Sad and frustrating changes

… but others were just sad or annoying (mixing dataclasses with anything remotely fancy breaks mypyc).

diff --git a/src/black/linegen.py b/src/black/linegen.py
index 76b553a..1691cc5 100644
--- a/src/black/linegen.py
+++ b/src/black/linegen.py
@@ -40,7 +38,8 @@ class CannotSplit(CannotTransform):
 """A readable split that fits the allotted line length is impossible."""


-@dataclass
+# This isn't a dataclass because @dataclass + Generic breaks mypyc.
+# See also https://github.com/mypyc/mypyc/issues/827.
 class LineGenerator(Visitor[Line]):
 """Generates reformatted Line objects. Empty lines are not emitted.

@@ -48,9 +47,11 @@ class LineGenerator(Visitor[Line]):
 in ways that will no longer stringify to valid Python code on the tree.
 """

- mode: Mode
- remove_u_prefix: bool = False
- current_line: Line = field(init=False)
+ def __init__(self, mode: Mode, remove_u_prefix: bool = False) -> None:
+ self.mode = mode
+ self.remove_u_prefix = remove_u_prefix
+ self.current_line: Line
+ self.__post_init__()

 def line(self, indent: int = 0) -> Iterator[Line]:
 """Generate a line.

diff --git a/src/black/trans.py b/src/black/trans.py
index 023dcd3..d918ef1 100644
--- a/src/black/trans.py
+++ b/src/black/trans.py
@@ -62,7 +71,6 @@ def TErr(err_msg: str) -> Err[CannotTransform]:
 return Err(cant_transform)


-@dataclass # type: ignore
 class StringTransformer(ABC):
 """
 An implementation of the Transformer protocol that relies on its
@@ -90,9 +98,13 @@ class StringTransformer(ABC):
 as much as possible.
 """

- line_length: int
- normalize_strings: bool
- __name__ = "StringTransformer"
+ __name__: Final = "StringTransformer"
+
+ # Ideally this would be a dataclass, but unfortunately mypyc breaks when used with
+ # `abc.ABC`.
+ def __init__(self, line_length: int, normalize_strings: bool) -> None:
+ self.line_length = line_length
+ self.normalize_strings = normalize_strings

Hackiness += 10000

One change stood out as the most hacky⁷ code I’ve probably ever written so far:

diff --git a/src/black/linegen.py b/src/black/linegen.py
index 76b553a..1691cc5 100644
--- a/src/black/linegen.py
+++ b/src/black/linegen.py
@@ -335,7 +336,9 @@ def transform_line(
 transformers = [left_hand_split]
 else:

- def rhs(line: Line, features: Collection[Feature]) -> Iterator[Line]:
+ def _rhs(
+ self: object, line: Line, features: Collection[Feature]
+ ) -> Iterator[Line]:
 """Wraps calls to `right_hand_split`.

 The calls increasingly `omit` right-hand trailers (bracket pairs with
@@ -362,6 +365,11 @@ def transform_line(
 line, line_length=mode.line_length, features=features
 )

+ # HACK: functions (like rhs) compiled by mypyc don't retain their __name__
+ # attribute which is needed in `run_transformer` further down. Unfortunately
+ # a nested class breaks mypyc too. So a class must be created via type ...
+ rhs = type("rhs", (), {"__call__": _rhs})()
+
 if mode.experimental_string_processing:
 if line.inside_brackets:
 transformers = [

I also had to change the code accessing __name__ to do so via __class__ so this hack could work. Minor correction though, top-level compiled functions still have __name__, it’s just nested functions that don’t have ’em⁸. I improved this comment before this landed.

The least bad workaround

I remember getting an impromptu “how to read mypyc-generated C code” lesson from @JelleZijlstra ⁹. I was trying to debug this runtime crash in this black.trans class:

class StringParser:
 """
 [snipped]
 """

 DEFAULT_TOKEN = -1

 # String Parser States
 START = 1
 DOT = 2
 NAME = 3
 PERCENT = 4
 SINGLE_FMT_ARG = 5
 LPAR = 6
 RPAR = 7
 DONE = 8

 # Lookup Table for Next State
 _goto: Dict[Tuple[ParserState, NodeType], ParserState] = {
 [snipped]
 }

 [snipped]

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "test2.py", line 30, in <module>
 (START, DEFAULT_TOKEN): DONE,
KeyError: 'DEFAULT_TOKEN

It turns out the generated code wanted to access the DEFAULT_TOKEN attribute from the globals dict, predictably blowing up.

 cpy_r_r146 = CPyStatic_globals;
 cpy_r_r147 = CPyStatics[17]; /* 'DEFAULT_TOKEN' */
 cpy_r_r148 = CPyDict_GetItem(cpy_r_r146, cpy_r_r147);

If this scares you, don’t fret, it’s not impossible to read. First, the globals dict is assigned to cpy_r_r146, then the name (or key) we’re looking up is read and stored in cpy_r_r147. Finally, cpy_r_r146 and cpy_r_r147 are passed to the CPyDict_GetItem function.

Effectively, the buggy C code was trying to do this (unnecessary lines were replaced with ...):

class StringParser:
 DEFAULT_TOKEN = -1
 START = 1
 DONE = 8
 ...

 __var_146 = globals()
 __var_147 = "DEFAULT_TOKEN"
 __var_148 = var_146[var_147]

 _goto = {
 (START, __var_148): DONE
 ...
 }

The root issue (mypyc#862) is that mypyc doesn’t understand class level scoping … or that negative integers are constants (using a positive integer fixed the crash). I embedded the date I workarounded this issue, I think it’s a pretty cute historical codebase quirk ❀

@@ -1811,20 +1826,20 @@ class StringParser:
 ```
 """

- DEFAULT_TOKEN = -1
+ DEFAULT_TOKEN: Final = 20210605

Phew, it’s faster

It was at this time I finally posted the first set of benchmark numbers, they weren’t scientific or reliable in any way, but it did prove mypyc was still a worthwhile way to improve performance. Formatting the Black codebase (so meta, I know) with compiled Black yielded a performance gain of 1.82x, impressive for the short amount of time I had put in!

I had to verify this experimental build wasn’t completely unstable though. I couldn’t just run the test suite with compiled Black installed, some tests use mocks which trigger the strict type checks mypyc does at runtime. Sadly, I had to mark these tests as straight up incompatible so they could be skipped.

It’s still alpha software

As of writing mypyc is still alpha software with rough edges everywhere, I already showed quite a few of them above, but there were more 😥 You can read the rest in this integration report I filed in the mypyc issue tracker.

And I know, y’all are wondering why Black didn’t use Cython, well I’m not sure as I wasn’t around in 2019 when @ambv first suggested using mypyc. Eitherway I didn’t look into Cython because at the time as it seemed like I only had the final scraps left and didn’t have too much work to do … which was both right and wrong depending on how you look at it.

Could Black be even faster if we had used Cython? Probably, but that would have required learning C which would have taken quite a while.

Anyway, it’s time for Pt. 2 - Optimization.

When I first landed the relevant PR it was an overall 2x improvement, but once Jelle added a stability hotfix the effective speedup for files that were changed is 50%. If you’re formatting a bunch of already well formatted files, the speedup is still 2x ↩︎
There’s too many typing related PEPs to copy and paste, but there’s an up to date list here: https://docs.python.org/3/library/typing.html#relevant-peps ↩︎
https://mypyc.readthedocs.io/en/stable/introduction.html#introduction ↩︎
I am well aware I don’t need to import List or Tuple from typing anymore. I’m just keeping my code examples compatible with older versions of Python. ↩︎
Well not quite exactly this, it was a crappier solution with a time complexity of O(n²) instead of O(n), oh and of course my code style was less than ideal and I didn’t use type annotations, but let’s not go there :) ↩︎
It’s a fork of lib2to3, see https://github.com/psf/black/blob/1e557184b0a9f43bfbff862669966bc5328517e9/src/blib2to3/README for why. ↩︎
What’s funny is that the PR which added this __name__ check had a review noting the hackiness of reading the attribute in the first place. It was dismissed as it would have taken too much effort to properly implement the required logic, turns out we got this fun code in return! ↩︎
It’s because internally mypyc implements nested functions as callable classes ↩︎
Reading the results searching from:ichard26#4772 in:black-formatter mypyc in the Python Discord server nets some interesting discussion and history :p ↩︎

Richard Si – Assorted ramblings

What's new in pip 26.1 - lockfiles and dependency cooldowns!

Python 3.9 is no longer supported

New features ✨

Experimental: installing from pylock files

What’s next?

We need your feedback!

Dependency cooldowns

Lifting several 2020 resolver limitations

Security fixes

Avoid mistreating a .tar.gz archive as a zip file (CVE-2026-3219)

Fix ACE caused by self check deferred imports (CVE-2026-6357)

Upgrade to urllib3 2.x

Deprecations & upcoming removals

Acknowledgements

What's new in pip 26.0 - prerelease and upload-time filtering!

New features ✨

Excluding distributions by upload time

Separate build constraints

--only-final and --all-releases

Installing from inline script metadata (PEP 723)

Experimental: In-process build dependencies

Deprecations & upcoming removals

Acknowledgements

What's new in pip 25.2 ... and a sneak peek of 25.3

Key changes

Deprecations & upcoming removals

Sneak peek: in-process build dependencies

What's new in pip 25.1 - Dependency groups!

Key features ✨

Dependency groups (PEP 735)

Package installation progress bar

Resumable downloads

Experimental lockfile (PEP 751) generation: pip lock

pip index versions is stable

Key bugfixes

Dependency resolution improvements

Legacy .egg distributions are only detected once

Deprecations & upcoming removals

Acknowledgements

Speeding up pip's Windows CI by setting TEMP to the D: drive

How slow was pip’s Windows CI?

Windows file I/O is poor on GitHub Actions

Moving TEMP to the D: drive

What about ReFS or Dev Drives?

What's new in pip 25.0

Key features

PEP 639: SPDX License Expressions

Network cache inherits permissions from root cache directory

Key bugfixes

Certificate and proxy options are used for build dependencies

Faster package candidate collection

Upcoming removals

What's new in pip 24.3

PSA: Legacy editable installs are deprecated

QoL improvements

Recursive requirement files are now detected

Installed packages with broken dependency metadata now raise a proper error

macOS 10.12 is resupported

Invalid wheel filenames are deprecated

Other changes

What's new in pip 24.2 — or why legacy editable installs are deprecated

PSA, PSA, PSA 📣

Legacy editable installs are deprecated

OK, so what now?

System HTTPS certificates by default(*)

Performance optimizations, duh!

Faster discovery of installed packages

Other optimizations

pip check just got a bit stricter

Oh yeah, --require-virtualenv is a thing

Summary

What's new in Black 23.1.0

Code style

The 2023 stable style

The 2023 preview style

Conditional expressions are parenthesized if needed now

Wrap multiple context managers in parentheses if targeting Python 3.9+

Packaging

mypyc wheels are available for all platforms (once again)

`--only-final` and `--all-releases`

Experimental lockfile (PEP 751) generation: `pip lock`

`pip index versions` is stable

Legacy `.egg` distributions are only detected once

`pip check` just got a bit stricter

Oh yeah, `--require-virtualenv` is a thing

`--skip-string-normalization` now prevents docstring prefixes from being normalized

`--skip-magic-trailing-comma` now removes useless trailing commas in subscripts

Spyder cell separator comments (`#%%`) are now normalized

Failing to put `fmt: on` at the same indent level as `fmt: off` no longer crashes

`-x` / `--skip-source-first-line`