PYTHON

Using Python’s pip to Manage Your Projects’ Dependencies – Real Python

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: A Beginner’s Guide to pip

The standard package manager for Python is pip. It allows you to install and manage packages that aren’t part of the Python standard library. If you’re looking for an introduction to pip, then you’ve come to the right place!

In this tutorial, you’ll learn how to:

  • Set up pip in your working environment
  • Fix common errors related to working with pip
  • Install and uninstall packages with pip
  • Manage projects’ dependencies using requirements files

You can do a lot with pip, but the Python community is very active and has created some neat alternatives to pip. You’ll learn about those later in this tutorial.

Getting Started With pip

So, what exactly does pip do? pip is a package manager for Python. That means it’s a tool that allows you to install and manage libraries and dependencies that aren’t distributed as part of the standard library. The name pip was introduced by Ian Bicking in 2008:

I’ve finished renaming pyinstall to its new name: pip. The name pip is [an] acronym and declaration: pip installs packages. (Source)

Package management is so important that Python’s installers have included pip since versions 3.4 and 2.7.9, for Python 3 and Python 2, respectively. Many Python projects use pip, which makes it an essential tool for every Pythonista.

The concept of a package manager might be familiar to you if you’re coming from another programming language. JavaScript uses npm for package management, Ruby uses gem, and the .NET platform uses NuGet. In Python, pip has become the standard package manager.

Finding pip on Your System

The Python installer gives you the option to install pip when installing Python on your system. In fact, the option to install pip with Python is checked by default, so pip should be ready for you to use after installing Python.

You can verify that pip is available by looking for the pip3 executable on your system. Select your operating system below and use your platform-specific command accordingly:

The where command on Windows will show you where you can find the executable of pip3. If Windows can’t find an executable named pip3, then you can also try looking for pip without the three (3) at the end.

The which command on Linux systems and macOS shows you where the pip3 binary file is located.

On Windows and Unix systems, pip3 may be found in more than one location. This can happen when you have multiple Python versions installed. If you can’t find pip in any location on your system, then you may consider reinstalling pip.

Instead of running your system pip directly, you can also run it as a Python module. In the next section, you’ll learn how.

Running pip as a Module

When you run your system pip directly, the command itself doesn’t reveal which Python version pip belongs to. This unfortunately means that you could use pip to install a package into the site-packages of an old Python version without noticing. To prevent this from happening, you should run pip as a Python module:

Notice that you use python -m to run pip. The -m switch tells Python to run a module as an executable of the python interpreter. This way, you can ensure that your system default Python version runs the pip command. If you want to learn more about this way of running pip, then you can read Brett Cannon’s insightful article about the advantages of using python -m pip.

Sometimes you may want to be more explicit and limit packages to a specific project. In situations like this, you should run pip inside a virtual environment.

Using pip in a Python Virtual Environment

To avoid installing packages directly into your system Python installation, you can use a virtual environment. A virtual environment provides an isolated Python interpreter for your project. Any packages that you use inside this environment will be independent of your system interpreter. This means that you can keep your project’s dependencies separate from other projects and the system at large.

Using pip inside a virtual environment has three main advantages. You can:

  1. Be sure that you’re using the right Python version for the project at hand
  2. Be confident that you’re referring to the correct pip instance when running pip or pip3
  3. Use a specific package version for your project without affecting other projects

Python has the built-in venv module for creating virtual environments. This module helps you create virtual environments with an isolated Python installation. Once you’ve activated the virtual environment, then you can install packages into this environment. The packages that you install into one virtual environment are isolated from all other environments on your system.

You can follow these steps to create a virtual environment and verify that you’re using the pip module inside the newly created environment:

Here you initialize a virtual environment named venv by using Python’s built-in venv module. After running the command above, Python creates a directory named venv/ in your current working directory. Then, you activate the virtual environment with the source command. The parentheses (()) surrounding your venv name indicate that you successfully activated the virtual environment.

Finally, you check the version of the pip3 and pip executables inside your activated virtual environment. Both point to the same pip module, so once your virtual environment is activated, you can use either pip or pip3. For consistency, you can also continue to use python -m pip inside the virtual environment.

Reinstalling pip When Errors Occur

When you run the pip command, you may get an error in some cases. Your specific error message will depend on your operating system:

Operating System Error Message
Windows 'pip' is not recognized as an internal or external command,
operable program or batch file.
Linux bash: pip: command not found
macOS zsh: command not found: pip

Error messages like these indicate that something went wrong with the installation of pip.

Getting errors like the ones shown above can be frustrating because pip is vital for installing and managing external packages. Some common problems with pip are related to how this tool was installed on your system.

Although the error messages for various systems differ, they all point to the same problem: Your system can’t find pip in the locations listed in your PATH variable. On Windows, PATH is part of the system variables. On macOS and Linux, PATH is part of the environment variables. You can check the contents of your PATH variable with this command:

The output of this command will show a list of locations (directories) on your disk where the operating system looks for executable programs. Depending on your system, locations can be separated by a colon (:) or a semicolon (;).

By default, the directory that contains the pip executable should be present in PATH after you install Python or create a virtual environment. However, missing pip is a common issue. Two supported methods can help you install pip again and add it to your PATH:

  1. The ensurepip module
  2. The get-pip.py script

The ensurepip module has been part of the standard library since Python 3.4. It was added to provide a straightforward way for you to reinstall pip if, for example, you skipped it when installing Python or you uninstalled pip at some point.
Select your operating system below and run ensurepip accordingly:

If pip isn’t installed yet, then this command installs it in your current Python environment. If you’re in an active virtual environment, then the command installs pip into that environment. Otherwise, it installs pip globally on your system. The --upgrade option ensures that the pip version is the same as the one declared in ensurepip.

Another way to fix your pip installation is to use the get-pip.py script. The get-pip.py file contains a full copy of pip as an encoded ZIP file. You can download get-pip.py directly from the PyPA bootstrap page. Once you have the script on your machine, then you run the Python script like this:

This script will install the latest version of pip, setuptools, and wheel in your current Python environment. If you only want to install pip, then you can add the --no-setuptools and --no-wheel options to your command.

If none of the methods above work, then it might be worth trying to download the latest Python version for your current platform. You can follow the Python Installation & Setup Guide to make sure that pip is appropriately installed and works without errors.

Installing Packages With pip

Python is considered a batteries included language. This means that the Python standard library contains an extensive set of packages and modules to help developers with their coding projects.

At the same time, Python has an active community that contributes an even more extensive set of packages that can help you with your development needs. These packages are published to the Python Package Index, also known as PyPI (pronounced Pie Pea Eye).

PyPI hosts an extensive collection of packages, including development frameworks, tools, and libraries. Many of these packages provide friendly interfaces to the Python standard library’s functionality.

Using the Python Package Index (PyPI)

One of the many packages that PyPI hosts is called requests. The requests library helps you to interact with web services by abstracting the complexities of HTTP requests. You can learn all about requests on its official documentation site.

When you want to use the requests package in your project, you must first install it into your environment. If you don’t want to install it in your system Python site-packages, then you can create a virtual environment first, as shown above.

Once you’ve created the virtual environment and activated it, then your command-line prompt shows the name of the virtual environment inside the parentheses. Any pip commands that you perform from now on will happen inside your virtual environment.

To install packages, pip provides an install command. You can run it to install the requests package:

In this example, you run pip with the install command followed by the name of the package that you want to install. The pip command looks for the package in PyPI, resolves its dependencies, and installs everything in your current Python environment to ensure that requests will work.

The pip install <package> command always looks for the latest version of the package and installs it. It also searches for dependencies listed in the package metadata and installs them to ensure that the package has all the requirements that it needs.

It’s also possible to install multiple packages in a single command:

By chaining the packages rptree and codetiming in the pip install command, you install both packages at once. You can add as many packages as you want to the pip install command. In cases like this, a requirements.txt file can come in handy. Later in this tutorial, you’ll learn how to use a requirements.txt file to install many packages at once.

You can use the list command to display the packages installed in your environment, along with their version numbers:

The pip list command renders a table that shows all installed packages in your current environment. The output above shows the version of the packages using an x.y.z placeholder format. When you run the pip list command in your environment, pip displays the specific version number that you’ve installed for each package.

To get more information about a specific package, you can look at the package’s metadata by using the show command in pip:

The output of this command on your system will list the package’s metadata. The Requires line lists packages, such as certifi, idna, charset-normalizer, and urllib3. These were installed because requests depends on them to work correctly.

Now that you’ve installed requests and its dependencies, you can import it just like any other regular package in your Python code. Start the interactive Python interpreter and import the requests package:

After starting the interactive Python interpreter, you imported the requests module. By calling requests.__version__, you verified that you were using the same version of requests as you expected. You also investigated requests.__file__ to confirm that you’ve imported the requests module from within your virtual environment.

Using a Custom Package Index

By default, pip uses PyPI to look for packages. But pip also gives you the option to define a custom package index.

Using pip with a custom index can be helpful when the PyPI domain is blocked on your network or if you want to work with packages that aren’t publicly available. Sometimes, system administrators also create their own internal package index to better control which package versions are available to pip users on the company’s network.

A custom package index must comply with PEP 503 – Simple Repository API to work with pip. You can get an impression of how such an API (Application Programming Interface) looks by visiting the PyPI Simple Index—but be aware that this is a large page with a lot of hard-to-parse content. Any custom index that follows the same API can be targeted with the --index-url option. Instead of typing --index-url, you can also use the -i shorthand.

For example, to install the rptree tool from the TestPyPI package index, you can run the following command:

With the -i option, you tell pip to look at a different package index instead of PyPI, the default one. Here, you’re installing rptree from TestPyPI rather than from PyPI. You can use TestPyPI to fine-tune the publishing process for your Python packages without cluttering the production package index on PyPI.

If you need to use an alternative index permanently, then you can set the index-url option in the pip configuration file. This file is called pip.conf, and you can find its location by running the following command:

With the pip config list command, you can list the active configuration. This command only outputs something when you have custom configurations set. Otherwise, the output is empty. That’s when the additive --verbose, or -vv, option can be helpful. When you add -vv, pip shows you where it looks for the different configuration levels.

If you want to add a pip.conf file, then you can choose one of the locations that pip config list -vv listed. A pip.conf file with a custom package index looks like this:

When you have a pip.conf file like this, pip will use the defined index-url to look for packages. With this configuration, you don’t need to use the --index-url option in your pip install command to specify that you only want packages that can be found in the Simple API of TestPyPI.

Installing Packages From Your Git Repositories

You’re not limited to packages hosted on PyPI or other package indexes. pip also provides the option to install packages from a Git repository. But even when a package is hosted on PyPI, like the Real Python directory tree generator, you can opt to install it from its Git repository:

With the git+https scheme, you can point to a Git repository that contains an installable package. You can verify that you installed the package correctly by running an interactive Python interpreter and importing rptree:

After starting the interactive Python interpreter, you import the rptree module. By calling rptree.__version__, you verify that you’re using the rptree module that’s based in your virtual environment.

Installing packages from a Git repository can be helpful if the package isn’t hosted on PyPI but has a remote Git repository. The remote repository you point pip to can even be hosted on an internal Git server on your company’s intranet. This can be useful when you’re behind a firewall or have other restrictions for your Python projects.

Installing Packages in Editable Mode to Ease Development

When working on your own package, installing it in an editable mode can make sense. By doing this, you can work on the source code while still using your command line like you would in any other package. A typical workflow is to first clone the repository and then use pip to install it as an editable package in your environment:

With the commands above, you installed the rptree package as an editable module. Here’s a step-by-step breakdown of the actions you just performed:

  1. Line 1 cloned the Git repository of the rptree package.
  2. Line 2 changed the working directory to rptree/.
  3. Lines 3 and 4 created and activated a virtual environment.
  4. Line 5 installed the content of the current directory as an editable package.

The -e option is shorthand for the --editable option. When you use the -e option with pip install, you tell pip that you want to install the package in editable mode. Instead of using a package name, you use a dot (.) to point pip to the current directory.

If you hadn’t used the -e flag, pip would’ve installed the package normally into your environment’s site-packages/ folder. When you install a package in editable mode, you’re creating a link in the site-packages to the local project path:

 ~/rptree/venv/lib/python3.12/site-packages/rptree.egg-link

Using the pip install command with the -e flag is just one of many options that pip install offers. You can check out pip install examples in the pip documentation. There you’ll learn how to install specific versions of a package or point pip to a different index that’s not PyPI.

In the next section, you’ll learn how requirements files can help with your pip workflows.

Using Requirements Files

The pip install command by default installs the latest published version of a package, but sometimes your code requires a specific package version to work correctly.

You want to create a specification of the dependencies and versions that you used to develop and test your application so that there are no surprises when you use the application in production.

Pinning Requirements

When you share your Python project with other developers, you may want them to use the same versions of external packages that you’re using. Maybe a specific version of a package contains a new feature that you rely on, or the version of a package that you’re using is incompatible with former versions.

These external dependencies are also called requirements. You’ll often find Python projects that pin their requirements in a file called requirements.txt or similar. The requirements file format allows you to specify precisely which packages and versions should be installed.

Running pip help shows that there’s a freeze command that outputs the installed packages in requirements format. You can use this command, redirecting the output to a file to generate a requirements file:

This command creates a requirements.txt file in your working directory with the following content:

Remember that x.y.z displayed above is a placeholder format for the package versions. Your requirements.txt file will contain real version numbers.

The freeze command dumps the name and version of the currently installed packages to standard output. You can redirect the output to a file that you can later use to install your exact requirements into another system. You can name the requirements file whatever you want. However, a widely adopted convention is to name it requirements.txt.

When you want to replicate the environment in another system, you can run pip install, using the -r switch to specify the requirements file:

In the command above, you tell pip to install the packages listed in requirements.txt into your current environment. The package versions will match the version constraints that the requirements.txt file contains. You can run pip list to display the packages you just installed, with their version numbers:

Now you’re ready to share your project! You can submit requirements.txt into a version control system like Git and use it to replicate the same environment on other machines. But wait, what happens if new updates are released for these packages?

Fine-Tuning Requirements

The problem with hardcoding your packages’ versions and dependencies is that packages are updated frequently with bug and security fixes. You probably want to leverage those updates as soon as they’re published.

The requirements file format allows you to specify dependency versions using comparison operators that give you some flexibility to ensure packages are updated while still defining the base version of a package.

Open requirements.txt in your favorite editor and turn the equality operators (==) into greater than or equal to operators (>=), like in the example below:

You can change the comparison operator to >= to tell pip to install an exact or greater version that has been published. When you set a new environment by using the requirements.txt file, pip looks for the latest version that satisfies the requirement and installs it.

Next, you can upgrade the packages in your requirements file by running the install command with the --upgrade switch or the -U shorthand:

If a new version is available for a listed package, then the package will be upgraded.

In an ideal world, new versions of packages would be backward compatible and would never introduce new bugs. Unfortunately, new versions can introduce changes that’ll break your application. To fine-tune your requirements, the requirements file syntax supports additional version specifiers.

Imagine that a new version, 3.0, of requests is published but introduces an incompatible change that breaks your application. You can modify the requirements file to prevent 3.0 or higher from being installed:

Changing the version specifier for the requests package ensures that any version greater than or equal to 3.0 doesn’t get installed. The pip documentation provides extensive information about the requirements file format, and you can consult it to learn more.

Separating Production and Development Dependencies

Not all packages that you install during the development of your applications will be production dependencies. For example, you’ll probably want to test your application, so you need a test framework. A popular framework for testing is pytest. You want to install it in your development environment, but you don’t want it in your production environment because it isn’t a production dependency.

You create a second requirements file, requirements_dev.txt, to list additional tools to set up a development environment:

Having two requirements files will demand that you use pip to install both of them, requirements.txt and requirements_dev.txt. Fortunately, pip allows you to specify additional parameters within a requirements file, so you can modify requirements_dev.txt to also install the requirements from the production requirements.txt file:

Notice that you use the same -r switch to install the production requirements.txt file. Now, in your development environment, you only have to run this single command to install all requirements:

Because requirements_dev.txt contains the -r requirements.txt line, you’ll install not only pytest but also the pinned requirements of requirements.txt. In a production environment, it’s sufficient to install the production requirements only:

With this command, you install the requirements listed in requirements.txt. In contrast to your development environment, your production environment won’t have pytest installed.

Freezing Requirements for Production

You created the production and development requirement files and added them to source control. These files use flexible version specifiers to ensure that you leverage bug fixes published by your dependencies. You’ve also tested your application and are now ready to deploy it to production.

You know that all the tests pass and the application works with the dependencies that you used in your development process, so you probably want to ensure that you deploy identical versions of dependencies to production.

The current version specifiers don’t guarantee that the identical versions will be deployed to production, so you want to freeze the production requirements before releasing your project.

After you’ve finished development with your current requirements, a workflow to create a new release of your current project can look like this:

Step Command Explanation
1 pytest Run your tests and verify that your code is working properly.
2 pip install -U -r requirements.txt Upgrade your requirements to versions that match the constraints in your requirements.txt file.
3 pytest Run your tests and consider downgrading any dependency that introduced errors to your code.
4 pip freeze > requirements_lock.txt Once the project works correctly, freeze the dependencies into a requirements_lock.txt file.

With a workflow like this, the requirements_lock.txt file will contain exact version specifiers and can be used to replicate your environment. You’ve ensured that when your users install the packages listed in requirements_lock.txt into their own environments, they’ll be using the versions that you intend them to use.

Freezing your requirements is an important step to ensure that your Python project works the same way for your users in their environments as it did in yours.

Uninstalling Packages With pip

Once in a while, you’ll have to uninstall a package. Either you found a better library to replace it, or it’s something that you don’t need. Uninstalling packages can be a bit tricky.

Notice that when you installed requests, you got pip to install other dependencies too. The more packages you install, the bigger the chance that multiple packages depend on the same dependency. This is where the show command in pip comes in handy.

Before you uninstall a package, make sure to run the show command for that package:

Notice the last two fields, Requires and Required-by. The show command tells you that requests requires certifi, idna, charset-normalizer, and urllib3. You probably want to uninstall those too. Notice that requests isn’t required by any other package. So it’s safe to uninstall it.

You should run the show command against all of the requests dependencies to ensure that no other libraries also depend on them. Once you understand the dependency order of the packages that you want to uninstall, then you can remove them using the uninstall command:

The uninstall command shows you the files that will be removed and asks for confirmation. If you’re sure that you want to remove the package because you’ve checked its dependencies and know that nothing else is using it, then you can pass a -y switch to suppress the file list and confirmation dialog:

Here you uninstall urllib3. Using the -y switch, you suppress the confirmation dialog asking you if you want to uninstall this package.

In a single call, you can specify all the packages that you want to uninstall:

You can pass in multiple packages to the pip uninstall command. If you didn’t add any additional switches, then you’d need to confirm uninstalling each package. By passing the -y switch, you can uninstall them all without any confirmation dialog.

You can also uninstall all the packages listed in a requirements file by providing the -r <requirements file> option. This command will prompt a confirmation request for each package, but you can suppress it with the -y switch:

Remember to always check the dependencies of packages that you want to uninstall. You probably want to uninstall all dependencies, but uninstalling a package used by others will break your working environment, and your project may not work correctly anymore.

The pip uninstall command can be really helpful when you need to uninstall a package from your system Python installation. Using pip uninstall is a good way to declutter your system if you accidentally install a package system-wide.

Exploring Alternatives to pip

The Python community provides excellent tools and libraries for you to use beyond pip. These include alternatives to pip that try to simplify and improve package management.

Here are some other package management tools that are available for Python:

Tool Description
Conda Conda is a package, dependency, and environment manager for many languages, including Python. It comes from Anaconda, which started as a data science package for Python. Consequently, it’s widely used for data science and machine learning applications. Conda operates its own index to host compatible packages.
Poetry Poetry will look very familiar to you if you’re coming from JavaScript and npm. Poetry goes beyond package management, helping you build distributions for your applications and libraries and deploying them to PyPI.
Pipenv Pipenv is another package management tool that merges virtual environment and package management in a single tool. Pipenv: A Guide to the New Python Packaging Tool is a great place to start learning about Pipenv and its approach to package management.
uv uv stands for universal, reflecting uv’s broad applicability planned for the future. The tool is being advertised as a Cargo for Python, aiming to become a drop-in replacement for pip. The uv project was unveiled in February 2024 by Astral, the company founded by Charlie Marsh, who gained fame after authoring Ruff.

Only pip comes bundled in the standard Python installation. If you want to use any alternatives listed above, then you have to follow the installation guides in their documentation. With so many options, you’re sure to find the right tools for your programming journey!

Conclusion

Many Python projects use the pip package manager to manage their dependencies. It’s included with the Python installer, and it’s an essential tool for dependency management in Python.

In this tutorial, you learned how to:

  • Set up and run pip in your working environment
  • Fix common errors related to working with pip
  • Install and uninstall packages with pip
  • Define requirements for your projects and applications
  • Pin dependencies in requirements files

In addition, you’ve learned about the importance of keeping dependencies up to date and alternatives to pip that can help you manage those dependencies.

By taking a closer look at pip, you’ve explored an essential tool in your Python development workflows. With pip, you can install and manage any additional packages that you find on PyPI. You can use external packages from other developers as requirements and concentrate on the code that makes your project unique.

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: A Beginner’s Guide to pip



Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button