Managing requirements.txt files to reproduce environments

As of the last few months - uv is the only tool you need

https://docs.astral.sh/uv/

ohkays let me check on this

I tried using pipreqs with pip-compile

it's seriously changed the game

makes python feel super modern

Basically i want to create a requirements.txt file for my project but for some reason no matter what method i am trying i am just missing out on many dependencies

so there exists a command

uv pip compile requirements.in --universal --output-file requirements.txt

so you can do it two ways

you can sue uv add to add requirements in a way that compiles nicely

or you can copy your existing requirements.txt into a requiremements.in and compile them that way

Let me explain you the scenario
So i started building an application and I forgot to create any kind of requirements.txt file or requirements.in file .

Then I used tools like pip

pip freeze

kedro-datasets[pandas-csvdataset]
kedro-datasets[ibis]
kedro-kubeflow==0.8.0
ibis-framework==9.0.0
s3fs==2024.9.0
psycopg2-binary==2.9.9

you're on the right path. either you add your dependencies to a requirements.in, or to the [project.dependencies] table of pyproject.toml (we in Kedro are moving towards the latter)

ok but what is the standard way while developing any application . Like the best way I was aware of till now is to manually add the dependency along-with the version. But is that how people do it ? because there are high chances of people forgetting to add dependencies manually.
I want to know the workflow to manage dependencies either to requirements.in or pyproject.toml .

If you do uv add it does the hard bit for you

I.e. it will work out the highest compatible versions across all your packages

so you mean whenever we want to make use of a new python package in the application we simply do uv add and it takes care of logging the package along with version somewhere like requirements.txt and we never miss out on anything

And it does a pip compile!

And it's written in rust so it's much faster!

It's a game changer

just for my understanding pip compile is needed if we want to publish our project as a python project and people can simply pip install and use it in their projects ?

Pip compile is used to make sure packages are compatible with each other

So if 2 packages have the common dependency(say X), pip compile will find the version of X which is compatible for both the packages?

Exactly!

But now you can just do uv add and uv remove as you work and forget about the fiddly part

I was just comparing uv with poetry looks like poetry also does the same thing . Any thoughts ?

Id encourage you to read through their docs it will make lots of sense

Poetry is pure python and doesn't follow PEP standards

Uv is written in rust so it's much much faster

The good bits of poetry are now in uv

Amazing 🙂

I got a requirements.txt now, how do uv to make use of that to setup the environment

So you can do uv pip install -r requirements.txt but the more modern way is to do uv add as you go along

getting into an issue
Resolved 147 packages in 922ms
error: Failed to prepare distributions
Caused by: Failed to fetch wheel: psycopg2-binary==2.9.9
Caused by: Build backend failed to determine requirements with `build_wheel()` (exit status: 1)

even though i removed this dependency from requirements file it still throws it

So that's an example of something that's not compatible with the others

Oh I bet you're talking to different python installations . If you type which python what shows up

which python nothing shows up

So basically what i was really doing is I am trying to setup a kedro project in my new laptop. I simply cloned the project from github and ran

uv pip compile requirements.in --universal --output-file requirements.txt

[project]
name = "warehouse-pipeline"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []

version = 1
requires-python = ">=3.13"

[[package]]
name = "warehouse-pipeline"
version = "0.1.0"
source = { virtual = "." }

<@U03R8FW4HUZ>

I think i was doing some mistake .

1. I created a venv using uv venv --python 3.9 in the kedro project folder.
2. Activated the environment
3. uv init Now I have proper python version declared in pyproject.toml earlier for some reason it took default latest python
4. Now i did uv pip install -r requirements.txt

Great

Some quick Questions

1. since the pyproject.toml introduced by uv init is way different than what was there earlier by kedro init . It won't create any issue
2. Also is there a way to completely get rid of requirements file and push all dependencies to pyproject.toml only

So 2 is the aim

You actually may want to combine the two pyproject tomls, its important if you want to package your kedro project but you can prob get away without if you don't

so yeaah no need to package it as it will be running like an application instead of getting used by other applications

But i want to move all dependencies to pyproject from requirements.txt and get rid of it.

One way is to copy paste the dependencies - naive way That i am aware of 😛

If there aren't too many spending time doing uv add will compile them too

There's probably a bulk way of you read the docs

I'm also not 100% sure what uv sync does but it may be what you're looking for

uv sync actually removes all the installed packages from requirements.txt becuase it actually trying to sync with dependencies defined in pyproject.toml or uv.lock which is empty for me currently 😛

@chatgpt please summarise this thread 😅

sorry for such big threads but i learned fundamentals of using a dependency management tool like uv and later explored poetry as well.

haha no worries! it was not a criticism. I'm happy that we helped you untangle some of these packaging tricks. what would be your main takeaways?

1. Basic pip installations in conda is not that powerful as compared to using tools like poetry and uv. Mind it Conda is too heavy an ecosystem as well 😂
2. Whether poetry or uv, both takes care of the virtual environment for your project, no need to rely on other tools like conda , pyenv for that.
3. Installing a dependency is lot easier by using commands like poetry add or uv add , they both are backed by powerful dependency resolution algorithms.
4. The best thing is they will maintain a list of dependencies for you in pyproject.toml
5. Both of them offers a nice way to lock the exact versions of dependencies and the transitive dependencies in some kind of lock file, This lock file can be checked into VCS and can be later used by developers to reproduce an exact similar environment in which the application or library was developed.
6. The above points are mainly from dependency management point of view and we get rid of tools like pip , pipreqs, pip-tools . One thing to note here is tools like poetry and uv does not just do dependency resolution but it manages for you and it does it really well.
7. At the end it makes the life easier to build and publish a package if needed.
8. Uv is superfast in everything .

Thanks for your help mate 🥂

amazing summary, thanks a lot!!

I think I heard 'blog post', was that just me? 👀

when uv 0.5 is out? 🙃

btw uv is on its way to surpass Poetry in weekly downloads 😱 https://clickpy.clickhouse.com/dashboard/uv