Project Overview¶
Command line interface¶
The command line interface is the recommended interface for working with venvstacks
:
$ venvstacks --help
Usage: venvstacks [OPTIONS] COMMAND [ARGS]...
Lock, build, and publish Python virtual environment stacks.
╭─ Options ───────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────╮
│ build Build (/lock/publish) Python virtual environment stacks. │
│ local-export Export layer environments for Python virtual environment stacks. │
│ lock Lock layer requirements for Python virtual environment stacks. │
│ publish Publish layer archives for Python virtual environment stacks. │
╰─────────────────────────────────────────────────────────────────────────────────╯
Working with environment stacks¶
Defining environment stacks¶
The environment layers to be published are defined in a venvstacks.toml
stack specification,
with a separate array of tables for each kind of layer definition.
For example, the following specification defines a pair of applications which use scikit-learn as a shared framework layer with numpy preinstalled in the runtime layer, all running in a controlled Python 3.11 base runtime:
[[runtimes]]
name = "cpython@3.11"
fully_versioned_name = "cpython@3.11.10"
requirements = [
"numpy",
]
[[frameworks]]
name = "sklearn"
runtime = "cpython@3.11"
requirements = [
"scikit-learn",
]
[[applications]]
name = "classification-demo"
launch_module = "launch_modules/sklearn_classification.py"
frameworks = ["sklearn"]
requirements = [
"scikit-learn",
]
[[applications]]
name = "clustering-demo"
launch_module = "launch_modules/sklearn_clustering.py"
frameworks = ["sklearn"]
requirements = [
"scikit-learn",
]
Locking environment stacks¶
$ venvstacks lock sklearn_demo/venvstacks.toml
The lock
subcommand takes the defined layer requirements from the specification,
and uses them to perform a complete combined resolution of all of the environment stacks
that ensures the different layers can be published separately,
but still work as expected when deployed to a target system.
The locking mechanism is defined such that only changes to modules a given layer uses from lower layers affect them, rather than upper layers needing to be rebuilt for every change to a lower layer.
Building environment stacks¶
$ venvstacks build sklearn_demo/venvstacks.toml
The build
subcommand performs the step of converting the layer specifications
and their locked requirements into a working Python environment
(either a base runtime environment,
or a layered virtual environment based on one of the defined runtime environments).
If the environments have not already been explicitly locked,
the build step will lock them as necessary.
This command is also a “build pipeline” command that allows locking, building, and publishing to be performed in a single step (see the command line help for details).
Publishing environment layer archives¶
$ venvstacks publish --tag-outputs --output-dir demo_artifacts sklearn_demo/venvstacks.toml
Once the environments have been successfully built,
the publish
command allows each layer to be converted to a separate
reproducible <https://reproducible-builds.org/>`__ binary archive suitable
for transferring to another system, unpacking, and using the unpacked environments
to run the included applications (needing only a small post-installation step using
a Python script embedded in the built layer archives to correctly relink the deployed
environments with each other in their deployed location on the target system).
Metadata regarding the layer definitions and the published artifacts is published
alongside the published archives (to demo_artifacts/__venvstacks__/
in the given example).
This metadata captures both input details (such as the hashes of the locked requirements
and the included launch modules) and output details
(such as the exact size and exact hash of the built layer archive).
Locally exporting environment stacks¶
$ venvstacks local-export --output-dir demo_export sklearn_demo/venvstacks.toml
Given that even considering the use of venvstacks
implies that some layer archives may be of
significant size (a fully built PyTorch archive weighs in at multiple gigabytes,
for example), packing and unpacking the layer archives can take a substantial amount of time.
To avoid that overhead when iterating on layer definitions and launch module details,
the local-export
subcommand allows the built environments to be copied to a different
location on the same system, with most of the same filtering steps applied as would be
applied when performing the archive pack-and-unpack steps (the omissions are details
related to reproducible builds, like clamping the maximum file modification times to known values).
Locally exporting environments produces much of the same metadata as publishing layer archives, but the details related specifically to the published archive (such as its size and expected contents hash) are necessarily omitted.
Contributing to venvstacks
development¶
venvstacks
is MIT Licensed and developed on GitHub.
If you have a suitable use case,
the easiest way to contribute to venvstacks
development is just to try it out,
and let us know how that goes. What did you like, what did you dislike, what just plain broke?
If anything does break, then please open an issue (if the problem hasn’t already been reported). If you’re not sure if some behaviour is a bug or not, or would just like to provide general feedback rather than file specific issues or suggestions, the following Discord channels are the best way to get directly in touch with the developers:
Discuss
venvstacks
in general in the#venvstacks
channel on the PyPA Discord Server.Discuss the use of
venvstacks
in LM Studio in the#dev-chat
channel on the LM Studio Discord Server.
The “Packaging” category on discuss.python.org is also a reasonable place to provide feedback.
For additional information, consult the developer documentation