Check out our new VS Code extension for experiment tracking and model development
MLEM objects could be saved in different locations, such as Git repositories, cloud object buckets, local directories, etc. In this section, we will discuss how to work with remote Git repositories and cloud buckets, but the same logic extends to other locations.
In our examples MLEM projects will exist in those locations, although it's
required for mlem list
only. Other commands such as mlem clone
,
mlem apply
, and mlem.api.load()
will work as they should whether there is a
MLEM project or not.
Models and Data stored inside MLEM projects can be listed with mlem list.
Let's list objects inside a remote Git example repository, without the need of cloning it.
$ mlem list https://github.com/iterative/example-mlem-get-started \
--rev simple
Which gives us:
Builders:
- pip_config
Data:
- iris.csv
Models:
- rf
Deployments:
- myservice
Envs:
- staging
One can use URL addresses to load objects (models, data, etc.) from remote repositories directly:
from mlem.api import load
model = load(
"rf",
project="https://github.com/iterative/example-mlem-get-started",
rev="simple"
)
You can easily download the object to your local machine so as to use it later
using the clone
command.
$ mlem clone rf \
--project https://github.com/iterative/example-mlem-get-started \
--rev simple \
ml_model
⏳️ Loading meta from https://github.com/iterative/example-mlem-get-started/tree/simple/.mlem/model/rf.mlem
🐏 Cloning https://github.com/iterative/example-mlem-get-started/tree/simple/.mlem/model/rf.mlem
💾 Saving model to .mlem/model/ml_model.mlem
MLEM can also be used with any cloud/remote supported by fsspec, e.g. s3. This is useful in scenarios where objects are stored in a remote location without the need to version them (otherwise we strongly recommend to use DVC)
To do so, one can use paths with the corresponding file system protocol & path
such as s3://<bucket>/
$ mlem init s3://example-mlem-get-started
$ mlem clone rf s3://example-mlem-get-started/rf
⏳️ Loading meta from .mlem/model/rf.mlem
🐏 Cloning .mlem/model/rf.mlem
💾 Saving model to s3://example-mlem-get-started/.mlem/model/rf.mlem
This model can now be loaded via API or can be used in CLI commands as though it existed locally:
from mlem.api import load
model = load("rf", project="s3://example-mlem-get-started")
$ mlem apply rf \
--project s3://example-mlem-get-started \
test_x.csv --json
[1, 0, 2, 1, 1, 0, 1, 2, 1, 1, 2, 0, 0, 0, 0, 1, 2, 1, 1, 2, 0, 2, 0, 2, 2, 2, 2, 2, 0, 0, 0, 0, 1, 0, 0, 2, 1, 0]
Working with remote projects and objects have several different use-cases, some of which we covered above: