Check out our new VS Code extension for experiment tracking and model development
The most important concept in MLEM is MLEM Object. Basically, MLEM is a library to create, manage and use different MLEM Objects, such as models, data and other types you can read about below.
For example, when you use mlem.api.save()
, you create a MLEM Object from a
supported Python structure. MLEM Objects can also be created with
mlem declare
.
MLEM Objects are saved as special metafiles in YAML format with the .mlem
extension. These may or may not have artifacts (other files or directories)
associated.
Typically, if MLEM Object has only one artifact, it will have the same file
name without .mlem
extension, for example model.mlem
and model
, or
data.csv
and data.csv.mlem
.
If MLEM Object have multiple artifacts, they will be stored in a directory
with the same name, for example model.mlem
+ model/data.pkl
+
model/data2.pkl
.
From a developer's perspective, MLEM Objects are instances of one of the
subclasses of MlemObject
class. MLEM is using extended
pydantic functionality to save and load
them from files.
You can get MlemObject
instance if you use load_meta
API method instead of
simple load
.
See also MLEM Object API
Each MLEM Object has an object_type
field which determines the type of the
object. Specific types may have additional properties, but all MLEM Objects have
the following fields:
description
- for storing user-provided descriptionparams
- arbitrary object with additional parameterslabels
- list of string labelslocation
- if the object is loaded, information about where it came fromYou can check out what methods MLEM Objects have in API Reference
Here are all the builtin MLEM Object types
Model and Data are special types that can have artifacts, so they have two additional fields:
artifacts
- a string-to-artifacts mapping, where artifact is an instance of
Artifact
which represents a file
stored somewhere (local/cloud/dvc cache etc)requirements
- a list of
Requirement
which are needed to use
that object in runtimeRepresents an ML model, but can be generalized to any model or even any
"function" or any "transformation", thanks to callable
ModelType.
Base class: mlem.core.objects.MlemModel
Fields (in addition to inherited):
model_type
(lazy) - ModelType,
which is polymorphic and holds metadata about model's framework, methods and
io.Represent data, which can be used as an input to one of Model's methods.
Base class: mlem.core.objects.MlemData
Fields (in addition to inherited):
reader
(lazy) - DataReader - how
to read saved files and resulting dataset metadatadata_type
(transient) - DataType
with dataset value and metadata (available once data is read)Represents a link (pointer) to another MLEM Object. More on that here
Base class: mlem.core.objects.MlemLink
Fields (in addition to inherited):
path
- path to MLEM Objectproject
- location of MLEM Project with referenced objectrev
- revision of the objectlink_type
- type of the referenced objectSome of the MLEM ABCs
are also MLEM Objects.