Edit on GitHub

Basic concepts

MLEM Objects

The most important concept in MLEM is MLEM Object. Basically, MLEM is a library to create, manage and use different MLEM Objects, such as models, data and other types you can read about below.

For example, when you use mlem.api.save(), you create a MLEM Object from a supported Python structure. MLEM Objects can also be created with mlem declare.

MLEM Objects are saved as special metafiles in YAML format with the .mlem extension. These may or may not have artifacts (other files or directories) associated.

Typically, if MLEM Object has only one artifact, it will have the same file name without .mlem extension, for example model.mlem and model, or data.csv and data.csv.mlem.

If MLEM Object have multiple artifacts, they will be stored in a directory with the same name, for example model.mlem + model/data.pkl + model/data2.pkl.

From a developer's perspective, MLEM Objects are instances of one of the subclasses of MlemObject class. MLEM is using extended pydantic functionality to save and load them from files.

You can get MlemObject instance if you use load_meta API method instead of simple load.

See also MLEM Object API

Common fields

Each MLEM Object has an object_type field which determines the type of the object. Specific types may have additional properties, but all MLEM Objects have the following fields:

  • params - arbitrary object with additional parameters
  • location - if the object is loaded, information about where it came from

You can check out what methods MLEM Objects have in API Reference

MLEM Object Types

Here are all the builtin MLEM Object types

Model and Data are special types that can have artifacts, so they have two additional fields:

  • artifacts - a string-to-artifacts mapping, where artifact is an instance of Artifact which represents a file stored somewhere (local/cloud/dvc cache etc)
  • requirements - a list of Requirement which are needed to use that object in runtime

Model

Represents an ML model, but can be generalized to any model or even any "function" or any "transformation", thanks to callable ModelType.

Base class: mlem.core.objects.MlemModel

Fields (in addition to inherited):

  • model_type (lazy) - ModelType, which is polymorphic and holds metadata about model's framework, methods and io.

Data

Represent data, which can be used as an input to one of Model's methods.

Base class: mlem.core.objects.MlemData

Fields (in addition to inherited):

  • reader (lazy) - DataReader - how to read saved files and resulting dataset metadata
  • data_type (transient) - DataType with dataset value and metadata (available once data is read)

Represents a link (pointer) to another MLEM Object. More on that here

Base class: mlem.core.objects.MlemLink

Fields (in addition to inherited):

  • path - path to MLEM Object
  • project - location of MLEM Project with referenced object
  • rev - revision of the object
  • link_type - type of the referenced object

Other types

Some of the MLEM ABCs are also MLEM Objects.

Content

🐛 Found an issue? Let us know! Or fix it:

Edit on GitHub

Have a question? Join our chat, we will help you:

Discord Chat