zimscraperlib.zim.creator

ZIM Creator helper

Convenient subclass of libzim.writer.Creator with: - easier configuration of commonly set props during init - start/stop methods to bypass the contextmanager - method to create an entry directly from args - direct method to add redirects without title - prevent exeption on double call to close()

Convenient subclasses of libzim.writer.Item with: - metadata set on initialization - metadata stored on object Sister subclass StaticItem (inheriting from it) with: - content stored on object - can be used to store a filepath and content read from it (not stored)

Classes:

• Creator –

  libzim.writer.Creator subclass

Functions:

• mimetype_for –

  mimetype as provided or guessed from fpath, path or content

Attributes:

• DUPLICATE_EXC_STR –

DUPLICATE_EXC_STR module-attribute

DUPLICATE_EXC_STR = compile(
    "^Impossible to add(.+)dirent\\'s title to add is(.+)existing dirent's title is(.+)",
    MULTILINE | DOTALL,
)

Creator

Creator(
    filename: Path,
    main_path: str,
    compression: str | None = None,
    *,
    workaround_nocancel: bool | None = True,
    ignore_duplicates: bool | None = False,
)

Bases: Creator

libzim.writer.Creator subclass

Note: due to the lack of a cancel() method in the libzim itself, it is not possible to stop a zim creation process. Should an error occur in your code, a Zim file with up-to-that-moment content will be created at destination.

To prevent this (creating an unwanted ZIM file) from happening, a workaround is in place. It prevents the libzim from finishing its process. While it results in no ZIM file being created, it also results in the zim temp folder to be left on disk and very frequently leads to a segmentation fault at garbage collection (on exit mostly).

Meaning you should exit right after an exception in your code (during zim creation) Use workaround_nocancel=False to disable the workaround.

By default, all metadata are validated for compliance with openZIM specification and conventions. Set metdata.APPLY_RECOMMENDATIONS to False to disable this validation

(you canstill do checks manually with the validation methods or your own logic).

Methods:

• add_item –

  Add a libzim.writer.Item

• add_item_for –

  Add a File or content at a specified path and get its path

• add_metadata –

  Really add the metadata to the ZIM, after ZIM creation has started.

• add_redirect –

  Add a redirect from path to target_path

• config_dev_metadata –

  Calls minimal set of mandatory metadata with default values for dev

• config_indexing –

  Toggle full-text indexing of entries

• config_metadata –

  Checks and prepare list of ZIM metadata

• finish –

  Triggers finalization of ZIM creation and create final ZIM file.

• start –

  Start creator operation at libzim level

Attributes:

• can_finish –
• ignore_duplicates –
• workaround_nocancel –

Source code in src/zimscraperlib/zim/creator.py

def __init__(
    self,
    filename: pathlib.Path,
    main_path: str,
    compression: str | None = None,
    *,
    workaround_nocancel: bool | None = True,
    ignore_duplicates: bool | None = False,
):
    super().__init__(filename=filename)
    self._metadata: dict[str, AnyMetadata] = {}
    self.__indexing_configured = False
    self.__indexing_value: bool = False
    self.can_finish = True

    self.set_mainpath(main_path)

    if compression:
        self.config_compression(
            getattr(libzim.writer.Compression, compression.lower())
        )

    self.workaround_nocancel = workaround_nocancel
    self.ignore_duplicates = ignore_duplicates

can_finish instance-attribute

can_finish = True

ignore_duplicates instance-attribute

ignore_duplicates = ignore_duplicates

workaround_nocancel instance-attribute

workaround_nocancel = workaround_nocancel

add_item

add_item(
    item: Item,
    *,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
)

Add a libzim.writer.Item

callback: either a single callable or a tuple containing the callable as first element then the arguments to pass to the callable. Note: you must not include the item itself in those arguments.

Source code in src/zimscraperlib/zim/creator.py

def add_item(  # pyright: ignore[reportIncompatibleMethodOverride]
    self,
    item: libzim.writer.Item,
    *,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
):
    """Add a libzim.writer.Item

    callback: either a single callable or a tuple containing the callable
    as first element then the arguments to pass to the callable.
    Note: you must __not__ include the item itself in those arguments."""
    if isinstance(callbacks, Callback):
        callbacks = [callbacks]
    elif callbacks is None:
        callbacks = []

    for callback in callbacks:
        if callback.callable:
            weakref.finalize(
                item, callback.func, *callback.get_args(), **callback.get_kwargs()
            )

    duplicate_ok = duplicate_ok or self.ignore_duplicates
    try:
        try:
            super().add_item(item)
        except RuntimeError as exc:
            if not DUPLICATE_EXC_STR.match(str(exc)) or not duplicate_ok:
                raise exc
    except Exception:
        if self.workaround_nocancel:
            self.can_finish = False  # pragma: no cover
        raise

add_item_for

add_item_for(
    path: str,
    title: str | None = None,
    *,
    fpath: Path | None = None,
    content: bytes | str | None = None,
    mimetype: str | None = None,
    is_front: bool | None = None,
    should_compress: bool | None = None,
    delete_fpath: bool | None = False,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
    index_data: IndexData | None = None,
    auto_index: bool = True,
)

Add a File or content at a specified path and get its path

mimetype is retrieved from content (magic) if not specified if magic finds it to be text/*, guesses the mimetype from the source filename (if using a file) or the destination path

is_front: whether this Item is a FRONT_ARTICLE or not. Those are considered user-facing Entries and thus part of suggestion, random, etc. Default (not set) sets it based on mimetype (see constants for list)

should_compress: specify whether this Item should be compressed or not. Default (not set) lets the libzim decide (based on mimetype)

Content specified either from content (str|bytes) arg or read from fpath Source file can be safely deleted after this call.

callback: see add_item()

Source code in src/zimscraperlib/zim/creator.py

def add_item_for(
    self,
    path: str,
    title: str | None = None,
    *,
    fpath: pathlib.Path | None = None,
    content: bytes | str | None = None,
    mimetype: str | None = None,
    is_front: bool | None = None,
    should_compress: bool | None = None,
    delete_fpath: bool | None = False,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
    index_data: IndexData | None = None,
    auto_index: bool = True,
):
    """Add a File or content at a specified path and get its path

    mimetype is retrieved from content (magic) if not specified
    if magic finds it to be text/*, guesses the mimetype from the source
    filename (if using a file) or the destination path

    is_front: whether this Item is a FRONT_ARTICLE or not. Those are considered
    user-facing Entries and thus part of suggestion, random, etc.
    Default (not set) sets it based on mimetype (see constants for list)

    should_compress: specify whether this Item should be compressed or not.
    Default (not set) lets the libzim decide (based on mimetype)

    Content specified either from content (str|bytes) arg or read from fpath
    Source file can be safely deleted after this call.

    callback: see add_item()"""
    if fpath is None and content is None:
        raise ValueError("One of fpath or content is required")

    if isinstance(callbacks, Callback):
        callbacks = [callbacks]
    elif callbacks is None:
        callbacks = []

    mimetype = mimetype_for(
        path=path, content=content, fpath=fpath, mimetype=mimetype
    )

    if is_front is None:
        is_front = mimetype in FRONT_ARTICLE_MIMETYPES
    hints: dict[libzim.writer.Hint, int] = {
        libzim.writer.Hint.FRONT_ARTICLE: is_front
    }

    if should_compress is not None:
        hints[libzim.writer.Hint.COMPRESS] = should_compress

    if delete_fpath and fpath:
        callbacks.append(Callback(func=delete_callback, args=(fpath,)))

    self.add_item(
        StaticItem(
            path=path,
            title=title,
            mimetype=mimetype,
            filepath=fpath,
            hints=hints,
            content=content,
            index_data=index_data,
            auto_index=auto_index,
        ),
        callbacks=callbacks,
        duplicate_ok=duplicate_ok,
    )
    return path

add_metadata

add_metadata(value: AnyMetadata)

Really add the metadata to the ZIM, after ZIM creation has started.

You would probably prefer to use config_metadata methods to check metadata before starting the ZIM, ensure all mandatory metadata are set, and avoid duplicate metadata name.

Source code in src/zimscraperlib/zim/creator.py

def add_metadata(  # pyright: ignore[reportIncompatibleMethodOverride]
    self, value: AnyMetadata
):
    """Really add the metadata to the ZIM, after ZIM creation has started.

    You would probably prefer to use config_metadata methods to check metadata
    before starting the ZIM, ensure all mandatory metadata are set, and avoid
    duplicate metadata name.
    """

    super().add_metadata(value.name, value.libzim_value, value.mimetype)

add_redirect

add_redirect(
    path: str,
    target_path: str,
    title: str | None = "",
    *,
    is_front: bool | None = None,
    duplicate_ok: bool | None = None,
)

Add a redirect from path to target_path

title is optional. when set, the redirect itself can be found on suggestions (indexed) if considered FRONT_ARTICLE

Source code in src/zimscraperlib/zim/creator.py

def add_redirect(
    self,
    path: str,
    target_path: str,
    title: str | None = "",
    *,
    is_front: bool | None = None,
    duplicate_ok: bool | None = None,
):
    """Add a redirect from path to target_path

    title is optional. when set, the redirect itself
    can be found on suggestions (indexed) if considered FRONT_ARTICLE"""
    hints: dict[libzim.writer.Hint, int] = {}
    if is_front is not None:
        hints[libzim.writer.Hint.FRONT_ARTICLE] = bool(is_front)

    duplicate_ok = duplicate_ok or self.ignore_duplicates

    try:
        try:
            super().add_redirection(path, title or path, target_path, hints)
        except RuntimeError as exc:
            if not DUPLICATE_EXC_STR.match(str(exc)) or not duplicate_ok:
                raise exc
    except Exception:
        if self.workaround_nocancel:
            self.can_finish = False  # pragma: no cover
        raise

config_dev_metadata

config_dev_metadata(
    extra_metadata: AnyMetadata
    | list[AnyMetadata]
    | None = None,
)

Calls minimal set of mandatory metadata with default values for dev

Extra metadata can be passed, and they are not checked for proper key prefix

Source code in src/zimscraperlib/zim/creator.py

def config_dev_metadata(
    self,
    extra_metadata: AnyMetadata | list[AnyMetadata] | None = None,
):
    """Calls minimal set of mandatory metadata with default values for dev

    Extra metadata can be passed, and they are not checked for proper key prefix
    """
    return self.config_metadata(
        std_metadata=DEFAULT_DEV_ZIM_METADATA,
        extra_metadata=(
            [extra_metadata]
            if isinstance(extra_metadata, MetadataBase)
            else extra_metadata
        ),
        fail_on_missing_prefix_in_extras=False,
    )

config_indexing

config_indexing(
    indexing: bool, language: str | None = None
)

Toggle full-text indexing of entries

Uses Language metadata's value (or "") if not set.

Note: title indexing is always performed by libzim and cannot be disabled via this method; only the full-text index is toggled.

Source code in src/zimscraperlib/zim/creator.py

def config_indexing(
    self,
    indexing: bool,  # noqa: FBT001
    language: str | None = None,
):
    """Toggle full-text indexing of entries

    Uses Language metadata's value (or "") if not set.

    Note: title indexing is always performed by libzim and cannot be
    disabled via this method; only the full-text index is toggled."""
    language = language or self._get_first_language_metadata_value() or ""
    if indexing and not is_valid_iso_639_3(language):
        raise ValueError("Not a valid ISO-639-3 language code")
    super().config_indexing(indexing, language)
    self.__indexing_configured = True
    self.__indexing_value = indexing
    return self

config_metadata

config_metadata(
    std_metadata: StandardMetadataList | list[AnyMetadata],
    extra_metadata: list[AnyMetadata] | None = None,
    *,
    fail_on_missing_prefix_in_extras: bool = True,
)

Checks and prepare list of ZIM metadata

Checks ensure that metadata value can be converted to bytes, including all requirements of the ZIM specifications and optionally openZIM conventions.

Metadata are only kept in memory at this stage, not yet passed to libzim.

They will be passed to libzim / writen to the ZIM on creator.start().

Parameters:

• std_metadata (StandardMetadataList | list[AnyMetadata]) –

  standard metadata defined in the ZIM specifications. Prefer to use StandardMetadataList which ensure mandatory metadata are all set.

• extra_metadata (list[AnyMetadata] | None, default: None ) –

  a list of extra metadata (not standard).

• fail_on_missing_prefix_in_extras (bool, default: True ) –

  disable the default check which force the X- prefix on extra metadata name which is a convention to distinguish these extra metadata

Source code in src/zimscraperlib/zim/creator.py

def config_metadata(
    self,
    std_metadata: StandardMetadataList | list[AnyMetadata],
    extra_metadata: list[AnyMetadata] | None = None,
    *,
    fail_on_missing_prefix_in_extras: bool = True,
):
    """Checks and prepare list of ZIM metadata

    Checks ensure that metadata value can be converted to bytes, including all
    requirements of the ZIM specifications and optionally openZIM conventions.

    Metadata are only kept in memory at this stage, not yet passed to libzim.

    They will be passed to libzim / writen to the ZIM on creator.start().

    Arguments:
        std_metadata: standard metadata defined in the ZIM specifications.
            Prefer to use StandardMetadataList which ensure mandatory metadata are
            all set.
        extra_metadata: a list of extra metadata (not standard).
        fail_on_missing_prefix_in_extras: disable the default check which force the
            X- prefix on extra metadata name which is a convention to distinguish
            these extra metadata

    """
    for fail_on_missing_prefix, metadata in [
        (False, metadata)
        for metadata in (
            std_metadata.values()
            if isinstance(std_metadata, StandardMetadataList)
            else std_metadata
        )
    ] + [
        (fail_on_missing_prefix_in_extras, metadata)
        for metadata in extra_metadata or []
    ]:
        if fail_on_missing_prefix and not metadata.name.startswith("X-"):
            raise ValueError(
                f"Metadata key {metadata.name} does not starts with X- as expected"
            )
        # if metadata.name in self._metadata:
        #     raise ValueError(f"{metadata.name} cannot be defined twice")
        self._metadata[metadata.name] = metadata

    return self

finish

finish(
    _: type[BaseException] | None = None,
    __: BaseException | None = None,
    ___: TracebackType | None = None,
)

Triggers finalization of ZIM creation and create final ZIM file.

Source code in src/zimscraperlib/zim/creator.py

def finish(
    self,
    _: type[BaseException] | None = None,
    __: BaseException | None = None,
    ___: TracebackType | None = None,
):
    """Triggers finalization of ZIM creation and create final ZIM file."""
    if not getattr(self, "can_finish", False):
        return
    try:
        super().__exit__(None, None, None)
    except RuntimeError:
        pass

start

start()

Start creator operation at libzim level

Includes creating metadata, including validating mandatory ones are set, and configuring indexing.

Source code in src/zimscraperlib/zim/creator.py

def start(self):
    """Start creator operation at libzim level

    Includes creating metadata, including validating mandatory ones are set,
    and configuring indexing.
    """
    if logger.isEnabledFor(logging.DEBUG):  # pragma: no cover
        self._log_metadata()

    if not all(self._metadata.get(key) for key in MANDATORY_ZIM_METADATA_KEYS):
        missing_keys = [
            key
            for key in MANDATORY_ZIM_METADATA_KEYS
            if not self._metadata.get(key)
        ]
        raise ValueError(
            "Mandatory metadata are not all set. Missing metadata: "
            f"{','.join(missing_keys)}. You should prefer to use "
            "StandardMetadataList if possible."
        )

    if (
        language := self._get_first_language_metadata_value()
    ) and not self.__indexing_configured:
        self.config_indexing(True, language)

    ftindex_tag = f"_ftindex:{'yes' if self.__indexing_value else 'no'}"
    tags_metadata = self._metadata.get(TagsMetadata.meta_name)
    if isinstance(tags_metadata, TagsMetadata):
        if not any(
            re.sub(r"\s+", "", part).startswith("_ftindex:")
            for tag in tags_metadata.value
            for part in tag.split(";")
        ):
            tags_metadata.value.append(ftindex_tag)
            logger.debug(f"Metadata: Tags has been altered with '{ftindex_tag}'")
    else:
        self._metadata[TagsMetadata.meta_name] = TagsMetadata([ftindex_tag])
        logger.debug(f"Metadata: Tags has been set with '{ftindex_tag}'")

    super().__enter__()

    for metadata in self._metadata.values():
        if isinstance(metadata, IllustrationBasedMetadata):
            self.add_illustration(metadata.illustration_size, metadata.libzim_value)
        else:
            self.add_metadata(metadata)
    self._metadata.clear()

    return self

mimetype_for

mimetype_for(
    path: str,
    content: bytes | str | None = None,
    fpath: Path | None = None,
    mimetype: str | None = None,
) -> str | None

mimetype as provided or guessed from fpath, path or content

Source code in src/zimscraperlib/zim/creator.py

def mimetype_for(
    path: str,
    content: bytes | str | None = None,
    fpath: pathlib.Path | None = None,
    mimetype: str | None = None,
) -> str | None:
    """mimetype as provided or guessed from fpath, path or content"""
    if not mimetype:
        mimetype = (
            get_file_mimetype(fpath)
            if fpath
            else get_content_mimetype(content[:2048])
            if content
            else None
        )
        # try to guess more-defined mime if it's text
        if (
            not mimetype
            or mimetype == "application/octet-stream"
            or mimetype.startswith("text/")
        ):
            mimetype = get_mime_for_name(
                filename=fpath if fpath else path, fallback=mimetype, no_ext_to=mimetype
            )
    return mimetype