Skip to content

zimscraperlib.zim

ZIM file creation tools

zim.creator: create files by manually adding each article zim.filesystem: zimwriterfs-like creation from a build folder zim.providers: contentProvider for serving libzim with data zim.items: item to add to creator zim.archive: read ZIM files, accessing or searching its content

Modules:

  • archive –

    ZIM Archive helper

  • creator –

    ZIM Creator helper

  • dedup –
  • filesystem –

    zimwriterfs-like tools to convert a build folder into a ZIM

  • indexing –

    Special item with customized index data and helper classes

  • items –

    libzim Item helpers

  • metadata –
  • providers –

    libzim Providers accepting a ref arg to keep it away from garbage collection

Classes:

  • Archive –
  • Creator –

    libzim.writer.Creator subclass

  • FileLikeProvider –

    Provider referrencing a file-like object

  • FileProvider –
  • Item –

    libzim.writer.Item returning props for path/title/mimetype

  • StaticItem –

    scraperlib Item with auto contentProvider from content or filepath

  • StringProvider –
  • URLItem –

    StaticItem to automatically fetch and feed an URL resource

  • URLProvider –

    Provider downloading content as it is consumed by the libzim

Functions:

  • make_zim_file –

    Creates a zimwriterfs-like ZIM file at {fpath} from {build_dir}

Archive

Bases: Archive

Methods:

Attributes:

counters property 

counters: CounterMap

metadata property 

metadata: dict[str, str]

key: value for all non-illustration metadata listed in .metadata_keys

tags property 

tags

get_content 

get_content(path: str) -> bytes

Actual content from a path

Source code in src/zimscraperlib/zim/archive.py 
68
69
70
def get_content(self, path: str) -> bytes:
    """Actual content from a path"""
    return bytes(self.get_item(path).content)

get_entry_by_id 

get_entry_by_id(id_: int) -> Entry

Entry from its Id in ZIM

Source code in src/zimscraperlib/zim/archive.py 
60
61
62
def get_entry_by_id(self, id_: int) -> libzim.reader.Entry:
    """Entry from its Id in ZIM"""
    return self._get_entry_by_id(id_)

get_item 

get_item(path: str) -> Item

Item from a path

Source code in src/zimscraperlib/zim/archive.py 
64
65
66
def get_item(self, path: str) -> libzim.reader.Item:
    """Item from a path"""
    return self.get_entry_by_path(path).get_item()

get_search_results 

get_search_results(
    query: str, start: int = 0, end: int | None = None
) -> Iterable[str]

paths iterator over search results for query

Source code in src/zimscraperlib/zim/archive.py 
86
87
88
89
90
91
92
93
94
95
def get_search_results(
    self, query: str, start: int = 0, end: int | None = None
) -> Iterable[str]:
    """paths iterator over search results for query"""
    search = libzim.search.Searcher(self).search(
        libzim.search.Query().set_query(query)
    )
    if end is None:
        end = search.getEstimatedMatches()
    return search.getResults(start, end)

get_search_results_count 

get_search_results_count(query: str) -> int

Estimated number of search results for query

Source code in src/zimscraperlib/zim/archive.py 
 97
 98
 99
100
101
102
def get_search_results_count(self, query: str) -> int:
    """Estimated number of search results for query"""
    search = libzim.search.Searcher(self).search(
        libzim.search.Query().set_query(query)
    )
    return search.getEstimatedMatches()

get_suggestions 

get_suggestions(
    query: str, start: int = 0, end: int | None = None
) -> Iterable[str]

paths iterator over suggestion matches for query

Source code in src/zimscraperlib/zim/archive.py 
72
73
74
75
76
77
78
79
def get_suggestions(
    self, query: str, start: int = 0, end: int | None = None
) -> Iterable[str]:
    """paths iterator over suggestion matches for query"""
    suggestion = libzim.suggestion.SuggestionSearcher(self).suggest(query)
    if end is None:
        end = suggestion.getEstimatedMatches()
    return suggestion.getResults(start, end)

get_suggestions_count 

get_suggestions_count(query: str) -> int

Estimated number of suggestion matches for query

Source code in src/zimscraperlib/zim/archive.py 
81
82
83
84
def get_suggestions_count(self, query: str) -> int:
    """Estimated number of suggestion matches for query"""
    suggestion = libzim.suggestion.SuggestionSearcher(self).suggest(query)
    return suggestion.getEstimatedMatches()

get_tags 

get_tags(*, libkiwix: bool = False) -> list[str]

List of ZIM tags, optionnaly expanded with libkiwix's hints

Source code in src/zimscraperlib/zim/archive.py 
44
45
46
47
48
49
50
51
52
53
54
def get_tags(self, *, libkiwix: bool = False) -> list[str]:
    """List of ZIM tags, optionnaly expanded with libkiwix's hints"""
    try:
        tags_meta = self.get_text_metadata("Tags")
    except RuntimeError:  # pragma: no cover
        tags_meta = ""

    if libkiwix:
        return convertTags(tags_meta)

    return tags_meta.split(";")

get_text_metadata 

get_text_metadata(name: str) -> str

Decoded value of a text metadata

Source code in src/zimscraperlib/zim/archive.py 
56
57
58
def get_text_metadata(self, name: str) -> str:
    """Decoded value of a text metadata"""
    return super().get_metadata(name).decode("UTF-8")

Creator 

Creator(
    filename: Path,
    main_path: str,
    compression: str | None = None,
    *,
    workaround_nocancel: bool | None = True,
    ignore_duplicates: bool | None = False,
)

Bases: Creator

libzim.writer.Creator subclass

Note: due to the lack of a cancel() method in the libzim itself, it is not possible to stop a zim creation process. Should an error occur in your code, a Zim file with up-to-that-moment content will be created at destination.

To prevent this (creating an unwanted ZIM file) from happening, a workaround is in place. It prevents the libzim from finishing its process. While it results in no ZIM file being created, it also results in the zim temp folder to be left on disk and very frequently leads to a segmentation fault at garbage collection (on exit mostly).

Meaning you should exit right after an exception in your code (during zim creation) Use workaround_nocancel=False to disable the workaround.

By default, all metadata are validated for compliance with openZIM specification and conventions. Set metdata.APPLY_RECOMMENDATIONS to False to disable this validation

(you canstill do checks manually with the validation methods or your own logic).

Methods:

  • add_item –

    Add a libzim.writer.Item

  • add_item_for –

    Add a File or content at a specified path and get its path

  • add_metadata –

    Really add the metadata to the ZIM, after ZIM creation has started.

  • add_redirect –

    Add a redirect from path to target_path

  • config_dev_metadata –

    Calls minimal set of mandatory metadata with default values for dev

  • config_indexing –

    Toggle full-text indexing of entries

  • config_metadata –

    Checks and prepare list of ZIM metadata

  • finish –

    Triggers finalization of ZIM creation and create final ZIM file.

  • start –

    Start creator operation at libzim level

Attributes:

Source code in src/zimscraperlib/zim/creator.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
def __init__(
    self,
    filename: pathlib.Path,
    main_path: str,
    compression: str | None = None,
    *,
    workaround_nocancel: bool | None = True,
    ignore_duplicates: bool | None = False,
):
    super().__init__(filename=filename)
    self._metadata: dict[str, AnyMetadata] = {}
    self.__indexing_configured = False
    self.__indexing_value: bool = False
    self.can_finish = True

    self.set_mainpath(main_path)

    if compression:
        self.config_compression(
            getattr(libzim.writer.Compression, compression.lower())
        )

    self.workaround_nocancel = workaround_nocancel
    self.ignore_duplicates = ignore_duplicates

can_finish instance-attribute 

can_finish = True

ignore_duplicates instance-attribute 

ignore_duplicates = ignore_duplicates

workaround_nocancel instance-attribute 

workaround_nocancel = workaround_nocancel

add_item 

add_item(
    item: Item,
    *,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
)

Add a libzim.writer.Item

callback: either a single callable or a tuple containing the callable as first element then the arguments to pass to the callable. Note: you must not include the item itself in those arguments.

Source code in src/zimscraperlib/zim/creator.py 
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
def add_item(  # pyright: ignore[reportIncompatibleMethodOverride]
    self,
    item: libzim.writer.Item,
    *,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
):
    """Add a libzim.writer.Item

    callback: either a single callable or a tuple containing the callable
    as first element then the arguments to pass to the callable.
    Note: you must __not__ include the item itself in those arguments."""
    if isinstance(callbacks, Callback):
        callbacks = [callbacks]
    elif callbacks is None:
        callbacks = []

    for callback in callbacks:
        if callback.callable:
            weakref.finalize(
                item, callback.func, *callback.get_args(), **callback.get_kwargs()
            )

    duplicate_ok = duplicate_ok or self.ignore_duplicates
    try:
        try:
            super().add_item(item)
        except RuntimeError as exc:
            if not DUPLICATE_EXC_STR.match(str(exc)) or not duplicate_ok:
                raise exc
    except Exception:
        if self.workaround_nocancel:
            self.can_finish = False  # pragma: no cover
        raise

add_item_for 

add_item_for(
    path: str,
    title: str | None = None,
    *,
    fpath: Path | None = None,
    content: bytes | str | None = None,
    mimetype: str | None = None,
    is_front: bool | None = None,
    should_compress: bool | None = None,
    delete_fpath: bool | None = False,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
    index_data: IndexData | None = None,
    auto_index: bool = True,
)

Add a File or content at a specified path and get its path

mimetype is retrieved from content (magic) if not specified if magic finds it to be text/*, guesses the mimetype from the source filename (if using a file) or the destination path

is_front: whether this Item is a FRONT_ARTICLE or not. Those are considered user-facing Entries and thus part of suggestion, random, etc. Default (not set) sets it based on mimetype (see constants for list)

should_compress: specify whether this Item should be compressed or not. Default (not set) lets the libzim decide (based on mimetype)

Content specified either from content (str|bytes) arg or read from fpath Source file can be safely deleted after this call.

callback: see add_item()

Source code in src/zimscraperlib/zim/creator.py 
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
def add_item_for(
    self,
    path: str,
    title: str | None = None,
    *,
    fpath: pathlib.Path | None = None,
    content: bytes | str | None = None,
    mimetype: str | None = None,
    is_front: bool | None = None,
    should_compress: bool | None = None,
    delete_fpath: bool | None = False,
    duplicate_ok: bool | None = None,
    callbacks: list[Callback] | Callback | None = None,
    index_data: IndexData | None = None,
    auto_index: bool = True,
):
    """Add a File or content at a specified path and get its path

    mimetype is retrieved from content (magic) if not specified
    if magic finds it to be text/*, guesses the mimetype from the source
    filename (if using a file) or the destination path

    is_front: whether this Item is a FRONT_ARTICLE or not. Those are considered
    user-facing Entries and thus part of suggestion, random, etc.
    Default (not set) sets it based on mimetype (see constants for list)

    should_compress: specify whether this Item should be compressed or not.
    Default (not set) lets the libzim decide (based on mimetype)

    Content specified either from content (str|bytes) arg or read from fpath
    Source file can be safely deleted after this call.

    callback: see add_item()"""
    if fpath is None and content is None:
        raise ValueError("One of fpath or content is required")

    if isinstance(callbacks, Callback):
        callbacks = [callbacks]
    elif callbacks is None:
        callbacks = []

    mimetype = mimetype_for(
        path=path, content=content, fpath=fpath, mimetype=mimetype
    )

    if is_front is None:
        is_front = mimetype in FRONT_ARTICLE_MIMETYPES
    hints: dict[libzim.writer.Hint, int] = {
        libzim.writer.Hint.FRONT_ARTICLE: is_front
    }

    if should_compress is not None:
        hints[libzim.writer.Hint.COMPRESS] = should_compress

    if delete_fpath and fpath:
        callbacks.append(Callback(func=delete_callback, args=(fpath,)))

    self.add_item(
        StaticItem(
            path=path,
            title=title,
            mimetype=mimetype,
            filepath=fpath,
            hints=hints,
            content=content,
            index_data=index_data,
            auto_index=auto_index,
        ),
        callbacks=callbacks,
        duplicate_ok=duplicate_ok,
    )
    return path

add_metadata 

add_metadata(value: AnyMetadata)

Really add the metadata to the ZIM, after ZIM creation has started.

You would probably prefer to use config_metadata methods to check metadata before starting the ZIM, ensure all mandatory metadata are set, and avoid duplicate metadata name.

Source code in src/zimscraperlib/zim/creator.py 
254
255
256
257
258
259
260
261
262
263
264
def add_metadata(  # pyright: ignore[reportIncompatibleMethodOverride]
    self, value: AnyMetadata
):
    """Really add the metadata to the ZIM, after ZIM creation has started.

    You would probably prefer to use config_metadata methods to check metadata
    before starting the ZIM, ensure all mandatory metadata are set, and avoid
    duplicate metadata name.
    """

    super().add_metadata(value.name, value.libzim_value, value.mimetype)

add_redirect 

add_redirect(
    path: str,
    target_path: str,
    title: str | None = "",
    *,
    is_front: bool | None = None,
    duplicate_ok: bool | None = None,
)

Add a redirect from path to target_path

title is optional. when set, the redirect itself can be found on suggestions (indexed) if considered FRONT_ARTICLE

Source code in src/zimscraperlib/zim/creator.py 
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
def add_redirect(
    self,
    path: str,
    target_path: str,
    title: str | None = "",
    *,
    is_front: bool | None = None,
    duplicate_ok: bool | None = None,
):
    """Add a redirect from path to target_path

    title is optional. when set, the redirect itself
    can be found on suggestions (indexed) if considered FRONT_ARTICLE"""
    hints: dict[libzim.writer.Hint, int] = {}
    if is_front is not None:
        hints[libzim.writer.Hint.FRONT_ARTICLE] = bool(is_front)

    duplicate_ok = duplicate_ok or self.ignore_duplicates

    try:
        try:
            super().add_redirection(path, title or path, target_path, hints)
        except RuntimeError as exc:
            if not DUPLICATE_EXC_STR.match(str(exc)) or not duplicate_ok:
                raise exc
    except Exception:
        if self.workaround_nocancel:
            self.can_finish = False  # pragma: no cover
        raise

config_dev_metadata 

config_dev_metadata(
    extra_metadata: AnyMetadata
    | list[AnyMetadata]
    | None = None,
)

Calls minimal set of mandatory metadata with default values for dev

Extra metadata can be passed, and they are not checked for proper key prefix

Source code in src/zimscraperlib/zim/creator.py 
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
def config_dev_metadata(
    self,
    extra_metadata: AnyMetadata | list[AnyMetadata] | None = None,
):
    """Calls minimal set of mandatory metadata with default values for dev

    Extra metadata can be passed, and they are not checked for proper key prefix
    """
    return self.config_metadata(
        std_metadata=DEFAULT_DEV_ZIM_METADATA,
        extra_metadata=(
            [extra_metadata]
            if isinstance(extra_metadata, MetadataBase)
            else extra_metadata
        ),
        fail_on_missing_prefix_in_extras=False,
    )

config_indexing 

config_indexing(
    indexing: bool, language: str | None = None
)

Toggle full-text indexing of entries

Uses Language metadata's value (or "") if not set.

Note: title indexing is always performed by libzim and cannot be disabled via this method; only the full-text index is toggled.

Source code in src/zimscraperlib/zim/creator.py 
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
def config_indexing(
    self,
    indexing: bool,  # noqa: FBT001
    language: str | None = None,
):
    """Toggle full-text indexing of entries

    Uses Language metadata's value (or "") if not set.

    Note: title indexing is always performed by libzim and cannot be
    disabled via this method; only the full-text index is toggled."""
    language = language or self._get_first_language_metadata_value() or ""
    if indexing and not is_valid_iso_639_3(language):
        raise ValueError("Not a valid ISO-639-3 language code")
    super().config_indexing(indexing, language)
    self.__indexing_configured = True
    self.__indexing_value = indexing
    return self

config_metadata 

config_metadata(
    std_metadata: StandardMetadataList | list[AnyMetadata],
    extra_metadata: list[AnyMetadata] | None = None,
    *,
    fail_on_missing_prefix_in_extras: bool = True,
)

Checks and prepare list of ZIM metadata

Checks ensure that metadata value can be converted to bytes, including all requirements of the ZIM specifications and optionally openZIM conventions.

Metadata are only kept in memory at this stage, not yet passed to libzim.

They will be passed to libzim / writen to the ZIM on creator.start().

Parameters:

  • std_metadata (StandardMetadataList | list[AnyMetadata]) –

    standard metadata defined in the ZIM specifications. Prefer to use StandardMetadataList which ensure mandatory metadata are all set.

  • extra_metadata (list[AnyMetadata] | None, default: None ) –

    a list of extra metadata (not standard).

  • fail_on_missing_prefix_in_extras (bool, default: True ) –

    disable the default check which force the X- prefix on extra metadata name which is a convention to distinguish these extra metadata

Source code in src/zimscraperlib/zim/creator.py 
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
def config_metadata(
    self,
    std_metadata: StandardMetadataList | list[AnyMetadata],
    extra_metadata: list[AnyMetadata] | None = None,
    *,
    fail_on_missing_prefix_in_extras: bool = True,
):
    """Checks and prepare list of ZIM metadata

    Checks ensure that metadata value can be converted to bytes, including all
    requirements of the ZIM specifications and optionally openZIM conventions.

    Metadata are only kept in memory at this stage, not yet passed to libzim.

    They will be passed to libzim / writen to the ZIM on creator.start().

    Arguments:
        std_metadata: standard metadata defined in the ZIM specifications.
            Prefer to use StandardMetadataList which ensure mandatory metadata are
            all set.
        extra_metadata: a list of extra metadata (not standard).
        fail_on_missing_prefix_in_extras: disable the default check which force the
            X- prefix on extra metadata name which is a convention to distinguish
            these extra metadata

    """
    for fail_on_missing_prefix, metadata in [
        (False, metadata)
        for metadata in (
            std_metadata.values()
            if isinstance(std_metadata, StandardMetadataList)
            else std_metadata
        )
    ] + [
        (fail_on_missing_prefix_in_extras, metadata)
        for metadata in extra_metadata or []
    ]:
        if fail_on_missing_prefix and not metadata.name.startswith("X-"):
            raise ValueError(
                f"Metadata key {metadata.name} does not starts with X- as expected"
            )
        # if metadata.name in self._metadata:
        #     raise ValueError(f"{metadata.name} cannot be defined twice")
        self._metadata[metadata.name] = metadata

    return self

finish 

finish(
    _: type[BaseException] | None = None,
    __: BaseException | None = None,
    ___: TracebackType | None = None,
)

Triggers finalization of ZIM creation and create final ZIM file.

Source code in src/zimscraperlib/zim/creator.py 
469
470
471
472
473
474
475
476
477
478
479
480
481
def finish(
    self,
    _: type[BaseException] | None = None,
    __: BaseException | None = None,
    ___: TracebackType | None = None,
):
    """Triggers finalization of ZIM creation and create final ZIM file."""
    if not getattr(self, "can_finish", False):
        return
    try:
        super().__exit__(None, None, None)
    except RuntimeError:
        pass

start 

start()

Start creator operation at libzim level

Includes creating metadata, including validating mandatory ones are set, and configuring indexing.

Source code in src/zimscraperlib/zim/creator.py 
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
def start(self):
    """Start creator operation at libzim level

    Includes creating metadata, including validating mandatory ones are set,
    and configuring indexing.
    """
    if logger.isEnabledFor(logging.DEBUG):  # pragma: no cover
        self._log_metadata()

    if not all(self._metadata.get(key) for key in MANDATORY_ZIM_METADATA_KEYS):
        missing_keys = [
            key
            for key in MANDATORY_ZIM_METADATA_KEYS
            if not self._metadata.get(key)
        ]
        raise ValueError(
            "Mandatory metadata are not all set. Missing metadata: "
            f"{','.join(missing_keys)}. You should prefer to use "
            "StandardMetadataList if possible."
        )

    if (
        language := self._get_first_language_metadata_value()
    ) and not self.__indexing_configured:
        self.config_indexing(True, language)

    ftindex_tag = f"_ftindex:{'yes' if self.__indexing_value else 'no'}"
    tags_metadata = self._metadata.get(TagsMetadata.meta_name)
    if isinstance(tags_metadata, TagsMetadata):
        if not any(
            re.sub(r"\s+", "", part).startswith("_ftindex:")
            for tag in tags_metadata.value
            for part in tag.split(";")
        ):
            tags_metadata.value.append(ftindex_tag)
            logger.debug(f"Metadata: Tags has been altered with '{ftindex_tag}'")
    else:
        self._metadata[TagsMetadata.meta_name] = TagsMetadata([ftindex_tag])
        logger.debug(f"Metadata: Tags has been set with '{ftindex_tag}'")

    super().__enter__()

    for metadata in self._metadata.values():
        if isinstance(metadata, IllustrationBasedMetadata):
            self.add_illustration(metadata.illustration_size, metadata.libzim_value)
        else:
            self.add_metadata(metadata)
    self._metadata.clear()

    return self

FileLikeProvider 

FileLikeProvider(
    fileobj: BytesIO,
    size: int | None = None,
    ref: object | None = None,
)

Bases: ContentProvider

Provider referrencing a file-like object

Use this to keep a single-copy of a content in memory. Useful for indexed content

Methods:

Attributes:

Source code in src/zimscraperlib/zim/providers.py 
42
43
44
45
46
47
48
49
50
51
52
53
54
55
def __init__(
    self,
    fileobj: io.BytesIO,
    size: int | None = None,
    ref: object | None = None,
):
    super().__init__()
    self.ref = ref
    self.fileobj = fileobj
    self.size = size

    if self.size is None:
        self.size = size or self.fileobj.seek(0, io.SEEK_END)
        self.fileobj.seek(0, io.SEEK_SET)

fileobj instance-attribute 

fileobj = fileobj

ref instance-attribute 

ref = ref

size instance-attribute 

size = size

gen_blob 

gen_blob() -> Generator[Blob]
Source code in src/zimscraperlib/zim/providers.py 
60
61
def gen_blob(self) -> Generator[libzim.writer.Blob]:
    yield libzim.writer.Blob(self.fileobj.getvalue())  # pragma: no cover

get_size 

get_size() -> int
Source code in src/zimscraperlib/zim/providers.py 
57
58
def get_size(self) -> int:
    return getattr(self, "size", -1)

FileProvider 

FileProvider(
    filepath: Path,
    size: int | None = None,
    ref: object | None = None,
)

Bases: FileProvider

Attributes:

Source code in src/zimscraperlib/zim/providers.py 
20
21
22
23
24
25
26
27
def __init__(
    self,
    filepath: pathlib.Path,
    size: int | None = None,  # noqa: ARG002
    ref: object | None = None,
):
    super().__init__(filepath)
    self.ref = ref

ref instance-attribute 

ref = ref

Item 

Item(
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[Hint, int] | None = None,
    **kwargs: Any,
)

Bases: Item

libzim.writer.Item returning props for path/title/mimetype

Methods:

Attributes:

Source code in src/zimscraperlib/zim/items.py 
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
def __init__(
    self,
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[libzim.writer.Hint, int] | None = None,
    **kwargs: Any,
):
    super().__init__()
    if path is not None:
        kwargs["path"] = path
    if title is not None:
        kwargs["title"] = title
    if mimetype is not None:
        kwargs["mimetype"] = mimetype
    if hints is not None:
        kwargs["hints"] = hints
    for k, v in kwargs.items():
        setattr(self, k, v)

should_index property 

should_index

get_hints 

get_hints() -> dict[Hint, int]
Source code in src/zimscraperlib/zim/items.py 
60
61
def get_hints(self) -> dict[libzim.writer.Hint, int]:
    return getattr(self, "hints", {})

get_mimetype 

get_mimetype() -> str
Source code in src/zimscraperlib/zim/items.py 
57
58
def get_mimetype(self) -> str:
    return getattr(self, "mimetype", "")

get_path 

get_path() -> str
Source code in src/zimscraperlib/zim/items.py 
51
52
def get_path(self) -> str:
    return getattr(self, "path", "")

get_title 

get_title() -> str
Source code in src/zimscraperlib/zim/items.py 
54
55
def get_title(self) -> str:
    return getattr(self, "title", "")

StaticItem 

StaticItem(
    content: str | bytes | None = None,
    fileobj: IOBase | None = None,
    filepath: Path | None = None,
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[Hint, int] | None = None,
    index_data: IndexData | None = None,
    *,
    auto_index: bool = True,
    **kwargs: Any,
)

Bases: Item

scraperlib Item with auto contentProvider from content or filepath

Sets a ref to itself on the File/String content providers so it outlives them We need Item to survive its ContentProvider so that we can track lifecycle more efficiently: now when the libzim destroys the CP, python will destroy the Item and we can be notified that we're effectively through with our content

By default, content is automatically indexed (either by the libzim itself for supported documents - text or html for now or by the python-scraperlib - only PDF supported for now). If you do not want this, set auto_index to False to disable both indexing (libzim and python-scraperlib).

It is also possible to pass index_data to configure custom indexing of the item.

If item title is not set by caller, it is automatically populated from index_data.

Methods:

Attributes:

Source code in src/zimscraperlib/zim/items.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
def __init__(
    self,
    content: str | bytes | None = None,
    fileobj: io.IOBase | None = None,
    filepath: pathlib.Path | None = None,
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[libzim.writer.Hint, int] | None = None,
    index_data: IndexData | None = None,
    *,
    auto_index: bool = True,
    **kwargs: Any,
):
    if content is not None:
        kwargs["content"] = content
    if fileobj is not None:
        kwargs["fileobj"] = fileobj
    if filepath is not None:
        kwargs["filepath"] = filepath
    super().__init__(
        path=path, title=title, mimetype=mimetype, hints=hints, **kwargs
    )
    if index_data:
        self.get_indexdata: Callable[[], IndexData] = lambda: index_data
    elif not auto_index:
        self.get_indexdata = no_indexing_indexdata  # index nothing
    else:
        self._get_auto_index()  # consider to add auto index

    # Populate item title from index data if title is not set by caller
    if (not getattr(self, "title", None)) and hasattr(self, "get_indexdata"):
        title = self.get_indexdata().get_title()
        if title:
            self.title = title

get_indexdata instance-attribute 

get_indexdata: Callable[[], IndexData] = lambda: index_data

should_index property 

should_index

title instance-attribute 

title = title

get_contentprovider 

get_contentprovider() -> ContentProvider
Source code in src/zimscraperlib/zim/items.py 
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
def get_contentprovider(self) -> libzim.writer.ContentProvider:
    # content was set manually
    content = getattr(self, "content", None)
    if content is not None:
        if not isinstance(content, str | bytes):
            raise AttributeError(f"Unexpected type for content: {type(content)}")
        return StringProvider(content=content, ref=self)

    # using a file-like object
    fileobj = getattr(self, "fileobj", None)
    if fileobj:
        return FileLikeProvider(
            fileobj=fileobj, ref=self, size=getattr(self, "size", None)
        )

    # we had to download locally to get size
    filepath = getattr(self, "filepath", None)
    if filepath:
        return FileProvider(
            filepath=filepath, ref=self, size=getattr(self, "size", None)
        )

    raise NotImplementedError("No data to provide`")

get_hints 

get_hints() -> dict[Hint, int]
Source code in src/zimscraperlib/zim/items.py 
60
61
def get_hints(self) -> dict[libzim.writer.Hint, int]:
    return getattr(self, "hints", {})

get_mimetype 

get_mimetype() -> str
Source code in src/zimscraperlib/zim/items.py 
57
58
def get_mimetype(self) -> str:
    return getattr(self, "mimetype", "")

get_path 

get_path() -> str
Source code in src/zimscraperlib/zim/items.py 
51
52
def get_path(self) -> str:
    return getattr(self, "path", "")

get_title 

get_title() -> str
Source code in src/zimscraperlib/zim/items.py 
54
55
def get_title(self) -> str:
    return getattr(self, "title", "")

StringProvider 

StringProvider(
    content: str | bytes, ref: object | None = None
)

Bases: StringProvider

Attributes:

Source code in src/zimscraperlib/zim/providers.py 
31
32
33
def __init__(self, content: str | bytes, ref: object | None = None):
    super().__init__(content)
    self.ref = ref

ref instance-attribute 

ref = ref

URLItem 

URLItem(
    url: str,
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[Hint, int] | None = None,
    *,
    use_disk: bool | None = None,
    **kwargs: Any,
)

Bases: StaticItem

StaticItem to automatically fetch and feed an URL resource

Appropriate for retrieving/bundling static assets that you don't need to post-process.

Uses URL's path as zim path if none provided Keeps single in-memory copy of content for HTML resources (indexed) Works transparently on servers returning a Content-Length header (most) Swaps a copy of the content either in memory or on disk (use_disk=True) in case the content size could not be retrieved from headers. Use tmp_dir to point location of that temp file.

Methods:

Attributes:

Source code in src/zimscraperlib/zim/items.py 
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
def __init__(
    self,
    url: str,
    path: str | None = None,
    title: str | None = None,
    mimetype: str | None = None,
    hints: dict[libzim.writer.Hint, int] | None = None,
    *,
    use_disk: bool | None = None,
    **kwargs: Any,
):
    if use_disk is not None:
        kwargs["use_disk"] = use_disk
    super().__init__(
        path=path, title=title, mimetype=mimetype, hints=hints, **kwargs
    )
    self.url = urllib.parse.urlparse(url)
    use_disk_set: bool = getattr(self, "use_disk", False)

    # fetch headers to retrieve size and type
    try:
        _, self.headers = stream_file(
            url, byte_stream=io.BytesIO(), only_first_block=True
        )
    except Exception as exc:
        raise OSError(f"Unable to access URL at {url}: {exc}") from None

    # HTML content will be indexed.
    # we proxy the content in the Item to prevent double-download of the resource
    # we use a value-variable to prevent race-conditions in the multiple
    # reads of the content in the provider
    if self.should_index:
        self.fileobj = io.BytesIO()
        self.size, _ = stream_file(self.url.geturl(), byte_stream=self.fileobj)
        return

    try:
        # Encoded data (compressed) prevents us from using Content-Length header
        # as source for the content (it represents length of compressed data)
        if self.headers.get("Content-Encoding", "identity") != "identity":
            raise ValueError("Can't trust Content-Length for size")
        # non-html, non-compressed data.
        self.size = int(self.headers["Content-Length"])
    except Exception:
        # we couldn't retrieve size so we have to download resource to
        target, self.size = self.download_for_size(
            self.url, on_disk=use_disk_set, tmp_dir=getattr(self, "tmp_dir", None)
        )
        # downloaded to disk and using a file path from now on
        if use_disk:
            self.filepath = target
        # downloaded to RAM and using a bytes object
        else:
            self.fileobj = target

fileobj instance-attribute 

fileobj = BytesIO()

filepath instance-attribute 

filepath = target

get_indexdata instance-attribute 

get_indexdata: Callable[[], IndexData] = lambda: index_data

should_index property 

should_index

size instance-attribute 

size = int(headers['Content-Length'])

title instance-attribute 

title = title

url instance-attribute 

url = urlparse(url)

download_for_size staticmethod 

download_for_size(
    url: ParseResult,
    tmp_dir: Path | None = None,
    *,
    on_disk: bool,
)

Download URL to a temp file and return its tempfile and size

Source code in src/zimscraperlib/zim/items.py 
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
@staticmethod
def download_for_size(
    url: urllib.parse.ParseResult,
    tmp_dir: pathlib.Path | None = None,
    *,
    on_disk: bool,
):
    """Download URL to a temp file and return its tempfile and size"""
    fpath = stream = None
    if on_disk:
        suffix = pathlib.Path(re.sub(r"^/", "", url.path)).suffix
        fpath = pathlib.Path(
            tempfile.NamedTemporaryFile(
                suffix=suffix, delete=False, dir=tmp_dir
            ).name
        )
    else:
        stream = io.BytesIO()
    size, _ = stream_file(url.geturl(), fpath=fpath, byte_stream=stream)
    return fpath or stream, size

get_contentprovider 

get_contentprovider()
Source code in src/zimscraperlib/zim/items.py 
299
300
301
302
303
304
305
def get_contentprovider(self):
    try:
        return super().get_contentprovider()
    except NotImplementedError:
        return URLProvider(
            url=self.url.geturl(), size=getattr(self, "size", None), ref=self
        )

get_hints 

get_hints() -> dict[Hint, int]
Source code in src/zimscraperlib/zim/items.py 
60
61
def get_hints(self) -> dict[libzim.writer.Hint, int]:
    return getattr(self, "hints", {})

get_mimetype 

get_mimetype() -> str
Source code in src/zimscraperlib/zim/items.py 
292
293
294
295
296
297
def get_mimetype(self) -> str:
    return getattr(
        self,
        "mimetype",
        self.headers.get("Content-Type", "application/octet-stream"),
    )

get_path 

get_path() -> str
Source code in src/zimscraperlib/zim/items.py 
286
287
def get_path(self) -> str:
    return getattr(self, "path", re.sub(r"^/", "", self.url.path))

get_title 

get_title() -> str
Source code in src/zimscraperlib/zim/items.py 
289
290
def get_title(self) -> str:
    return getattr(self, "title", "")

URLProvider 

URLProvider(
    url: str,
    size: int | None = None,
    ref: object | None = None,
)

Bases: ContentProvider

Provider downloading content as it is consumed by the libzim

Useful for non-indexed content for which feed() is called only once

Methods:

Attributes:

Source code in src/zimscraperlib/zim/providers.py 
69
70
71
72
73
74
75
76
77
78
def __init__(self, url: str, size: int | None = None, ref: object | None = None):
    super().__init__()
    self.url = url
    self.size = size if size is not None else self.get_size_of(url)
    self.ref = ref

    session = requests.Session()
    session.mount("http", get_retry_adapter())
    self.resp = session.get(url, stream=True)
    self.resp.raise_for_status()

ref instance-attribute 

ref = ref

resp instance-attribute 

resp = get(url, stream=True)

size instance-attribute 

size = size if size is not None else get_size_of(url)

url instance-attribute 

url = url

gen_blob 

gen_blob() -> Generator[Blob]
Source code in src/zimscraperlib/zim/providers.py 
91
92
93
94
95
def gen_blob(self) -> Generator[libzim.writer.Blob]:  # pragma: no cover
    for chunk in self.resp.iter_content(10 * 1024):
        if chunk:
            yield libzim.writer.Blob(chunk)
    yield libzim.writer.Blob(b"")

get_size 

get_size() -> int
Source code in src/zimscraperlib/zim/providers.py 
88
89
def get_size(self) -> int:
    return getattr(self, "size", -1)

get_size_of staticmethod 

get_size_of(url: str) -> int | None
Source code in src/zimscraperlib/zim/providers.py 
80
81
82
83
84
85
86
@staticmethod
def get_size_of(url: str) -> int | None:
    _, headers = stream_file(url, byte_stream=io.BytesIO(), only_first_block=True)
    try:
        return int(headers["Content-Length"])
    except Exception:
        return None

make_zim_file 

make_zim_file(
    *,
    build_dir: Path,
    fpath: Path,
    name: str,
    main_page: str,
    illustration: str,
    title: str,
    description: str,
    date: date | None = None,
    language: str = "eng",
    creator: str = "-",
    publisher: str = "-",
    tags: Sequence[str] | None = None,
    source: str | None = None,
    flavour: str | None = None,
    scraper: str | None = None,
    long_description: str | None = None,
    without_fulltext_index: bool = False,
    redirects: Sequence[tuple[str, str, str]] | None = None,
    redirects_file: Path | None = None,
    rewrite_links: bool = True,
    workaround_nocancel: bool = True,
    ignore_duplicates: bool = True,
    disable_metadata_checks: bool = False,
)

Creates a zimwriterfs-like ZIM file at {fpath} from {build_dir}

main_page: path of item to serve as main page illustration: relative path to illustration file in build_dir tags: list of str tags to add to meta redirects: list of (src, dst, title) tuple to create redirects from rewrite_links controls whether to rewrite HTML/CSS content -> add namespaces to relative links workaround_nocancel: disable workaround to prevent ZIM creation on error

Source code in src/zimscraperlib/zim/filesystem.py 
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
def make_zim_file(
    *,
    build_dir: pathlib.Path,
    fpath: pathlib.Path,
    name: str,
    main_page: str,
    illustration: str,
    title: str,
    description: str,
    date: datetime.date | None = None,
    language: str = "eng",
    creator: str = "-",
    publisher: str = "-",
    tags: Sequence[str] | None = None,
    source: str | None = None,
    flavour: str | None = None,
    scraper: str | None = None,
    long_description: str | None = None,
    without_fulltext_index: bool = False,  # noqa: ARG001
    redirects: Sequence[tuple[str, str, str]] | None = None,
    redirects_file: pathlib.Path | None = None,
    rewrite_links: bool = True,  # noqa: ARG001
    workaround_nocancel: bool = True,
    ignore_duplicates: bool = True,
    disable_metadata_checks: bool = False,
):
    """Creates a zimwriterfs-like ZIM file at {fpath} from {build_dir}

    main_page: path of item to serve as main page
    illustration: relative path to illustration file in build_dir
    tags: list of str tags to add to meta
    redirects: list of (src, dst, title) tuple to create redirects from
    rewrite_links controls whether to rewrite HTML/CSS content
      -> add namespaces to relative links
    workaround_nocancel: disable workaround to prevent ZIM creation on error"""

    # sanity checks
    if not build_dir.exists() or not build_dir.is_dir():
        raise OSError(f"Incorrect build_dir: {build_dir}")

    illustration_path = build_dir / illustration
    if not illustration_path.exists() or not illustration_path.is_file():
        raise OSError(f"Incorrect illustration: {illustration} ({illustration_path})")

    with open(illustration_path, "rb") as fh:
        illustration_data = fh.read()

    # disable recommendations if requested
    metadata.APPLY_RECOMMENDATIONS = not disable_metadata_checks

    zim_file = Creator(
        filename=fpath,
        main_path=main_page,
        ignore_duplicates=ignore_duplicates,
    ).config_metadata(
        metadata.StandardMetadataList(
            # mandatory
            Name=metadata.NameMetadata(name),
            Title=metadata.TitleMetadata(title),
            Description=metadata.DescriptionMetadata(description),
            Date=metadata.DateMetadata(date or datetime.date.today()),  # noqa: DTZ011
            Language=metadata.LanguageMetadata(language),
            Creator=metadata.CreatorMetadata(creator),
            Publisher=metadata.PublisherMetadata(publisher),
            Illustration_48x48_at_1=metadata.DefaultIllustrationMetadata(
                illustration_data
            ),
            # optional
            Tags=metadata.TagsMetadata(list(tags)) if tags else None,
            Source=metadata.SourceMetadata(source) if source else None,
            Flavour=metadata.FlavourMetadata(flavour) if flavour else None,
            Scraper=metadata.ScraperMetadata(scraper) if scraper else None,
            LongDescription=(
                metadata.LongDescriptionMetadata(long_description)
                if long_description
                else None
            ),
        )
    )

    zim_file.start()
    try:
        logger.debug(f"Preparing zimfile at {zim_file.filename}")

        # recursively add content from build_dir
        logger.debug(f"Recursively adding files from {build_dir}")
        add_to_zim(build_dir, zim_file, build_dir)

        if redirects or redirects_file:
            logger.debug("Creating redirects")
            add_redirects_to_zim(
                zim_file, redirects=redirects, redirects_file=redirects_file
            )

    # prevents .finish() which would create an incomplete .zim file
    # this would leave a .zim.tmp folder behind.
    # UPSTREAM: wait until a proper cancel() is provided
    except Exception:
        if workaround_nocancel:
            zim_file.can_finish = False  # pragma: no cover
        raise
    finally:
        zim_file.finish()