DOWARC: a domain ontology for web archiving and web preservation

The act of creating an archive through the process of capturing data into a WARC file.

An object created as a derivative of a web archiving data entity. During the derivation process, modification of the information contained in the original object is allowed; however, some of the original data must be maintained, at a representational and/or content level. An example of Derivative is a list of URLs extracted from a WARC file. This class must be used only when describing web archiving data object. Subclass of prov-o:Entity.

The act of extracting data from a WARC file and/or a CDX file.

An unambiguous reference (character string or numeric) to a data object.

The creation of an index (CDX or CDXJ) file of a WARC file. Source: https://pywb.readthedocs.io/en/master/manual/indexing.html

An action undertaken, or an activity occurring, within or outside the repository that influence its capability to preserve web archiving data objects. Subclass of premis:Event

Web archiving data object subject to digital preservation. Subclass of premis:Object.

The act of making a WARC file, or resources within a WARC file, availavable for use

The act of checking the quality of a web archive using a combination of data-driven and manual checks. This process includes the act of patching, during which time missing content is added to the archive to imprve its completeness or quality.

Retrieval and rendering of the data stored in a WARC file over HTTP. This may be through a Wayback-style system via a browser, or programatically replayed in some other way.

An optional parameter indicating the algorithm name and calculated value of a digest applied to the full block of the record. An example is a SHA-1 labelled Base32 ([RFC3548]) value. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The WARC-Record-IDs of any records created as part of the same capture event as the current record. A capture event comprises the information automatically gathered by a retrieval against a single target-URI; for example, it might be represented by a "response"or "revisit"record plus its associated "request"record. This field may be used to associate records of types "request", "response", "resource", "metadata", and "revisit"with one another when they arise from a single capture event (When so used, any WARC Concurrent-To association shall be considered bidirectional even if the header only appears on one record.) As an exception to the general rule, it is allowed to repeat several WARC-Concurrent-To fields within the same WARC record. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

A 14-digit UTC timestamp formatted according to YYYY-MM-DDThh:mm:ssZ, described in the W3C profile of ISO8601 [W3CDTF]. The timestamp shall represent the instant that data capture for record creation began. Multiple records written as part of a single capture event (see section 5.7) use the same WARC-Date, even though the times of their writing will not be exactly synchronized. All records have a WARC-Date field. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The filename containing the current "warcinfo"record.The WARC-Filename field might be used in "warcinfo"type records and is not used for other record types. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The numeric Internet address contacted to retrieve any included content. An IPv4 address is written as a "dotted quad"; an IPv6 address is written as per [RFC1884]. For an HTTP retrieval, this is the IP address used at retrieval time corresponding to the hostname in the record"s target-Uri. The WARC-IP-Address field is used on "response", "resource", "request", "metadata", and "revisit"records, but is not used on ‘warcinfo’, "conversion"or "continuation"records. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The content-type of the record"s payload as determined by an independent check. The WARC-Identified-Payload-Type field might be used on WARC records with a well-defined payload and is not used on records without a well-defined payload. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

An optional parameter indicating the algorithm name and calculated value of a digest applied to the payload referred to or contained by the record - which is not necessarily equivalent to the record block. The payload of an application/http block is its ‘entity-body’ (per [RFC2616]). In contrast to WARC-Block-Digest, the WARC-Payload-Digest field may also be used for data not actually present in the current record block, for example when a block is left off in accordance with a ‘revisit’ profile (see ‘revisit’), or when a record is segmented (the WARC-Payload-Digest recorded in the first segment of a segmented record shall be the digest of the payload of the logical record). The WARC-Payload-Digest field might be used on WARC records with a well-defined payload and is not used on records without a well-defined payload.

A URI signifying the kind of analysis and handling applied in a "revisit"record. (Like an XML namespace, the URI may, but need not, return human-readable or machine-readable documentation.) If reading software does not recognize the given URI as a supported kind of handling, it does not attempt to interpret the associated record block.The section "revisit"defines two initial profile options for the WARC-Profile header for "revisit"records. The WARC-Profile field is mandatory on "revisit"type records and undefined for other record types.(Source: ISO 28500; https://iipc.github.io/warc-specifications/)

An identifier assigned to the current record that is globally unique for its period of intended use. No identifier scheme is mandated by this specification, but each record-id shall be a legal URI and clearly indicate a documented and registered scheme to which it conforms (e.g., via a URI scheme prefix such as "http:"or "urn:"). All records have a WARC-Record-ID field. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The WARC-Record-ID of a single record for which the present record holds additional content. The WARC-Refers-To field might be used to associate a "metadata"record to another record it describes. The WARC-Refers-To field might also be used to associate a record of type "revisit"or "conversion"with the preceding record which helped determine the present record content. The WARC-Refers-To field is not used in "warcinfo", "response", ‘resource’, "request", and "continuation"records.(Source: ISO 28500; https://iipc.github.io/warc-specifications/)

Reports the current record"s relative ordering in a sequence of segmented records. In the first segment of any record that is completed in one or more later "continuation"WARC records, this parameter is mandatory. Its value there is "1". In a "continuation"record, this parameter is also mandatory. Its value is the sequence number of the current segment in the logical whole record, increasing by 1 in each next segment. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

Identifies the starting record in a series of segmented records whose content blocks are reassembled to obtain a logically complete content block. This field is mandatory on all "continuation"records, and is not used in other records.(Source: ISO 28500; https://iipc.github.io/warc-specifications/)

In the final record of a segmented series, reports the total length of all segment content blocks when concatenated together. This field is mandatory on the last "continuation"record of a series, and is not used elsewhere. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The original URI whose capture gave rise to the information content in this record. In the context of web harvesting, this is the URI that was the target of a crawler’s retrieval request. For a ‘revisit’ record, it is the URI that was the target of a retrieval request. Indirectly, such as for a ‘metadata’, or ‘conversion’ record, it is a copy of the WARC-Target-URI appearing in the original record to which the newer record pertains. The URI in this value shall be properly escaped according to [RFC3986] and written with no internal whitespace. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

For practical reasons, writers of the WARC format may place limits on the time or storage allocated to archiving a single resource. As a result, only a truncated portion of the original resource may be available for saving into a WARC record. Any record might indicate that truncation of its content block has occurred and give the reason with a "WARC-Truncated"field. For example, if the capture of what appeared to be a multi-gigabyte resource was cut short after a transfer time limit was reached, the partial resource could be saved to a WARC record with this field. The WARC-Truncated field might be used on any WARC record. The WARC field Content-Length still reports the actual truncated size of the record block. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The type of WARC record: one of "warcinfo", "response", "resource", "request", "metadata", "revisit","conversion", or "continuation". Other types of WARC records may be defined in extensions of the core format. Types are further described in WARC Record Types. All records have a WARC-Type field. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

When present, indicates the WARC-Record-ID of the associated "warcinfo"record for this record. Typically, the Warcinfo-ID parameter is used when the context of the applicable "warcinfo"record is unavailable, such as after distributing single records into separate WARC files. WARC writing applications (such web crawlers) might choose to always record this parameter.The WARC-Warcinfo-ID field value overrides any association with a previously occurring (in the WARC) "warcinfo"record, thus providing a way to protect the true association when records are combined from different WARCs. The WARC-Warcinfo-ID field might be used in any record type except "warcinfo". (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The fields contained within a CDX or CDXJ file. Source: https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2015/

An index (normally a CDX or CDXJ) file of a WARC file. Source: https://pywb.readthedocs.io/en/master/manual/indexing.html

A collection of WARC records aggregating all the resources harvested by web crawling activities, which, taken together, provide an exhaustive representation of a live web resource (e.g., a web site, a web page). Subclass of ore:Aggregation.

An activity which relates to or involves creating, indexing, modifying or replaying a WARC file.

A container file that aggregates a sequence of structured information chunks (WARC records), storing data and metadata. WARC files are the output of web crawling automated activities. A WARC file can contain one or more digital object with multiple media types (i.e., MIME). Subclass of ore:Aggregation.

An RDF graph that represents and describes the relationships between the WARC file and the resources it aggregates, as well as the relationships between the aggregated resources. A WARC graph can be used to represent the relationship between the resources stored in a WARCcollection and the web page(s) or web site(s) from which the resources were harvested. In this respect, a WARCgraph can provide a representation of the live web object being archived. A WARCgraph can be a subgraph of another WARCgraph (hasPart/isPartOf).

A WARC record is a component of a WARC file. A WARC record includes data retrieved from a source in the live Wide World Web, together with synthetic (meta)data providing information about the retrieved content. A WARC record has a header and a content block carrying the payload for the live web data object that was harvested over the HTTP. A WARC record is both an aggregated resource (component of a WARC file) and an aggregation (it includes sub-elements of its own).

Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a WARC record. (Source: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#terms-and-definitions)

A resource aggregated in a WARC record.

Beginning of a WARC record, consisting of one first line declaring the record to be in the WARC format with a given version number, followed by lines of named fields up to a blank line. (Source: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#terms-and-definitions)

Set of elements consisting of a name, a colon, and a value, with long values continued on indented lines. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

Data object referred to, or contained by a WARC record as a meaningful subset of the content block. (Source: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#terms-and-definitions)

Subclass of "foaf:Agent"and "prov:Agent", specified in a web archiving environment. "Class: foaf:Agent. An agent (eg. person, group, software or physical artifact)."(Source: http://xmlns.com/foaf/spec/#term_Agent) - "Class: prov:Agent. An agent is something that bears some form of responsibility for an activity taking place, for the existence of an entity, or for another agent"s activity."(Source: https://www.w3.org/TR/prov-o/#Agent)

Canonicalization is applied to URIs to remove trivial differences in the URIs that do not reflect that the URI reference different resources. Examples include removing session ID parameters, unneccessary port declerations (e.g. :80 when crawling HTTP). (Source: https://nlevitt.github.io/warc-specifications/specifications/cdx-format/openwayback-cdxj/#appendix-a--canonicalization)

The "offset"is the number of bytes from the start of the WARC when the relevant record begins. (Source: https://nlevitt.github.io/warc-specifications/specifications/cdx-format/openwayback-cdxj/)

The number of octets in the compressed record in the WARC file. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

A timestamp for this record, stored as the 14 digits from the text form of a WARC::Date without the associated marker characters, i.e. YYYYmmddHHMMSS instead of YYYY-mm-ddTHH:MM:SSZ. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The name of the WARC volume containing this record. This value is assumed to be relative to the directory containing the CDX file and to always be a Unix-style filename, regardless of the local file name conventions. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The URL used in the request, as for the "a"field, but with the hostname translated to SURT form and the scheme component removed so that the value starts with the TLD. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

This field seems to be very common in CDX files, but no values have been observed and this field appears to be completely undocumented. Ignored on read, but written as "-"if included when building an index. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The MIME type reported in the Content-Type header of the response. Written as "-"if the record does not contain an HTTP response with an entity body. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

Content-Type of the resource record - in alexa-made dat file (Source: https://archive.org/web/researcher/dat_file_format.php)

Typically the base32-encoded SHA1 digest of the response payload. This value is copied from the WARC-Payload-Digest header of a record if available, otherwise "-"is written. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The URL that was used in the request that produced this response. The url_prefix search key matches a prefix of this value. This value is copied from the WARC-Target-URI header and is written as "-"if the record does not have that header. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The contents of the Location header of the response, URL-escaped. Some implementations do not properly set this field. Written as "-"if not present or if the record does not contain an HTTP response. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

The HTTP status code used in the response. Written as "-"if the record does not contain an HTTP response. (Source: https://metacpan.org/pod/WARC::Index::File::CDX)

An optional parameter indicating the algorithm name and calculated value of a digest applied to the payload referred to or contained by the record - which is not necessarily equivalent to the record block. The payload of an application/http block is its ‘entity-body’ (per [RFC2616]). In contrast to WARC-Block-Digest, the WARC-Payload-Digest field may also be used for data not actually present in the current record block, for example when a block is left off in accordance with a ‘revisit’ profile (see ‘revisit’), or when a record is segmented (the WARC-Payload-Digest recorded in the first segment of a segmented record shall be the digest of the payload of the logical record). The WARC-Payload-Digest field might be used on WARC records with a well-defined payload and is not used on records without a well-defined payload.

The WARC-Refers-To-Date field might be used in ‘revisit’ records and is not used in "warcinfo", "response", "metadata", "conversion", ‘resource’, "request", and "continuation"records. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The WARC-Refers-To-Target-URI field might be used in ‘revisit’ records and is not used in "warcinfo", "response", "metadata", "conversion", ‘resource’, "request", and "continuation"records. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)

The original URI whose capture gave rise to the information content in this record. In the context of web harvesting, this is the URI that was the target of a crawler’s retrieval request. For a ‘revisit’ record, it is the URI that was the target of a retrieval request. Indirectly, such as for a ‘metadata’, or ‘conversion’ record, it is a copy of the WARC-Target-URI appearing in the original record to which the newer record pertains. The URI in this value shall be properly escaped according to [RFC3986] and written with no internal whitespace. (Source: ISO 28500; https://iipc.github.io/warc-specifications/)