AVAA Toolkit $VERSION

The Audio Video Annotations Analysis Toolkit

Introduction

Corpora analysis is a complex task, requiring to learn editors for different file formats and multiple tools, often command-line based, or with programming knowledge prerequisite.

$ATK makes it easy to create pipelines connecting ecosystems to process raw data (automated transcriptions, formats conversion..), and query large corpora of annotations coming from various sources to extract advanced statistics and generate beautiful, always up-to-date charts and timelines.

$ATK is also a flexible converter ; it takes as input XML files describing the style and operations to generate an HTML document, and takes care of exporting only relevant portions of videos and their thumbnail snapshots, minimizing final document size and potential load times if hosted online.

Annotations Formats

$ATK understands the following file formats

GPL-3.0 License TEI, CHA and TEXTGRID formats are available thanks to theTEI-CORPO project

Raw Media Formats

$ATK can also process the following media types

Multiple Licenses Most media processing made possible byFFmpeg

Installation

Simply extract the latest release zip

Windows

To start $ATK, simply double-click the launcher file avaa-toolkit.exe

The executable is not signed, so Windows will ask for confirmation before starting it.

In case $ATK fails to start, check the troubleshooting section.

Linux / macOS

On those systems $ATK can be started by running the shell script avaa-toolkit.sh but it must be set as executable first:

  1. Open a terminal and navigate to $ATK's directory with the cd command.
  2. Run the following command to make $ATK's launcher executable: sudo chmod +x avaa-toolkit.sh
  3. Start $ATK by running the command ./avaa-toolkit.sh or double-click the launcher file.

Note that at least Java 11 is required, you can install latest version using trusted adoptium.net binaries.

Update

When $ATK is already installed, follow these steps to update:

  1. Close $ATK if it is running
  2. Delete these folders: scripts, editor, tests
  3. Download the latest release zip
  4. From the zip, copy to your installation folder (replace existing) avaa-toolkit.jar, include, scripts, editor, tests
  5. Restart toolkit

Editor

An editor for $ATK's XML documents is available in the browser.

To begin, start $ATK by running the launcher (avaa-toolkit.exe on Windows or avaa-toolkit.sh on Linux/macOS),
then navigate with your browser to avaa-toolkit.org
If internet is not available, use the provided offline editor in your installation folder (open index.html)

By default, the editor is allowed to create and edit files in the projects folder.

It is possible to add other folders to the editor, just edit avaa-config.xml file.

XML Structure

$ATK will process XML files and convert them to HTML.
It expects a document with the following structure

Queries

$ATK is all about querying and filtering annotations. Complex queries can be expressed to extract only specific annotations.
This is done via the SELECT tag, various attributes can be combined to make a curated selection of annotations:

Attributes of type regexp (*-match) have additional options:

When multiple attributes are used, the selection will consist only of the annotations fulfilling all the constraints.

##scripts##

Processor Pipelines

$ATK features an advanced pipeline system easing automation of complex tasks.

A pipeline is created for each section of the document, and initially contains a virtual copy of the corpus and its associated media files.
The corpus and its media files are then modified sequentially by each processor inside the pipeline.

Pipeline Input Modes

The pipeline can be fed different initial media files, by defining the processor-pipeline-input setting.

  1. corpus: this is the default mode if the processor is placed at the beginning of a section, and will feed the pipeline with the corpus media files
  2. section-assets: this is the default mode if the processor is placed after a view which exported clips, and will feed the pipeline with all the exported clips/snapshots (of the section) until this processor was reached
  3. all-assets: this mode must be manually selected, and will feed the pipeline with all the exported clips/snapshots of the document until this processor was reached

The corpus mode is useful to process corpus files directly (audio-anonymization, formats conversion...), while for instance all-assets mode could be used to apply effects only on the exported media of the document intended for sharing with peers.

Pipeline Chain

Processors inside a pipeline (that is for now, a section of the document) are executed one after another, each processor using the results of the previous one to work on.

Complex chains of processors can be built to automate heavy tasks alleviating the burden of manually running each step and verifying its consistency.

Pipeline and Views

Views placed after a processor (in the same section) will inherit its modified media files when exporting clips and snapshots.
This can be helpful extracting annotations from cuts of raw media files, to avoid processing long corpus media file when testing samples ; or preprocessing a media file before it is exported into clips during later views generation.

Processors generating annotations will make these annotations immediately available in the main corpus (and not only for the current pipeline), hence for all subsequent views and processors in the document.

Settings

Settings can be modified at any time via the Local Settings block.

Styling

It is possible to change the style via CSS. The HTML code generated makes it easy to target specific elements or apply styling rules for the whole page. Each view has its own structure of elements, and a simple "Inspect Element" from browser will reveal selectors.

Embedded CSS

Styles can be defined directly in the XML file, by using a STYLE tag.

These styles will only apply to this specific HTML document.

<STYLE> .view-timeline td { border-color:red; } .view-timeline tr.tier-header { text-align:right; } </STYLE>

CSS File

Styles can be defined in a separate CSS file, that must be placed in the include folder.

All the generated HTML documents will load this file and have these styles in common.

e.g. my-styles.css h2 { color:green; } section { border-left: 2px solid gray; }

Styling Views

Views generate simple HTML code and try to follow common guidelines so that applying styles is straightforward

Annotations' text labels always have the annotation class, so for instance to change the color of all annotations:

.view .annotation { color:red; }

Making PDF

$ATK can also generate PDF, though interactive features like videos or dynamic charts won't work in this format, for obvious reasons.

Chrome (or Chromium) must be installed on the system (alternatively on Windows $ATK will try to use Edge).

Chrome/Edge executable should be detected automatically, if that fails it is required to provide its path in avaa-config.xml

If Chrome/Edge is not available, it is recommended to install Chrome Headless Shell and then provide its path in avaa-config.xml

Command Line

$ATK is made for the command line and can integrate seamlessly in any tool chain.

##cli##

Troubleshooting

Installation and first run

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError:
    org/avaatoolkit/Main has been compiled by a more recent version of the Java Runtime (class file version 55.0),
        this version of the Java Runtime only recognizes class file versions up to X

Solution: Your version of java runtime is outdated, follow these steps


java.net.BindException: Couldn't bind to any port in the range `42042:42042`.
    at org.glassfish.grizzly.AbstractBindingHandler.bind(AbstractBindingHandler.java)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java)
    at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:)
    at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java)
    at org.avaatoolkit.server.Daemon.start(Daemon.java)
    at org.avaatoolkit.Main.main(Main.java)

Solution: The toolkit is already started with the --server argument, close it before running a new instance.
Solution: Your firewall has a strict policy regarding localhost port bindings, add a rule to allow localhost:42042

Custom Java Runtime

On some operating systems, the installed java runtime might not be up-to-date and prevent $ATK from executing properly.
To run $ATK, at least java 11 is required. To install a valid runtime:

Alternatively using the OpenJDK archives:

  • Go to Open JDK and download the archive for your system
  • Extract the archive into $ATK installation's folder
  • Rename the extracted jdk-22.x.x folder to jdk
  • The directory path should be avaa-toolkit / jdk / bin /
  • The launcher should now use the provided runtime in the jdk folder automatically

Advanced Video Processing

Some processors require a full FFmpeg version to work.

FFmpeg Clipping Caveats

Short clips or sync issues

When generating really short clips (under 1 sec), it is possible that the clips will consist of only one frozen image.

This is because by default FFmpeg will be instructed to do a copy of the video stream (vcopy), which saves considerable processing time, at the expense of less accurate clipping.

When perfect accuracy is required for clips, it is recommended to force FFmpeg re-encoding, for instance by defining the Setting video-codec = h264

Licenses

About licensed material

This documentation includes attributions to licensed material such as libraries and software modules.
These notices are written explicitly in each relevant component and for convenience listed again below.

Components License Notices

Some modules are not included in $ATK but rather installed on demand whenever a component requires it.
Other modules are included or integrated in $ATK to provide a better overall user experience.
This behavior is indicated by a little icon preceding the license, as well as a tooltip describing its inclusion method.

Redistributed Material

Additional libraries are packaged with the produced HTML document, and therefore redistributed by the end user.

jQuery

jQuery simplifies DOM manipulation, some components use it to initialize content in the browser.

🗀 MIT License jQuery jquery.com/license

D3 - Data Visualization Library

D3 has unparalleled flexibility in building custom and dynamic visualizations.
Charts generated by $ATK are actually rendered right in the browser with D3.

🗀 ISC License D3 d3js.org

GSAP - GreenSock Animation Platform

GSAP is incredible and we deemed its inclusion valuable for providing a robust interactivity and animation framework for future $ATK components.

🗀 Custom GreenSock License GSAP gsap.com/licensing

Tipped - Tooltip solution based on jQuery

Tipped features easy to use and customizable tooltips.
$ATK views sometimes use these tooltips for instance to show snapshots or videos in a small popup when an annotation is clicked or hovered.

🗀 CC BY 4.0 License Tipped github.com/staaky/tipped

FileSaver.js - Save files generated in the browser

FileSaver.js provides a simple interface to save (as a "download") files created directly in the browser.
We believe $ATK components can benefit from the presence of the FileSaver library.

🗀 MIT License FileSaver.js github.com/eligrey/FileSaver.js

AVAA Toolkit Core Libraries

$ATK itself is built with Java, and makes use of various libraries (via Maven) which are compiled into the final JAR executable distributed to the toolkit users.

NewPipe Extractor

A library for extracting things from streaming sites, $ATK includes this library to provide an easy API for downloading PeerTube videos.

🗀 GPL-3.0 License NewPipe Extractor github.com/TeamNewPipe/NewPipeExtractor

Bramp FFmpeg

A FFmpeg CLI Wrapper for Java, used to execute FFmpeg and read progress feedback.

🗀 BSD-2-Clause License Bramp FFmpeg github.com/bramp/ffmpeg-cli-wrapper

Mozilla Rhino

Rhino is the JavaScript engine used to execute all components' scripts.

🗀 Mozilla Public License 2.0 Rhino github.com/mozilla/rhino

JCommander

The best library for parsing command-line arguments.

🗀 Apache-2.0 License JCommander github.com/cbeust/jcommander

Grizzly

This library is used to spawn server sockets, and brings WebSocket sessions (that's how the Editor can interact with $ATK).

🗀 Eclipse Public License 2.0 Grizzly github.com/eclipse-ee4j/grizzly

Jsoup

Jsoup simplifies HTML/XML parsing via a CSS selectors syntax.

🗀 MIT License Jsoup jsoup.org

Jspecify

An artifact of fully-specified annotations to power static-analysis checks, beginning with nullness analysis.

🗀 Apache-2.0 License Jspecify jspecify.org

Jchardet

Jchardet is a Java port of the source from mozilla's automatic charset detection algorithm.

🗀 Mozilla Public License 1.1 Jchardet jchardet.sourceforge.net

OSHI

A JNA-based (native) Operating System and Hardware Information library, to get processes details and CPU usage.

🗀 MIT License OSHI github.com/oshi/oshi

Apache Commons

Apache Commons is a set of commonly needed features implemented as reusable Java components.

🗀 Apache-2.0 License Apache Commons commons.apache.org

SLF4J

A simple facade abstraction for various logging frameworks.

🗀 MIT License SLF4J slf4j.org

Logback

A reliable, generic, fast and flexible logging framework.

🗀 LGPL 2.1 License Logback logback.qos.ch

Lombok

Automate Java boilerplate code via annotations.

🗀 LGPL 2.1 License Lombok projectlombok.org

OkHttp

OkHttp is an efficient HTTP client.

🗀 Apache-2.0 License OkHttp square.github.io/okhttp

MongoDB Driver

The MongoDB Synchronous Driver provides an easy API for interacting with a MongoDB Server.

🗀 Apache-2.0 License MongoDB Driver mongodb.com/docs/drivers/java/sync/current

Metadata Extractor

Drew Noakes' excellent library for extracting metadata (Exif/IPTC/XMP...) from image files.

🗀 Apache-2.0 License Metadata Extractor github.com/drewnoakes/metadata-extractor

What about $ATK's License?

We are currently working on that.