NOTICE: This software (or technical data) was produced for the U.S. Government under contract, and is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2023 The MITRE Corporation. All Rights Reserved.

OpenMPF 9.0.x

9.0.0: May 2024

Documentation

Created a new Quality Selection Guide.

Quality Selection

Can now specify a QUALITY_SELECTION_PROPERTY and QUALITY_SELECTION_THRESHOLD for choosing exemplars, artifacts, and controlling feed-forward behavior.
The following old job properties and old system properties are no longer supported. The tables show the new properties that should be used instead:

Old Job Property	New Job Properties
`CONFIDENCE_THRESHOLD`	`QUALITY_SELECTION_PROPERTY` `QUALITY_SELECTION_THRESHOLD`
`ARTIFACT_EXTRACTION_POLICY_TOP_CONFIDENCE_COUNT`	`ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT`
`FEED_FORWARD_TOP_CONFIDENCE_COUNT`	`FEED_FORWARD_TOP_QUALITY_COUNT`

Old System Property	New System Properties
`detection.confidence.threshold`	`detection.quality.selection.prop` `detection.quality.selection.threshold`
`detection.artifact.extraction.policy.top.confidence.count`	`detection.artifact.extraction.policy.top.quality.count`

By default, QUALITY_SELECTION_PROPERTY is set to the value of detection.quality.selection.prop system property, which, by default, is CONFIDENCE. In most cases this preserves the previous behavior.
By default, QUALITY_SELECTION_THRESHOLD is set to the value of detection.quality.selection.threshold system property, which, by default, is -Infinity. This setting disables the threshold. Previously, the default value of detection.confidence.threshold was -1, which disabled the threshold for most components.
Components that previously used CONFIDENCE_THRESHOLD now have QUALITY_SELECTION_PROPERTY=CONFIDENCE. Also, QUALITY_SELECTION_THRESHOLD is set to the previous value of CONFIDENCE_THRESHOLD. For example, see this commit for changes made to the OcvYoloDetection component.
EXEMPLAR_POLICY is now set to QUALITY by default. This setting results in choosing the detection within each track with the maximum quality according to the QUALITY_SELECTION_PROPERTY. Previously, the selection was always made based on highest detection confidence.
Similarly, the new FEED_FORWARD_TOP_QUALITY_COUNT and ARTIFACT_EXTRACTION_POLICY_TOP_QUALITY_COUNT properties use QUALITY_SELECTION_PROPERTY and QUALITY_SELECTION_THRESHOLD.
Refer to the Quality Selection Guide for details.

Transformer Tagging Component

This component uses a user-specified corpus JSON file to match known phrases against each sentence in the input text data.
The input text sentences that generate match scores above the threshold are called "trigger sentences". These sentences are grouped by "tag" based on which entry in the corpus they matched against.
The underlying all-mpnet-base-v2 model was trained on a variety of text data in order to understand the commonalities in phrasing, subject, and context.
Refer to the README for details.

Keyword Tagging Component Output

Updated the Keyword Tagging Component to generate output in the same format as the Transformer Tagging Component. For example, the output properties used to take the form <INPUT> TRIGGER WORDS and <INPUT> TRIGGER WORDS OFFSET:

TEXT TRIGGER WORDS
TEXT TRIGGER WORDS OFFSET
TRANSLATION TRIGGER WORDS
TRANSLATION TRIGGER WORDS OFFSET

Now the output properties take the form <INPUT> <TAG> TRIGGER WORDS and <INPUT> <TAG> TRIGGER WORDS OFFSET:

TEXT TRAVEL TRIGGER WORDS
TEXT TRAVEL TRIGGER WORDS OFFSET
TRANSLATION TRAVEL TRIGGER WORDS
TRANSLATION TRAVEL TRIGGER WORDS OFFSET

Notice that in the above example the new output properties include the word TRAVEL. If trigger words are detected for other tags, such as FINANCIAL and VEHICLE, those words will be used in separate TRIGGER WORDS and TRIGGER WORDS OFFSET output properties.
This change enables the job consumer to determine which trigger words are associated with each entry in the TAGS output property.
Refer to the "Outputs" section of the README for details.

Reporting Component Processing Time

The JSON output object contains a new section for reporting component processing time in milliseconds. For example:

"timing": {
  "processingTime": 1514,
  "actions": [
      {
        "name": "OCV YOLO VEHICLE DETECTION ACTION",
        "processingTime": 1431
      },
      {
        "name": "TENSORFLOW VEHICLE COLOR DETECTION (WITH FF REGION) ACTION",
        "processingTime": 83
      }
  ]
},

This does not include the time sub-jobs spent waiting in queues, or processing time by the Workflow Manager, such as the time to perform media inspection.
Also, the above JSON is reported in the TiesDB job record within the dataObject field.

NLP Text Splitter Utility

The new NLP Text Splitter utility uses spaCy or Where's the Point (WtP) models for determining how to break up text into sentences.
Supports both CPU processing and optional GPU processing.
Updated the Azure Translation Component to use this utility to ensure that translation requests are within the 50,000 character limit.
Refer to the README for details.

CLIP Component Video Support

The CLIP Component now supports processing videos in addition to the previous ability to process images. Specify the batch size using DETECTION_FRAME_BATCH_SIZE.
The component also supports a new, larger, and more accurate ViT-L/14 model in addition to the previous ViT-B/32 model. Both models are supported via the optional Triton server as well as within the component itself for non-Triton deployments.
Refer to the README for performance metrics.
The NUMBER_OF_TEMPLATES property has been renamed to TEMPLATE_TYPE and now accepts one of the following values: openai_1, openai_7, openai_80.

Import Root Certificates for Components

Can now specify a MPF_CA_CERTS environment variable for component Docker services to import root certificates.
May be useful when components need to communicate with external web services.
Refer to the README for details.

Docker Secrets for Environment Variables

Can now use Docker secrets for environment variables in the Docker compose file.
This prevents exposing information as plain text in docker-compose.yml.
May be useful for environment variables like:
- Workflow Manager username and password: WFM_USER and WFM_PASSWORD
- Keystore password when enabling Workflow Manager HTTPS: KEYSTORE_PASSWORD
- Azure credentials: MPF_PROP_ACS_URL and MPF_PROP_ACS_SUBSCRIPTION_KEY
Refer to the README for details.

Features

[#1692] Create a TransformerTagging component
[#1718] Support a QUALITY_SELECTION_PROP to specify how the WFM should choose an exemplar
[#1754] Report amount of time components spent executing a job
[#1756] Support MPF_CA_CERTS for components
[#1771] Azure Translation: Identify character limits. Split text using NLP Text Splitter.
[#1798] Add NLP Text Splitter to Python Component SDK

Updates

[#1694] Update CLIP component to support videos
[#1706] Update KeywordTagging to work with TransformerTagging
[#1745] Support using docker secrets for environment variables in docker-compose.yml
[#1769] Upgrade to proto3 and clean up .proto files
[#1774] Update how TransformerTagging tokenizes sentences
[#1785] Upgrade to OpenCV 4.9
[#1786] Modify the behavior of Markup when CONFIDENCE is the bounding box label to be displayed
[#1797] Further update Azure Translation and STT language maps
[#1803] Upgrade Postgres client used by Workflow Manager

Bug Fixes

[#1781] Markup boxes are not drawn when animation is disabled and there are gaps in a track
[#1799] Keyword Tagging removes newlines so character offsets don't line up with original text

OpenMPF 8.0.x

8.0.4: May 2024

Bug Fixes

[#1805] Workflow Manager incorrectly detects whether certain videos are constant or variable frame rate

8.0.3: April 2024

Bug Fixes

[#1788] Azure Speech and Translation: Update supported language mappings

8.0.2: March 2024

Documentation

Updated the REST API with new [GET] /rest/queues and [GET] /rest/queues/{name} endpoints.

Updates

[#1776] Add REST endpoint for retrieving the ActiveMQ message counts for each queue

8.0.1: March 2024

Updates

[#1768] Add Option to Merge Text Sections in TikaTextDetection

Bug Fixes

[#1763] Media inspections fails when ffprobe does not specify a stream "codec_type"

8.0.0: December 2023

Documentation

Created a new OpenID Connect Guide.
Updated the Admin Guide and User Guide to remove /workflow-manager from the Workflow Manager base URL. The Admin Guide includes a section for the new Hawtio web console.
Updated the REST API to use path parameters for pipelines, tasks, actions, and algorithms endpoints.
Updated the Component Descriptor Reference with algorithm.trackType.
Updated the C++ Batch Component API, Python Batch Component API, and Java Batch Component API to remove the ability to get the detection type since track type is now specified in descriptor.json.
Created a new Trigger Guide.
Created a new Roll Up Guide.

OpenID-Connect (OIDC) Authentication

The Workflow Manager can now optionally use an OpenID Connect (OIDC) provider to handle authentication for users of the web UI and clients of the REST API. The URI for the OIDC provider is specified using the OIDC_ISSUER_URI environment variable.
When enabled, OIDC is used to authenticate components when they register with the Workflow Manager.
When CALLBACK_USE_OIDC is set to true, the Workflow Manager will send a token in job request callbacks.
When TIES_DB_USE_OIDC is set to true, the Workflow Manager will send a token when posting to a TiesDb server.
When OIDC is not enabled, the Workflow Manager uses basic authentication with usernames and passwords, as in previous versions of OpenMPF.
Refer to the OpenID Connect Guide for more information on the various OIDC environment variables and a Keycloak example.

Embedded ActiveMQ Broker and Hawtio

ActiveMQ is now part of the Workflow Manager Spring Boot web application and is no longer run as a separate Docker service. This enables ActiveMQ to integrate with Spring Security so it can be protected by the Workflow Manager's OIDC support.
The Workflow Manager is the sender or recipient of all ActiveMQ messages, so embedding ActiveMQ in the Workflow Manager prevents a network hop on all messages.
The ActiveMQ management page has been replaced by Hawtio, which is more feature rich and can be used to monitor the state of the ActiveMQ queues used for communication between the Workflow Manager and the components. The Hawtio web console can be accessed by selecting "Hawtio" from the "Configuration" dropdown menu in the top menu bar of the web UI.
Importantly, the base URL for the Workflow Manager is now http://localhost:8080 instead of http://localhost:8080/workflow-manager. /workflow-manager is no longer part of the path. This change was made to enable Hawtio integration.

REST API Updates

The following changes have been made to the REST endpoints to address a limitation with Swagger (OpenAPI). These changes enable the REST endpoints to properly show up in the Swagger page, which is accessed by selecting "REST API" from the "Configuration" dropdown menu in the top menu bar of the web UI.

Old REST Endpoint	New REST Endpoint
[GET] /rest/pipelines?name={name}	[GET] /rest/pipelines/{name}
[GET] /rest/tasks?name={name}	[GET] /rest/tasks/{name}
[GET] /rest/actions?name={name}	[GET] /rest/actions/{name}
[GET] /rest/algorithms?name={name}	[GET] /rest/algorithms/{name}
[DELETE] /rest/pipelines?name={name}	[DELETE] /rest/pipelines/{name}
[DELETE] /rest/tasks?name={name}	[DELETE] /rest/tasks/{name}
[DELETE] /rest/actions?name={name}	[DELETE] /rest/actions/{name}

In general, the name is now specified as part of the URL path instead of as a URL parameter.
/ and ; characters are no longer allowed in these names.

Packaging and Deployment

Each component's descriptor.json now requires an algorithm.trackType field. This is used by the Workflow Manager to determine the kind of tracks that may be generated by the component (e.g. FACE, TEXT, CLASS, etc.). This is now used in place of the component API calls that were used to get the detection type.

Component API Updates

The following changes were made since the track type is now part of each component's descriptor.json:
- Removed GetDetectionType() from the CPP Component API.
- Removed detection_type from the Python Component API.
- Removed getDetectionType() from the Java Component API.

Changes to JSON Output Object

New JSON output objects use action instead of source in the track type group. Also, source is removed from each track.
Consider this example of the old JSON output:

"output": {
  "FACE": [
    {
      "source": "+#MOG MOTION DETECTION (WITH AUTO-ORIENTATION) PREPROCESSOR ACTION#OCV FACE DETECTION (WITH AUTO-ORIENTATION) ACTION",
      "algorithm": "FACECV",
      "tracks": [
        {
          "id": "4bcba9b95b92a5115b7da1097fcffa962480d0b4424a656772bef12161d775c1",
          "startOffsetFrame": 0,
          "stopOffsetFrame": 0,
          "startOffsetTime": 0,
          "stopOffsetTime": 0,
          "type": "FACE",
          "source": "+#MOG MOTION DETECTION (WITH AUTO-ORIENTATION) PREPROCESSOR ACTION#OCV FACE DETECTION (WITH AUTO-ORIENTATION) ACTION",
          "confidence": 8.799637,
          ...

The corresponding new JSON output is:

"output": {
  "FACE": [
    {
      "action": "OCV FACE DETECTION (WITH AUTO-ORIENTATION) ACTION",
      "algorithm": "FACECV",
      "tracks": [
        {
          "id": "4bcba9b95b92a5115b7da1097fcffa962480d0b4424a656772bef12161d775c1",
          "startOffsetFrame": 0,
          "stopOffsetFrame": 0,
          "startOffsetTime": 0,
          "stopOffsetTime": 0,
          "type": "FACE",
          "confidence": 8.799637,
          ...

Trigger Support

A TRIGGER property can now be added to any action in a pipeline. It will only be used if FEED_FORWARD_TYPE is provided and set to something other than NONE. The TRIGGER property is used to conditionally control if the Workflow Manager executes that action. Each feed-forward track that is not executed is passed to the next stage of the pipeline. This results in skipping untriggered actions.
The value of TRIGGER takes the form <prop_name>=<prop_value1>[;<prop_value2>...]. For example, if the value is CLASSIFICATION=car then the Workflow Manager would only execute the associated action using feed-forward tracks from the previous stage in the pipeline if those tracks have the CLASSIFICATION track property with a value of car. This could be useful to skip a license plate detection action. To enable the action to trigger on more than just car tracks you can provide a list of valid values. For example, CLASSIFICATION=car;truck;bus.
The Trigger Guide goes into more detail and provides an example of a pipeline with multiple speech-to-text stages. TRIGGER is used to select which speech-to-text algorithm is executed based on the detected language in the media.

Roll Up Support

The Workflow Manager can be configured to replace the values of track and detection properties after receiving tracks and detections from a component. For example, the CLASSIFICATION property may be set to "car", "bus", and "truck". Those can be rolled up into "vehicle".
To use this feature, set the ROLL_UP_FILE property to the path of a JSON file that matches the format of this example:

[
    {
        "propertyToProcess": "CLASSIFICATION",
        "originalPropertyCopy": "ORIGINAL CLASSIFICATION",
        "groups": [
            {
                "rollUp": "vehicle",
                "members": [
                    "truck",
                    "car",
                    "bus"
                ]
            }
       ]
    }
]

Refer to the Roll Up Guide for an explanation and more details.

Changed All "whitelist" References to "allow list"

In an effort to be more culturally sensitive, all references to "whitelist" have been removed or renamed to "allow list".
The whitelist. prefix has been removed from the entries in the mediaType.properties file. For example, whitelist.image/gif=VIDEO is now image/gif=VIDEO.
The OcvDnnDetection component FEED_FORWARD_WHITELIST_FILE property has been renamed to FEED_FORWARD_ALLOW_LIST_FILE.
The OcvYoloDetection component CLASS_WHITELIST_FILE property has been renamed to CLASS_ALLOW_LIST_FILE.

Argos Translation Component

This new component utilizes Argos Translate to translate input text from a given source language to English. It can be used in a feed-forward pipeline to process tracks with language and/or script identifiers from an upstream stage.
Refer to the README for details.

Whisper Speech-to-Text and Translation Component

This new component utilizes OpenAI Whisper to perform language detection, speech-to-text transcription, or speech translation.
If multiple languages are spoken in a single piece of media, language detection will detect only one of them.
Note that Whisper is not designed to return a transcription in the source language when performing translation, so we implemented the component to perform an additional transcribe call when configured to perform translation.
Refer to the README for details.

Contrastive Language–Image Pre-training (CLIP) Component

This new component utilizes CLIP to classify images using the 80 COCO classes, 1000 ImageNet classes, or a list of user-provided classes. It can run on a CPU or GPU, and can make calls to an NVIDIA Triton inference server.
Classification is performed by taking the class names and filling in one or more text prompts. For example, "a photo of {}", where "{}" can be "dog" or "cat". An embedding is generated using the text prompt(s) for each class and compared against the image embedding to get a match score. Optionally, users can provide a list of their own text prompts.
OpenAI trained the CLIP model using a wide variety of images and their respective captions from the Internet. This may make it suitable for a wide variety of classification tasks without further training (known as zero-shot classification). For example, a user could make up a list of classes for arbitrary objects like "walrus", "paperclip", "pizza", etc., and use the default text prompts.
It is also possible to use CLIP to classify concepts like scenes and sentiment. For example, using a text prompt of "a {} scene" where the classes are "safe", "violent", and "dangerous".
Optionally, the CLIP component can return the image embedding as the track FEATURE. For example, this can be used for search and retrieval tasks by comparing it to other embeddings enrolled in a database.
Refer to the README for details.

Features

[#1547] Create Argos translation component
[#1574] Update the WFM to support an optional TRIGGER property on any action
[#1598] Create a Whisper component for speech-to-text and and translation
[#1644] Create CLIP component for processing images
[#1704] Update Workflow Manager to authenticate users and REST clients using OIDC
[#1730] Update Workflow Manager to optionally use OIDC when sending callbacks and posting to TiesDb
[#1733] Update Workflow Manager to use an embedded ActiveMQ broker
[#1793] Add Roll Up support to Workflow Manager

Updates

[#799] Avoid unnecessary serialization between Camel routes
[#949] Change /pipelines?name=MYPIPELINE REST endpoint to /pipelines/MYPIPELINE
[#1643] Remove LONG_SPEAKER_ID and instead only use SPEAKER_ID
[#1645] Refactor camel code
[#1705] Change all references to "whitelist" to "allow list" and "blacklist" to "block list"
[#1759] Disable markup animation by default

Bug Fixes

[#1642] InProgressBatchJobsService.setProcessedAction is now called when a previous task produces no tracks
[#1755] The Workflow Manager logs page does not properly handle multi-byte characters

OpenMPF 7.2.x

7.2.6: January 2024

Documentation

Created a new Health Check Guide.

Health Check Support

The C++ and Python component executors can be configured to run health checks on components prior to running jobs. Health checks are configured using environment variables:
- HEALTH_CHECK: When set to "ENABLED", the component executor will run health checks.
- HEALTH_CHECK_TIMEOUT: When set to a positive integer, specifies the minimum number of seconds between health checks. When absent or set to 0, a health check will run before every job.
- HEALTH_CHECK_RETRY_MAX_ATTEMPTS: When set to a positive integer, specifies the number of consecutive health check failures that will cause the component service to exit. When absent or set to 0, the component service will never exit because of a failed health check.
Also, an INI file must be provided at $MPF_HOME/plugins/<component-name>/health/health-check.ini. For example:

media=$MPF_HOME/plugins/OcvFaceDetection/health/meds_faces_image.png
min_num_tracks=2
media_type=IMAGE

[job_properties]
JOB PROP1=VALUE1
JOB PROP2=VALUE2

[media_properties]
MEDIA PROP=MEDIA VALUE

Refer to the Health Check Guide for an explanation and more details.

Features

[#1731] Implement health checks for C++ and Python components

Updates

[#1727] Update ffmpeg to 6.1

7.2.5: November 2023

Updates

[#1715] Upgrade ActiveMQ to 5.17.6

Bug Fixes

[#1711] When selecting detections with the highest confidence, Workflow Manager should consistently handle detections with equal confidence

7.2.4: September 2023

Bug Fixes

[#1707] Fix bug where TiesDB check status reports NO_TIES_DB_URL_IN_JOB instead of MEDIA_MIME_TYPES_ABSENT

7.2.3: June 2023

Bug Fixes

[#1697] Prevent OcvYoloDetection component from deadlocking on strange frame sizes when using Triton

7.2.2: June 2023

Updates

[#1693] Add property to enable/disable SAS in AzureSpeech

Bug Fixes

[#1695] Fix memory leak in KeywordTagging component

7.2.1: June 2023

Bug Fixes

[#1678] Fix bug where ffmpeg hangs when processing some kinds of unsupported/corrupted media

7.2.0: May 2023

Documentation

Created a new TiesDb Guide.
Updated the Component Descriptor Reference with outputChangedCounter.
Updated the REST API with a new [POST] /rest/jobs/tiesdbrepost endpoint.
Updated the REST API [POST] /rest/jobs response with tiesDbCheckStatus and outputObjectUri.

TiesDb Re-Post

Added a new [POST] /rest/jobs/tiesdbrepost endpoint that accepts an array of job ids as an input and will attempt to re-post the job assertions (records) to TiesDb for each one.
Added a "TiesDb" column to the Job Status page. If there is a problem posting a record to the TiesDb server the column will contain an "ERROR" button. Clicking on it will provide a description of the error and a button that can be used to re-post the associated job records.

TiesDb Checking

If the TIES_DB_URL job property or ties.db.url system property is set when submitting a job creation request, then the Workflow Manager will attempt to check TiesDb for existing job results before running the job again.
The Workflow Manager will attempt to use the most-recently-created job results, preferring jobs that completed without errors or warnings, and preferring jobs that completed with warnings over completed with errors.
To prevent this check, set SKIP_TIES_DB_CHECK=true. That will force the job to run and attempt to post the new job results to TiesDb.
When using TiesDb, we strongly recommend providing both the MEDIA_HASH and MIME_TYPE in the media.metadata map in the job request. This will enable the Workflow Manager to skip media inspection. When using S3 object storage, this means that the Workflow Manager will not need to download the media before checking TiesDb for existing job records.
The [POST] /rest/jobs response now contains a tiesDbCheckStatus and outputObjectUri field. tiesDbCheckStatus will be set to one of the following values:
- NOT_REQUESTED
- NO_TIES_DB_URL_IN_JOB
- MEDIA_HASHES_ABSENT
- MEDIA_MIME_TYPES_ABSENT
- NO_MATCH
- FOUND_MATCH
When there is a FOUND_MATCH, the outputObjectUri will be set to the URI of the old TiesDb record if S3 copy is not enabled.
By default, the ties.db.s3.copy.enabled system property is set to true. This means that the Workflow Manager will attempt to copy all of the artifacts, markup, and derivative media associated with the job in TiesDb from the S3 locations associated with the old job to the new S3 location specified in the new job. A new JSON output object will be generated. To disable this behavior set the system property, or TIES_DB_S3_COPY_ENABLED, to false. Then the Workflow Manager will simply provide a link to the old JSON as the result of the new job.
If there is a problem copying between S3 locations, the "TiesDb" column to the Job Status page will show a "COPY ERROR" button. Clicking on it will provide a description of the error.

TiesDb Linked Media

Added support for LINKED_MEDIA_HASH in the media.properties section of the job creation request. When specified, the value of LINKED_MEDIA_HASH will be used instead of the actual media hash when creating a record in TiesDb, and also when looking for existing records in TiesDb.
This feature can be used to submit a transcoded (or thumbnail) version of an image to process instead of the source image. For example, the source image may be in a format not supported by OpenMPF. In this case, the value of LINKED_MEDIA_HASH can be set to the source image, but the rest of the job creation request would specify the media.mediaUri and media.metadata for the transcoded version of that image.

Output Changed Counter

Added the output.changed.counter system property to the Workflow Manager and outputChangedCounter field to each component's descriptor.json. These values are used when calculating the hash for a job when its record is posted to TiesDb, and also when checking TiesDb for existing records when a new job is submitted.
If the Workflow Manager is updated for any reason that should invalidate pre-existing job results, such as a change to the fields in the JSON output object, or significant improvements to track merging, for example, then the value of output.changed.counter should be incremented by one. This will ensure that records in TiesDb will not be used so that all future jobs will need to be (re)run at least once until the counter is incremented again.
The same is true for each component. If a component is updated for any reason that should invalidate pre-existing job results, such as changes to input or output properties, or substantial improvements to the algorithm, then the value of outputChangedCounter should be incremented by one.

Changes to JSON Output Object

New JSON output objects will include tiesDbSourceJobId and tiesDbSourceMediaPath when the Workflow Manager can use previous job results stored in TiesDB. Note that the Workflow Manager will not generate a new JSON output object unless S3_RESULTS_BUCKET is set to a valid value, S3 access and secret keys are provided, and TIES_DB_S3_COPY_ENABLED=true.

ffprobe for Media Inspection

The Workflow Manager media inspection behavior now uses ffprobe with -print_format json to return more precise FPS values for the media.mediaMetadata in the JSON output object. For example, the previous version of the Workflow Manager would return 29.97, where the new version will return 29.97002997002997. In multi-hour-long vidoes this can prevent cases where the last few frames were being ignored.
The previous version of the Workflow Manager was using both ffmpeg and OpenCV to determine the number of frames in a video. We removed the OpenCV frame counter in this version because the ffprobe approach is more accurate. The ffprobe command replaces the old ffmpeg command.

Web User Interface

Updated the Job Status page to be more efficient. Searching a database of hundreds of thousands of jobs takes a long time. By limiting the search to one page of results at a time the UI is more responsive.
Removed timeout and bootout. The user session will no longer automatically end due to time out, or due to the same user logging in from a different host or browser. These behaviors were deemed too disruptive by end users.
Updated the Job Status page to include a "TiesDb" column that reports TiesDb status, such as when posting records to TiesDb and when retrieving existing records.

Features

[#1438] Create a REST endpoint that will attempt to re-post to TiesDb
[#1613] Check TiesDb before running a job
[#1650] Create TiesDb records for thumbnail jobs under the parent media

Updates

[#1342] Use ffprobe to get FPS during media inspection
[#1564] Use ffprobe's JSON output instead of regexes during media inspection
[#1601] Update the Workflow Manager jobs table to be more efficient
[#1611] Remove Workflow Manager timeout and bootout behavior

OpenMPF 7.1.x

7.1.12: March 2023

Bug Fixes

[#1667] Handle Webp files with extra data at the end that cause components to crash

7.1.10: March 2023

Updates

[#1662] Monitor StorageBackend

7.1.9: February 2023

Bug Fixes

[#1675] Prevent upgrade of cudnn in yolo server dockerfile

7.1.8: February 2023

Bug Fixes

[#1649] Install specific version of libcudnn8 in Docker build

7.1.7: February 2023

Updates

[#1674] Update SPEAKER_ID logic, set LONG_SPEAKER_ID=0

7.1.5: January 2023

Features

[#1542] Update Azure Speech Detection component to select transcription language based on feed-forward track
[#1543] Update audio transcoder to accept subsegments
[#1605] Update Azure Translation to use detected language from upstream

7.1.1: December 2022

Bug Fixes

[#1634] Update version numbers to 7.1

7.1.0: December 2022

Documentation

Updated the Object Storage Guide with S3_UPLOAD_OBJECT_KEY_PREFIX.
Updated the Markup Guide with MARKUP_TEXT_LABEL_MAX_LENGTH.

Exemplar Selection Policy

The policy for selecting the exemplar detection for each track can now be set using the EXEMPLAR_POLICY job property with following values:
- CONFIDENCE: Select the detection with the maximum confidence. If some confidences are the same, select the detection with the lower frame number. This is the default setting.
- FIRST: Select the detection with the lowest frame number
- LAST: Select the detection with the highest frame number
- MIDDLE: Select the detection with the frame number closest to the middle frame of the track, preferring the detection with the lower frame number if there is an even number of frames

Automatic Rotation and Horizontal Flip Enabled by Default

It is no longer necessary to explicitly set AUTO_ROTATE and AUTO_FLIP to true since that is now the default value.
These properties affect all video and image components that use the MPFImageReader and MPFVideoCapture tools. When true, if the image has EXIF data, or there is metadata associated with a video that ffmpeg understands, the tools will use that information to properly orient the frames before returning the frames to the component for processing.

Support S3 Object Storage Key Prefix

Set the S3_UPLOAD_OBJECT_KEY_PREFIX job property or s3.upload.object.key.prefix system property to add a prefix to object keys when the Workflow Manager uploads objects to the S3 object store. This affects the JSON output object, artifacts, markup files, and derivative media.
Specifically, the Workflow Manager will upload objects to <S3_RESULTS_BUCKET>/<S3_UPLOAD_OBJECT_KEY_PREFIX><file-hash-first-two-chars>/<file-hash-second-two-chars>/<file-hash>.
For example, if you wish to add "work/" to the object key, then set S3_UPLOAD_OBJECT_KEY_PREFIX=work/.

Features

[#1526] Allow markup to display more than 10 characters in the text part of the label
[#1527] Enable the Workflow Manager to select the middle detection as the exemplar
[#1566] Make AUTO_ROTATE and AUTO_FLIP true by default
[#1569] Modify C++ and Python component executor to automatically add the job name to log messages
[#1621] Make S3 object keys used for upload configurable

Updates

[#1602] Update Workflow Manager to use Spring Boot
[#1631] Update byte-buddy, Mockito, and Hibernate versions to resolve build issue. Most notably, update Hibernate to 5.6.14.
[#1632] Update ActiveMQ to 5.17.3

Bug Fixes

[#1581] Don't change track start and end frame when FEED_FORWARD_TOP_CONFIDENCE_COUNT is disabled
[#1595] Work around how Ubuntu only recognizes certificate files that end in .crt
[#1610] Prevent premature pipeline creation when using web UI
[#1612] At startup, prevent Workflow Manager from consuming from queues before purging them

OpenMPF 7.0.x

7.0.3: September 2022

Bug Fixes

[#1561] Fix logging for Python components when running through CLI runner
[#1583] Can now properly view media while job is in progress
[#1587] Fix bugs in amq_detection_component's use of select

7.0.2: August 2022

Bug Fixes

[#1562] Fix bug where an ffmpeg change prevented detecting video rotation

7.0.0: July 2022

Documentation

Updated the Development Environment Guide by replacing steps for CentOS 7 with Ubuntu 20.04.
Added the Derivative Media Guide.
Updated the Batch Component APIs with revised error codes.
Updated the Python Batch Component API and Python base Docker image README with instructions for using pyproject.toml and setup.cfg.
Updated the Admin Guide and User Guide with images that show the new TiesDb and Callback columns in the job status UI.
Updated the REST API with the pipelineDefinition, frameRanges, and timeRanges fields now supported by the [POST] /rest/jobs endpoint.
Updated the OcvYoloDetection component README with information on using the NVIDIA Triton inference server.
Updated the Markup Guide with MARKUP_ANIMATION_ENABLED and MARKUP_LABELS_TRACK_INDEX_ENABLED.
Updated the Contributor Guide with new steps for generating documentation.

Transition from CentOS 7 to Ubuntu 20.04

All the Docker images that previously used CentOS 7 as a base now use Ubuntu 20.04.
We decided not to use CentOS 8, which is a version of CentOS Stream, due to concerns about stability.
Also, Ubuntu is a very common OS within the AI and ML space, and has significant community support.

Use Job Id that Enables Load Balancing

The Workflow Manager can now optionally accept job ids of the form <openmpf-instance-wfm-hostname>-<job-id> through the REST endpoints, where <job-id> is the same as the shorter id used in previous releases. The <openmpf-instance-wfm-hostname>- prefix enables better tracking and separation of jobs run across multiple Workflow Manager instances in a cluster.
The prefix can be set in the docker-compose.yml file by assigning {{.Node.Hostname}} to the NODE_HOSTNAME environment variable for the Workflow Manager service, or hard-coding NODE_HOSTNAME to the desired hostname.
The shorter version of the id can still be used in REST requests, but the longer id will always be returned by the Workflow Manager when responding to those requests.
The shorter id will always be used internally by the Workflow Manager, meaning the job status web UI and log messages will all use the shorter job id.

Support for Derivative Media

The TikaImageDetection component now returns MEDIA tracks instead of IMAGE tracks when extracting images from documents, such as PDFs, Word documents, and PowerPoint slides. The document is considered the "source", or "parent", media, and the images are considered the "derivative", or "child", media.
Actions can now be configured with SOURCE_MEDIA_ONLY=true or DERIVATIVE_MEDIA_ONLY=true, which will result in only performing the action on that kind of media. Feed forward can still be used to pass track information from one stage to another. The tracks will skip the stages (actions) that don't apply.
This enables complex pipelines like one that extracts text from a PDF using TikaTextDetection, OCRs embedded images using EastTextDetection and TesseractOCRTextDetection, and runs all of the TEXT tracks through KeywordTagging.
Added the following pipelines to the TikaImageDetection component:
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA TESSERACT OCR PIPELINE
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA TESSERACT OCR AND KEYWORD TAGGING PIPELINE
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA TESSERACT OCR (WITH EAST REGIONS) AND KEYWORD TAGGING PIPELINE
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA TESSERACT OCR (WITH EAST REGIONS) AND KEYWORD TAGGING AND MARKUP PIPELINE
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA OCV FACE PIPELINE
- TIKA IMAGE DETECTION WITH DERIVATIVE MEDIA OCV FACE AND MARKUP PIPELINE

Report when Job Callbacks and TiesDb POSTs Fail

The job status UI displays two new columns, one that indicates the status of posting to TiesDB, and one that indicates the status of posting the job callback to the job producer.
Additionally, the [GET] /rest/jobs/{id} endpoint now includes a tiesDbStatus and callbackStatus field.
Note that, by design, the JSON output itself does not contain these statuses.

Allow Pipelines to be Specified in a Job Request

Optionally, the pipelineDefinition field can be provided instead of the pipelineName field when using the [POST] /rest/jobs endpoint in order to specify a pipeline on the fly for that specific job run. It will not be saved for later reuse.
The format of the pipeline definition is similar to that in a descriptor.json file, with separate sections for defining tasks and actions. Pre-existing tasks and actions known to the Workflow Manager can be specified in the definition. They do not need to be defined again.
This feature is a convenient alternative to creating persistent definitions using the [POST] /rest/pipelines, [POST] /rest/tasks, and [POST] /rest/actions endpoints. For example, this feature could be used to quickly add or remove a motion preprocessing stage from a pipeline.

Allow User-Specified Segment Boundaries

Optionally, multiple frameRanges and/or timeRanges fields can be provided when using the [POST] /rest/jobs endpoint in order to manually specify segment boundaries. These values will override the normal segmenting behavior of the Workflow Manager.
Note that overlapping ranges will be combined and large ranges may still be split up according to the value of TARGET_SEGMENT_LENGTH and VFR_TARGET_SEGMENT_LENGTH.
Note that frameRanges is specified using the frame number and timeRanges is specified in milliseconds.

Add Triton Inference Server support to YOLO component

The OcvYoloDetection component now supports the ability to send requests to an NVIDIA Triton Inference Server by setting ENABLE_TRITON=true. If set to false, the component will process jobs using OpenCV DNN on the local host running the Docker service, as per normal.
By default TRITON_SERVER=ocv-yolo-detection-server:8001, which corresponds to the ocv-yolo-detection-server entry in your docker-compose.yml file. Refer to the example entry within docker-compose.components.yml . That entry uses a pre-built and pre-configured version of the Triton server.
The Triton server runs the YOLOv4 model within the TensorRT framework, which performs a warmup operation when the server starts up to determine which optimizations to enable for the available GPU hardware. *.engine files are generated within the yolo_engine_file Docker volume for later reuse.
To further improve inferencing speed, shared memory can be configured between the ocv-yolo-detection client service and the ocv-yolo-detection-server service if they are running on the same host. Set TRITON_USE_SHM=true and configure the server with a /dev/shm:/dev/shm Docker volume.
Depending on the available GPU hardware, the Triton server can achieve speeds that are 5x faster than OpenCV DNN with tracking enabled, no shared memory, and nearly 9x faster with tracking disabled, with shared memory. Our tests used a single RTX 2080 GPU.

Removed Unused and Redundant Error Codes

The error codes shown on the left were redundant and replaced with the corresponding error codes on the right:

Old Error Code	New Error Code
MPF_IMAGE_READ_ERROR	MPF_COULD_NOT_READ_MEDIA
MPF_BOUNDING_BOX_SIZE_ERROR	MPF_BAD_FRAME_SIZE
MPF_JOB_PROPERTY_IS_NOT_INT	MPF_INVALID_PROPERTY
MPF_JOB_PROPERTY_IS_NOT_FLOAT	MPF_INVALID_PROPERTY
MPF_INVALID_FRAME_INTERVAL	MPF_INVALID_PROPERTY
MPF_DETECTION_TRACKING_FAILED	MPF_OTHER_DETECTION_ERROR_TYPE

Also, the following error codes are no longer being used and have been removed:

MPF_UNRECOGNIZED_DATA_TYPE
- All media types can now be processed since we support the UNKNOWN (a.k.a. "generic") media type
MPF_INVALID_DATAFILE_URI
- The Workflow Manager will reject a job with an invalid media URI before it gets to a component
MPF_INVALID_START_FRAME
MPF_INVALID_STOP_FRAME
MPF_INVALID_ROTATION

Markup Improvements

By default, the Markup component draws bounding boxes to fill in the gaps between detections in each track by interpolating the box size and position. This can now be disabled by setting the job property MARKUP_ANIMATION_ENABLED=false, or the system property markup.video.animation.enabled=false. Disabling this feature can be useful to prevent floating boxes from cluttering the marked-up frames.
The Markup component will now start each bounding box label with a track index like [0] that can be used to correlate the box with the track in the JSON output object. The JSON output now contains an index field for every track, relative to each piece of media, that is simply an integer that starts at 0 and counts upward. This can be disabled by setting the job property MARKUP_LABELS_TRACK_INDEX_ENABLED=false, or the system property markup.labels.track.index.enabled=false.

Changes to JSON Output Object

Components that generate MEDIA tracks will result in new derivative media entries in the JSON output file. This means it's possible to provide a single piece of media as an input and have more than one media entry in the JSON output. The output will always include the original media.
Each media entry in the JSON output now contains a parentMediaId in addition to the mediaId. The parentMediaId for original source media will always be set to -1; otherwise, for derivative media, the parentMediaId is set the mediaId of the source media from which the child media was derived.
Each media entry also contains a new frameRanges and timeRanges collection.
The JSON output file also contains a new index field for every track, relative to each piece of media.

Features

[#792] Perform detection on images extracted from PDFs
[#1283] Add user-specified segment boundaries
[#1374] Transition from CentOS 7 to Ubuntu 20.04
[#1396] Report when job callbacks and TiesDb POSTs fail
[#1398] Add Triton Inference Server support to YOLO component
[#1428] Allow pipelines to be specified in a job request
[#1454] Transition from Clair scans to Trivy scans
[#1485] Use pyproject.toml and setup.cfg instead of setup.py

Updates

[#803] Update Tika Image Detection to generate one track per piece of extracted media
[#808] Update Tika Text Detection component to not use leading zeros for PAGE_NUM
[#1105] Remove dependency on QT from C++ SDK
[#1282] Use job id that enables load balancing
[#1303] Update Tika Image Detection to return MEDIA tracks
[#1319] Review existing error codes and remove unused or redundant error codes
[#1384] Update Apache Tika to 2.4.1 for TikaImageDetection and TikaTextDetection Components
[#1436] CLI Runner should initialize a component once when handling multiple jobs
[#1465] Remove YoloV3 support from OcvYoloDetection component
[#1513] Update to Spring 5.3.18
[#1528] CLI runner should also sort by startOffsetTime
[#1540] Upgrade to Java 17
[#1549] Allow markup animation to be disabled
[#1550] Add track index to markup

Bug Fixes

[#1372] Tika Image Detection no longer misses images in PowerPoint and Word documents
[#1449] Simon data is now refreshed when clicking the Processes tab
[#1495] Fix bug where invalid CSRF token found for /workflow-manager/login

OpenMPF 6.3.x

6.3.14: May 2022

Bug Fixes

[#1530] Fix S3 code memory leak

6.3.12: April 2022

Updates

[#1519] Upgrade to OpenCV 4.5.5

Bug Fixes

[#1520] S3 code now retries on most 400 errors

6.3.11: April 2022

Documentation

Updated the Object Storage Guide with S3_SESSION_TOKEN, S3_USE_VIRTUAL_HOST, S3_HOST, and S3_REGION.

Updates

[#1496] Update S3 client code
[#1514] Update Tomcat to 8.5.78

6.3.10: March 2022

Bug Fixes

[#1486] Fix bug where MOVING was being added to immutable map twice
[#1498] Can now provide media metadata when frameTimeInfo is missing
[#1501] MPFVideoCapture now properly reads frames from videos with rotation metadata
[#1502] Detections with HORIZONTAL_FLIP will no longer result in illformed detections and incorrectly padded regions
[#1503] Videos with rotation metadata will no longer result in corrupt markup

6.3.8: January 2022

Bug Fixes

[#1469] TENSORFLOW VEHICLE COLOR DETECTION pipelines no longer refer to YOLO tasks that no longer exist

6.3.7: January 2022

Updates

[#1466] Upgrade log4j to 2.17.1

6.3.6: December 2021

Updates

[#1457] Upgrade log4j to 2.16.0

6.3.5: November 2021

Updates

[#1451] Make concurrent callbacks configurable

6.3.4: November 2021

Bug Fixes

[#1441] Modify AdminStatisticsController so that it doesn't hold all jobs in memory at once

6.3.3: October 2021

Features

[#1425] Make protobuf size limit configurable

6.3.2: October 2021

Bug Fixes

[#1420] Sphinx component no longer omits audio at end of video files
[#1422] Media inspection now correctly calculates milliseconds from ffmpeg duration

6.3.1: September 2021

Features

[#1404] Improve OcvDnnDetection vehicle color detection

Updates

[#1251] Add version to JSON output object
[#1272] Update Keyword Tagging to work on multiple inputs
[#1350] Retire old components to the graveyard: DlibFaceDetection, DarknetDetection, and OcvPersonDetection

Bug Fixes

[#1010] mpf.output.objects.enabled now behaves as expected
[#1271] Azure speech component no longer omits audio at end of video files
[#1389] NLP text correction component now properly reads the value of FULL_TEXT_CORRECTION_OUTPUT
[#1403] Corrected README to state that the Azure Speech Component doesn't support v2 of the API
[#1406] Speech detections in videos are no longer dropped if using keyword tagging
[#1411] Exception no longer occurs when adding SHRUNK_TO_NOTHING=TRUE to an immutable map in multiple pipeline stages
[#1413] Speech detections in videos are no longer dropped if using translation

6.3.0: September 2021

Documentation

Updated the API documents, Development Environment Guide, Node Guide, Install Guide, User Guide, Admin Guide, and others to clarify the difference between Docker and non-Docker behaviors.
Transformed Packaging and Registering a Component document into Component Descriptor Reference.
Split Media Segmentation Guide from User Guide.
Updated and renamed the Workflow Manager document to Workflow Manager Architecture.
Updated the various Docker guides to clarify the difference between building Docker images from scratch versus building them using pre-built base images on Docker Hub, emphasizing the latter.
Updated the Contributor Guide to document the hotfix pull request process.

TiesDb Integration

TiesDb is a PostgreSQL DB with a RESTful API that stores media metadata. The metadata entries are queried using the hash (sha256, md5) of the media file. TIES stands for Triage Import Export Schema. TiesDb is deployed and managed externally to OpenMPF. For more information please contact us.
When a job completes, OpenMPF can post assertions to media entries that exist in TiesDb. In general, one assertion is generated for each algorithm run on a piece of media. It contains the job status, algorithm name, detection type (FACE, TEXT, MOTION, etc.), and number of tracks generated, as well as a link to the full JSON output object.
Each assertion serves as a lasting record so that job producers may first check TiesDb to see if an algorithm was run on a piece of media before submitting the same job to OpenMPF again.
To enable TiesDb support, set the TIES_DB_URL job property or ties.db.url system property to the <http|https>://<host>:<port> part of the URL. The Workflow Manager will append the /api/db/supplementals?sha256Hash=<hash> part. Here is an example of a TiesDb POST:

{
  "dataObject": {
    "sha256OutputHash": "1f8f2a8b2f5178765dd4a2e952f97f5037c290ee8d011cd7e92fb8f57bc75f17",
    "outputType": "FACE",
    "algorithm": "FACECV",
    "processDate": "2021-09-09T21:37:30.516-04:00",
    "pipeline": "OCV FACE DETECTION PIPELINE",
    "outputUri": "file:///home/mpf/git/openmpf-projects/openmpf/trunk/install/share/output-objects/1284/detection.json",
    "jobStatus": "COMPLETE",
    "jobId": 1284,
    "systemVersion": "6.3",
    "trackCount": 1,
    "systemHostname": "openmpf-master"
  },
  "system": "OpenMPF",
  "securityTag": "UNCLASSIFIED",
  "informationType": "OpenMPF FACE",
  "assertionId": "4874829f666d79881f7803207c7359dc781b97d2c68b471136bf7235a397c5cd"
}

Natural Language Processing (NLP) Text Correction Component

This component utilizes the CyHunspell library, which is a Python port of the Hunspell spell-checking library, to perform post-processing correction of OCR text. In general, it's intended to be used in a pipeline after a component like TesseractOCRTextDetection that generates TEXT tracks. These tracks are then fed-forward into NlpTextCorrection, which will add a CORRECTED TEXT property to the existing tracks. The TESSERACT OCR TEXT DETECTION WITH NLP TEXT CORRECTION PIPELINE performs this behavior. The component can also run on its own to process plain text files. Refer to the README for details.

Azure Cognitive Services (ACS) Read Component

This component utilizes the Azure Cognitive Services Read Detection REST endpoint to extract formatted text from documents (PDFs), images, and videos. Refer to the README for details.

Updates

[#1151] Now supports IN_PROGRESS_WITH_WARNINGS status
[#1234] Now sorts JSON output object media by media id
[#1341] Added job id to all batch-job-specific Workflow Manager log messages
[#1349] Improved reporting and recording job status
[#1353] Updated the Workflow Manager to remove and warn about zero-size detections
[#1382] Updated Tika version to 1.27 for TikaImageDetection and TikaTextDetection components
[#1387] Markup can now be configured in a component's descriptor.json

Bug Fixes

[#1080] Batch jobs no longer prematurely set to 100% completion during artifact extraction
[#1106] When a job ends in ERROR or CANCELLED_BY_SHUTDOWN the job status UI now shows an End Date
[#1158] JSON output object URI no longer changes when callback fails
[#1317] TikaTextDetection no longer generates first PDF track at PAGE_NUM 2
[#1337] Now using MPF_BAD_FRAME_SIZE instead of MPF_DETECTION_FAILED for OpenCV empty/resize exception
[#1359] Image detection tracks no longer have endOffsetFrameInclusive set to 1
[#1373] When uploading large files through the Workflow Manager web UI, now more than the first 865032704 bytes get written
[#1379] TikaImageDetection component now avoids conflicts by no longer using the same path when extracting images for jobs with multiple pieces of media
[#1386] FeedForwardFrameCropper in the Python SDK now handles negative coordinates properly
[#1391] If a job is configured to upload markup and markup fails, the job no longer gets stuck

Known Issues

[#1372] TikaImageDetection misses images in PowerPoint and Word documents
[#1389] NlpTextCorrection does not properly read the value of FULL_TEXT_CORRECTION_OUTPUT

OpenMPF 6.2.x

6.2.5: July 2021

Updates

[#1367] Enable cross-origin resource sharing on Workflow Manager

6.2.4: June 2021

Bug Fixes

[#1356] AzureSpeech now properly reports when media is missing audio stream
[#1357] AzureSpeech now handles case where speaker id is not present

6.2.2: June 2021

Updates

[#1333] Combine media name and job id into one WFM log line
[#1336] Remove duplicate "Setting status of job to COMPLETE" Workflow Manager log line and other improvements
[#1338] Update OpenCV DNN Detection component to optionally use feed-forward confidence values

Bug Fixes

[#1237] Fixed jQuery DataTables bug: "int parameter 'draw' is present but cannot be translated into a null value"
[#1254] Jobs table no longer flickers when polling is enabled and the search box is used
[#1308] Prevent OCV YOLO Tracking from generating zero-sized detections
[#1313] Fix JSON output object timestamps for variable frame rate videos

6.2.1: May 2021

Updates

[#1330] Return error codes for models_ini_parser.py exceptions

Bug Fixes

[#1331] Decoding certain heic images no longer causes Workflow Manager to segfault

6.2.0: May 2021

Tesseract OCR Text Detection Component Support for Videos

The component can now process videos in addition to images and PDFs. Each video frame is processed sequentially. The MAX_PARALLEL_SCRIPT_THREADS property determines how many threads to use to process each frame, one thread per language or script.
Note that for videos without much text, it may be faster to disable threading by setting MAX_PARALLEL_SCRIPT_THREADS=1. This will allow the component to reuse TessAPI instances instead of creating new ones for every frame. Please refer to the Known Issues section.
Resolved issues: #1285

Updates

[#1086] Added support for COULD_NOT_OPEN_MEDIA and COULD_NOT_READ_MEDIA error types
[#1159] Split IssueCodes.REMOTE_STORAGE into REMOTE_STORAGE_DOWNLOAD and REMOTE_STORAGE_UPLOAD
[#1250] Modified /rest/jobs/{id} to include the job's media
[#1312] Created NETWORK_ERROR error code for when a component can't connect to an external server. Updated Python HTTP retry code to return NETWORK_ERROR. This affects the Azure components.

Known Issues

[#1008] Use global TessAPI instances with parallel processing

OpenMPF 6.1.x

6.1.6: May 2021

Handle Variable Frame Rate Videos

The Workflow Manager will attempt to detect if a video is constant frame rate (CFR) or variable frame rate (VFR) during media inspection. If no determination can be made, it will default to VFR behavior. If CFR, the JSON output object will have a HAS_CONSTANT_FRAME_RATE=true property in the mediaMetadata field.
When MPFVideoCapture handles a CFR video it will use OpenCV to set the frame position, unless the position is within 16 frames of the current position, in which case it will iteratively use OpenCV grab() to advance to the desired frame.
When MPFVideoCapture handles a VFR video it will always iteratively use OpenCV grab() to advance to the desired frame because setting the frame position directly has been shown to not work correctly on VFR videos.
When a video is split into multiple segments, MPFVideoCapture must iteratively use grab() to advance from frame 0 to the start of the segment. This introduces performance overhead. To mitigate this we recommend using larger video segments than those used for CFR videos.
In addition to the existing TARGET_SEGMENT_LENGTH and MIN_SEGMENT_LENGTH job properties (detection.segment.target.length and detection.segment.minimum.length system properties) for CFR videos, the Workflow Manager now supports the VFR_TARGET_SEGMENT_LENGTH and VFR_MIN_SEGMENT_LENGTH job properties (detection.vfr.segment.target.length and detection.vfr.segment.minimum.length system properties) for VFR videos.
Note that the timestamps associated with tracks and detections in a VFR video may be wrong. Please refer to the Known Issues section.
Resolved issues: #1307

Updates

[#1287] Updated Tika Text Detection Component to break up large chunks of text. The component now generates tracks with both a PAGE_NUM property and SECTION_NUM property. Please refer to the README.

Known Issues

[#1313] Incorrect JSON output object timestamps for variable frame rate videos
[#1317] Tika Text Detection component generates first PDF track at PAGE_NUM 2

6.1.5: April 2021

Updates

[#1300] Parallelized S3 artifact upload. Use the detection.artifact.extraction.parallel.upload.count system property to configure the number of parallel uploads.

6.1.4: April 2021

Updates

[#1299] Improved artifact extraction performance when there is no rotation or flip

6.1.3: April 2021

Updates

[#1295] Improved artifact extraction and markup JNI memory utilization
[#1297] Limited Workflow Manager IO threads to a reasonable number

Bug Fixes

[#1296] Fixed ActiveMQ job priorities

6.1.2: April 2021

Updates

[#1294] Limited ffmpeg threads to a reasonable number

6.1.1: April 2021

Bug Fixes

[#1292] Don't skip artifact extraction for failed media

6.1.0: April 2021

OpenMPF Command Line Runner

The Command Line Runner allows users to run jobs with a single component without the Workflow Manager.
It outputs results in a JSON structure that is a subset of the regular OpenMPF output.
It only supports C++ and Python components.
See the README for more information.

C++ Batch Component API

Component code should no longer configure Log4CXX. The component executor now handles configuring Log4CXX. Component code should call log4cxx::Logger::getLogger("<component-name>") to get access to the logger. Calls to log4cxx::xml::DOMConfigurator::configure(logconfig_file); should be removed.

Python Batch Component API

Component code should no longer configure logging. The component executor now handles configuring logging. Calls to mpf.configure_logging should be replaced with logging.getLogger('<component-name>').

Docker Component Base Images

In order to support running a component through the CLI runner, C++ component developers should set the LD_LIBRARY_PATH environment variable in the final stage of their Dockerfiles. It should generally be set like: ENV LD_LIBRARY_PATH $PLUGINS_DIR/<component-name>/lib.
Because of the logging changes mentioned above, components no longer need to set the COMPONENT_LOG_NAME environment variable in their Dockerfiles.
Added the openmpf_python_executor_ssb base image . It can be used instead of openmpf_python_component_build and openmpf_python_executor to simplify Dockerfiles for Python components that are pure Python and have no build time dependencies.

Label Moving vs. Non-Moving Tracks

The Workflow Manager can now identify whether a track is moving or non-moving. This is determined by calculating the average bounding box for a track by averaging the size and position of all the detections in the track. Then, for each detection in the track, the intersection over union (IoU) is calculated between that detection and the average detection. If the IoU for at least MOVING_TRACK_MIN_DETECTIONS number of detections is less than or equal to MOVING_TRACK_MAX_IOU, then the track is considered a moving track.
Added the following Workflow Manager job properties. These can be set for any video job:
- MOVING_TRACK_LABELS_ENABLED: When set to true, attempt to label tracks as either moving or non-moving objects. Each track will have a MOVING property set to TRUE or FALSE.
- MOVING_TRACKS_ONLY: When set to true, remove any tracks that were marked as not moving.
- MOVING_TRACK_MAX_IOU: The maximum IoU overlap between detection bounding boxes and the average per-track bounding box for objects to be considered moving. Value is expected to be between 0 and 1. Note that the lower IoU, the more likely the object is moving.
- MOVING_TRACK_MIN_DETECTIONS: The minimum number of moving detections for a track to be labeled as moving.

Markup Improvements

Users can now watch videos directly in the OpenMPF web UI within the media pop-up dialog for each job. Most modern web browsers support videos encoded in VP9 and H.264. If a video cannot be played, users have the option to download it and play it using a stand-alone media player.
To set the markup encoder use MARKUP_VIDEO_ENCODER. The default encoder has changed from mjpeg to vp9. As a result, it will take longer to generate marked up videos, but they will be higher quality and can be viewed in the web UI.
Each bounding box in the marked up media is now labeled. By default, the label shows the track-level CLASSIFICATION and associated confidence value. The information shown in the label can be changed by setting MARKUP_LABELS_TEXT_PROP_TO_SHOW and MARKUP_LABELS_NUMERIC_PROP_TO_SHOW. To show information for each individual detection, rather than the entire track, set MARKUP_LABELS_FROM_DETECTIONS=TRUE.
Exemplar detections in video tracks include a star icon in their label.
Optionally, set MARKUP_VIDEO_MOVING_OBJECT_ICONS_ENABLED=TRUE to show icons that represent if the track is moving or non-moving.
Optionally, set MARKUP_VIDEO_BOX_SOURCE_ICONS_ENABLED=TRUE to show icons that represent the source of the detection. For example, if the box is the result of an algorithm detection, tracking performing gap fill, or Workflow Manager animation.
Each frame of a marked-up video now has a frame number in the upper right corner.
Please refer to the Markup Guide for the complete set of markup properties, icon definitions, and encoder considerations.

Updates

[#1181] Updated the Tesseract OCR Text Detection component from Tesseract version 4.0.0 to 4.1.1
[#1232] Updated the Azure Speech Detection component from Azure Batch Transcription version 2.0 to 3.0

Bug Fixes

[#1187] EXIF orientation is now preserved during markup and artifact extraction
[#1257] Updated OUTPUT_LAST_TASK_ONLY to work on all media types

OpenMPF 6.0.x

6.0.11: March 2021

Bug Fixes

[#1284] Updated the Azure Translation component to count emoji as 2 characters

6.0.10: March 2021

Updates

[#1270] The Azure Cognitive Services components now retry HTTP requests

6.0.9: March 2021

Bug Fixes

[#1273] Setting TRANSLATION to the empty string no longer prevents Keyword Tagging

6.0.6: March 2021

Bug Fixes

[#1265] Updated the Tika Text Detection component to handle spreadsheets
[#1268] Updated the Tika Text Detection component to remove metadata

6.0.5: February 2021

Bug Fixes

[#1266] The Azure Translation component now handles the final segment correctly when guessing sentence breaks

6.0.4: February 2021

Updates

[#1264] Updated the Azure Translation component to handle large amounts of text
[#1269] AzureTranslation no longer tries to translate text that is already in the TO_LANGUAGE

6.0.3: February 2021

OpenCV YOLO Detection Component

This new component utilizes the OpenCV Deep Neural Networks (DNN) framework to detect and classify objects in images and videos using Darknet YOLOv4 models trained on the COCO dataset. It supports both CPU and GPU modes of operation. Tracking is performed using a combination of intersection over union, pixel difference after Fast Fourier transform ( FFT) phase correlation, Kalman filtering, and OpenCV MOSSE tracking. Refer to the README for details.

6.0.2: January 2021

Bug Fixes

[#1249] FFmpeg no longer reports different frame counts for the same piece of media

6.0.1: December 2020

Bug Fixes

[#1238] The JSON output object is now generated when remote media cannot be downloaded.

6.0.0: December 2020

Upgrade to OpenCV 4.5.0

Updated core framework and components from OpenCV 3.4.7 to OpenCV 4.5.0.
OpenCV is now built with CUDA support, including cuDNN (CUDA Deep Neural Network library) and cuBLAS (CUDA Basic Linear Algebra Subroutines library). All C++ components that use the base C++ builder and executor Docker images have CUDA support built in, giving developers the option to make use of it.
Added GPU support to the OcvDnnDetection component.

Azure Cognitive Services (ACS) Translation Component

This new component utilizes the Azure Cognitive Services Translator REST endpoint to translate text from one language (locale) to another. Generally, it's intended to operate on feed-forward tracks that contain detections with TEXT and TRANSCRIPT properties. It can also operate on plain text file inputs. Refer to the README for details.

Interoperability Package

Added algorithm field to the element that describes a collection of tracks generated by an action in the JSON output object. For example:

"output": {
  "FACE": [{
    "source": "+#MOG MOTION DETECTION PREPROCESSOR ACTION#OCV FACE DETECTION ACTION",
    "algorithm": "FACECV",
    "tracks": [{ ... }],
    ...
   },

Merge Tasks in JSON Output Object

The output of two tasks in the JSON output object can be merged by setting the OUTPUT_MERGE_WITH_PREVIOUS_TASK property to true. This is a Workflow Manager property and can be set on any task in any pipeline, although it has no effect when set on the first task or the Markup task.
When the output of two tasks are merged, the tracks for the previous task will not be shown in the JSON output object, and no artifacts are generated for it. The task will be listed under TRACKS MERGED, if it's not already listed under TRACKS SUPPRESSED due to the mpf.output.objects.last.task.only system property setting, or OUTPUT_LAST_TASK_ONLY property. The tracks associated with the second task will inherit the detection type and algorithm of the previous task.
For example, the TESSERACT OCR TEXT DETECTION WITH KEYWORD TAGGING PIPELINE is defined as the TESSERACT OCR TEXT DETECTION TASK followed by the KEYWORD TAGGING (WITH FF REGION) TASK. The second task sets OUTPUT_MERGE_WITH_PREVIOUS_TASK to true. The resulting JSON output object contains one set of keyword-tagged OCR tracks that have the TEXT detection type and TESSERACTOCR algorithm (both inherited from the TESSERACT OCR TEXT DETECTION TASK):

"output": {
  "TRACKS MERGED": [{
    "source": "+#TESSERACT OCR TEXT DETECTION ACTION",
    "algorithm": "TESSERACTOCR"
  }],
  "TEXT": [{
    "source": "+#TESSERACT OCR TEXT DETECTION ACTION#KEYWORD TAGGING (WITH FF REGION) ACTION",
    "algorithm": "TESSERACTOCR",
    "tracks": [{
      "type": "TEXT",
      "trackProperties": {
         "TAGS": "ANIMAL",
         "TEXT": "The quick brown fox",
         "TEXT_LANGUAGE": "script/Latin",
         "TRIGGER_WORDS": "fox",
         "TRIGGER_WORDS_OFFSET": "16-18"
         ...

Note that you can use the OUTPUT_MERGE_WITH_PREVIOUS_TASK setting on multiple tasks. For example, if you set it as a job property it will be applied to all tasks (with the exception of Markup - in which case the task before Markup is used), so you will only get the output of the last task in the pipeline. The last task will inherit the detection type and algorithm of the first task in the pipeline.

Tesseract Custom Dictionaries

The Tesseract component Docker image now contains an /opt/mpf/tessdata_model_updater binary that you can use to update *.traineddata models with a custom dictionary, as well as extract files from existing models. Refer to the DICTIONARIES guide to learn how to use the tool.
In general, legacy *.traineddata models are more influenced by words in their dictionary than more modern LSTM *.traineddata models. Also, refer to the known issue below.

Known Issues

[#1243] Unpacking a *.traineddata model, for example, in order to modify its dictionary, and then repacking it may result in dropping some of the words present in the original dictionary file. This may be due to some kind of compression or filtering. It's unknown what effect this has on OCR results.

OpenMPF 5.1.x

5.1.3: December 2020

Setting Properties as Docker Environment Variables

Any property that can be set as a job property can now be set as a Docker environment variable by prefixing it with MPF_PROP_. For example, setting the MPF_PROP_TRTIS_SERVER environment variable in the trtis-detection service in your docker-compose.yml file will have the same effect as setting the TRTIS_SERVER job property.
Properties set in this way will take precedence over all other property types (job, algorithm, media, etc). It is not possible to change the value of properties set via environment variables at runtime and therefore they should only be used to specify properties that will not change throughout the entire lifetime of the service.

Updates

The mpf.output.objects.censored.properties system property can be used to prevent properties from being shown in JSON output objects. The value for these properties will appear as <censored>.
The Azure Speech Detection component now retries without diarization when diarization is not supported by the selected locale.

Bug Fixes

[#1230] The Azure Speech Detection component now uses a UUID for the recording id associated with a piece of media in order to prevent deleting a piece of media while it's in use.

5.1.1: December 2020

Updates

Only generate FRAME_COUNT warning when the frame difference is > 1. This can be configured using the warn.frame.count.diff system property.

Bug Fixes

[#1209] The Keyword Tagging component now generates video tracks in the JSON output object.
[#1212] The Keyword Tagging component now preserves the detection bounding box and confidence.

5.1.0: November 2020

Media Inspection Improvements

The Workflow Manager will now handle video files that don't have a video stream as an AUDIO type, and handle video files that don't have a video or audio stream as an UNKNOWN type. The JSON output object contains a new media.mediaType field that will be set to VIDEO, AUDIO, IMAGE, or UNKNOWN.
The Workflow Manager now configures Tika with custom MIME type support . Currently, this enables the detection of video/vnd.dlna.mpeg-tts and image/jxr MIME types.
If the Workflow Manager cannot use Tika to determine the media MIME type then it will fall back to using the Linux file command with a custom magicfile .
OpenMPF now supports Apple-optimized PNGs and HEIC images. Refer to the Bug Fixes section below.

EAST Text Region Detection Component Improvements

The TEMPORARY_PADDING property has been separated into TEMPORARY_PADDING_X and TEMPORARY_PADDING_Y so that X and Y padding can be configured independently.
The MERGE_MIN_OVERLAP property has been renamed to MERGE_OVERLAP_THRESHOLD so that setting it to a value of 0 will merge all regions that touch, regardless of how small the amount of overlap.
Refer to the README for details.

MPFVideoCapture and MPFImageReader Tool Improvements

These tools now support a ROTATION_FILL_COLOR property for setting the fill color for pixels near the corners and edges of frames when performing non-orthogonal rotations. Previously, the color was hardcoded to BLACK. That is still the default setting for most components. Now the color can be set to WHITE, which is the default setting for the Tesseract component.
These tools now support a ROTATION_THRESHOLD property for adjusting the threshold at which the frame transformer performs rotation. Previously, the value was hardcoded to 0.1 degrees. That is still the default value. Rotation is not performed on any ROTATION value less than that threshold. The motivation is that some algorithms detect small rotations (for example, on structured text) when there is no rotation. In such cases rotating the frame results in fewer detections.
OpenMPF now uses FFmpeg when counting video frames. Refer to the Bug Fixes section below.

Azure Cognitive Services (ACS) Form Detection Component

This new component utilizes the Azure Cognitive Services Form Detection REST endpoint to extract formatted text from documents (PDFs) and images. Refer to the README for details.
This component is capable of performing detections using a specified ACS endpoint URL. For example, different endpoints support receipt detection, business card detection, layout analysis, and support for custom models trained with or without labeled data.
This component may output the following detection properties depending on the endpoint, model, and media being processed: TEXT, TABLE_CSV_OUTPUT, KEY_VALUE_PAIRS_JSON, and DOCUMENT_JSON_FIELDS.

Keyword Tagging Component

This new component performs the same keyword tagging behavior that was previously part of the Tesseract component, but does so on feed-forward tracks that generate detections with TEXT and TRANSCRIPT properties. Refer to the README for details.
In addition to the Tesseract component, keyword tagging behavior has been removed from the Tika Text component and ACS OCR component.
Example pipelines have been added to the following components which make use of a final Keyword Tagging component stage:
- Tesseract
- Tika Text
- ACS OCR
- Sphinx
- ACS Speech

Optionally Skip Media Inspection

The Workflow Manager will skip media inspection if all of the required media metadata is provided in the job request. The MEDIA_HASH and MIME_TYPE fields are always required. Depending on the media data type, other fields may be required or optional:
- Images
  - Required: FRAME_WIDTH, FRAME_HEIGHT
  - Optional: HORIZONTAL_FLIP, ROTATION
- Videos
  - Required: FRAME_WIDTH, FRAME_HEIGHT, FRAME_COUNT, FPS, DURATION
  - Optional: HORIZONTAL_FLIP, ROTATION
- Audio files
  - Required: DURATION

Updates

Update OpenMPF Python SDK exception handling for Python 3. Now instead of raising an EnvironmentError, which has been deprecated in Python 3, the SDK will raise an mpf.DetectionError or allow the underlying exception to be thrown.

Bug Fixes

[#1028] OpenMPF can now properly handle Apple-optimized PNGs, which have a non-standard data chunk named CgBI before the IHDR chunk. The Workflow Manager uses pngdefry to convert the image into a standard PNG for processing. Before this fix, Tika would throw an error when trying to determine the MIME type of the Apple-optimized PNG.
[#1130] OpenMPF can now properly handle HEIC images. The Workflow Manager uses libheif to convert the image into a standard PNG for processing. Before this fix, the HEIC image was sometimes falsely identified as a video and the Workflow Manager would fail to count the number of frames.
[#1171] The MIME type in the JSON output object is no longer null when there is a frame counting exception.
[#1192] When processing videos, the frame count is now obtained from both OpenCV and FFmpeg. The lower of the two is used. If they don't match, a FRAME_COUNT warning is generated. Before this fix, on some videos OpenCV would return frame counts that were magnitudes higher than the frames that could actually be read. This resulted in failing to process many video segments with a BAD_FRAME_SIZE error.

OpenMPF 5.0.x

5.0.9: October 2020

Bug Fixes

[#1200] The MPFVideoCapture and MPFImageReader tools now properly handle cropping to frame regions when the region coordinates fall outside of the frame boundary. There was a bug that would result in an OpenCV error. Note that the bug only occurred when cropping was not performed with rotation or flipping.

5.0.8: October 2020

Updates

The Tesseract component now supports a TESSDATA_MODELS_SUBDIRECTORY property. The component will look for tessdata files in <MODELS_DIR_PATH>/<TESSDATA_MODELS_SUBDIRECTORY>. This allows users to easily switch between tessdata , tessdata_best, and tessdata_fast subdirectories.

Bug Fixes

[#1199] Added missing synchronized to InProgressBatchJobsService, which was resulting in some jobs staying IN_PROGRESS indefinitely.

5.0.7: September 2020

TensorRT Inference Server (TRTIS) Object Detection Component

This new component detects objects in images and videos by making use of an NVIDIA TensorRT Inference Server ( TRTIS), and calculates features that can later be used by other systems to recognize the same object in other media. We provide support for running the server as a separate service during a Docker deployment, but an external server instance can be used instead.
By default, the ip_irv2_coco model is supported and will optionally classify detected objects using COCO labels . Additionally, features can be generated for whole frames, automatically-detected object regions, and user-specified regions. Refer to the README .

5.0.6: August 2020

Enable OcvDnnDetection to Annotate Feed-forward Detections

The OcvDnnDetection component can now by configured to operate only on certain feed-forward detections and annotate them with supplementary information. For example, the following pipeline can be configured to generate detections that have both CLASSIFICATION and COLOR detection properties:

DarknetDetection (person + vehicle) --> OcvDnnDetection (vehicle color)

For example:

  "detectionProperties": {
    "CLASSIFICATION": "car",
    "CLASSIFICATION CONFIDENCE LIST": "0.397336",
    "CLASSIFICATION LIST": "car",
    "COLOR": "blue",
    "COLOR CONFIDENCE LIST": "0.93507; 0.055744",
    "COLOR LIST": "blue; gray"
  }

The OcvDnnDetection component now supports the following properties:
- CLASSIFICATION_TYPE: Set this value to change the CLASSIFICATION* part of each output property name to something else. For example, setting it to COLOR will generate COLOR, COLOR LIST, and COLOR CONFIDENCE LIST. When handling feed-foward detections, the pre-existing CLASSIFICATION* properties will be carried over and the COLOR* properties will be added to the detection.
- FEED_FORWARD_WHITELIST_FILE: When FEED_FORWARD_TYPE is provided and not set to NONE, only feed-forward detections with class names contained in the specified file will be processed. For, example, a file with only " car" in it will result in performing the exclude behavior (below) for all feed-foward detections that do not have a CLASSIFICATION of "car".
- FEED_FORWARD_EXCLUDE_BEHAVIOR: Specifies what to do when excluding detections not specified in the FEED_FORWARD_WHITELIST_FILE. Acceptable values are:
  - PASS_THROUGH: Return the excluded detections, without modification, along with any annotated detections.
  - DROP: Don't return the excluded detections. Only return annotated detections.

Updates

Make interop package work with Java 8 to better support exernal job producers and consumers.

5.0.5: August 2020

Updates

Configure Camel not to auto-acknowledge messages. Users can now see the number of pending messages in the ActiveMQ management console for queues consumed by the Workflow Manager.
Improve Tesseract OSD fallback behavior. This prevents selecting the OSD rotation from the fallback pass without the OSD script from the fallback pass.

5.0.4: August 2020

Updates

Retry job callbacks when they fail. The Workflow Manager now supports the http.callback.timeout.ms and http.callback.retries system properties.
Drop "duplicate paged in from cursor" DLQ messages.

5.0.3: July 2020

Updates

Update ActiveMQ to 5.16.0.

5.0.2: July 2020

Updates

Disable video segmentation for ACS Speech Detection to prevent issues when generating speaker ids.

5.0.1: July 2020

Updates

Updated Tessseract component with MAX_PIXELS setting to prevent processing large images.

5.0.0: June 2020

Documentation

Updated the openmpf-docker repo README and SWARM guides to describe the new build process, which now includes automatically copying the openmpf repo source code into the openmpf-build image instead of using various bind mounts, and building all of the component base builder and executor images.
Updated the openmpf-docker repo README with the following sections:
Updated the CONTRIBUTING guide for Docker deployment with information on the new build process and component base builder and executor images.
Updated the Install Guide with a pointer to the "Quick Start" section on DockerHub.
Updated the REST API with the new endpoints for getting, deleting, and creating actions, tasks, and pipelines, as well as a change to the [GET] /rest/info endpoint.
Updated the C++ Batch Component API to describe changes to the GetDetection() calls, which now return a collection of detections or tracks instead of an error code, and to describe improvements to exception handling.
Updated the C++ Batch Component API , Python Batch Component API, and Java Batch Component API with MIME_TYPE, FRAME_WIDTH, and FRAME_HEIGHT media properties.
Updated the Python Batch Component API with information on Python3 and the simplification of using a dict for some of the data members.

JSON Output Object

Renamed stages to tasks for clarity and consistency with the rest of the code.
The media element no longer contains a message field.
Each detectionProcessingError element now contains a code field.
Errors and warnings are now grouped by mediaId and summarized using a details element that contains a source , code, and message field. Refer to this comment for an example of the JSON structure. Note that errors and warnings generated by the Workflow Manager do not have a mediaId.
- When an error or warning occurs in multiple frames of a video for a single piece of media it will be represented in one details element and the message will list the frame ranges.

Interoperability Package

Renamed JsonStage.java to JsonTask.java.
Removed JsonJobRequest.java.
Modified JsonDetectionProcessingError.java by removing the startOffset and stopOffset fields and adding the following new fields: startOffsetFrame, stopOffsetFrame, startOffsetTime, stopOffsetTime, and code.
Updated JsonMediaOutputObject.java by removing message field.
Added JsonMediaIssue.java and JsonIssueDetails.java.

Persistent Database

The input_object column in the job_request table has been renamed to job and the content now contains a serialized form of BatchJob.java instead of JsonJobRequest.java.

C++ Batch Component API

The GetDetection() calls now return a collection instead of an error code:
- std::vector<MPFImageLocation> GetDetections(const MPFImageJob &job)
- std::vector<MPFVideoTrack> GetDetections(const MPFVideoJob &job)
- std::vector<MPFAudioTrack> GetDetections(const MPFAudioJob &job)
- std::vector<MPFGenericTrack> GetDetections(const MPFGenericJob &job)
MPFDetectionException can now be constructed with a what parameter representing a descriptive error message:
- MPFDetectionException(MPFDetectionError error_code, const std::string &what = "")
- MPFDetectionException(const std::string &what)

Python Batch Component API

Simplified the detection_properties and frame_locations data members to use a Python dict instead of a custom data type.

Full Docker Conversion

Each component is now encapsulated in its own Docker image which self-registers with the Workflow Manager at runtime. This deconflicts component dependencies, and allows for greater flexibility when deciding which components to deploy at runtime.
The Node Manager image has been removed. For Docker deployments, component services should be managed using Docker tools external to OpenMPF.
In Docker deployments, streaming job REST endpoints are disabled, the Nodes web page is no longer available, component tar.gz packages cannot be registered through the Component Registration web page, and the mpf command line script can now only be run on the Workflow Manager container to modify user settings. The preexisting features are now reserved for non-Docker deployments and development environments.
The OpenMPF Docker stack can optionally be deployed with Kibana (which depends on Elasticsearch and Filebeat) for viewing log files. Refer to the openmpf-docker README .

Docker Component Base Images

A base builder image and executor image are provided for C++ (README), Python (README), and Java (README) component development. Component developers can also refer to the Dockerfile in the source code for each component as reference for how to make use of the base images.

Restrict Media Types that a Component Can Process

Each component service now supports an optional RESTRICT_MEDIA_TYPES Docker environment variable that specifies the types of media that service will process. For example, RESTRICT_MEDIA_TYPES: VIDEO,IMAGE will process both videos and images, while RESTRICT_MEDIA_TYPES: IMAGE will only process images. If not specified, the service will process all of the media types it natively supports. For example, this feature can be used to ensure that some services are always available to process images while others are processing long videos.

Import Additional Root Certificates into the Workflow Manager

Additional root certificates can be imported into the Workflow Manager at runtime by adding an entry for MPF_CA_CERTS to the workflow-manager service's environment variables in docker-compose.core.yml . MPF_CA_CERTS must contain a colon-delimited list of absolute file paths. Of note, a root certificate may be used to trust the identity of a remote object storage server.

DockerHub

Pushed prebuilt OpenMPF Docker images to DockerHub. Refer to the "Quick Start" section of the OpenMPF Workflow Manager image documentation.

Version Updates

Updated from Oracle Java 8 to OpenJDK 11, which required updating to Tomcat 8.5.41. We now use Cargo to run integration tests.
Updated OpenCV from 3.0.0 to 3.4.7 to update Deep Neural Networks (DNN) support.
Updated Python from 2.7 to 3.8.2.

FFmpeg

We are no longer building separate audio and video encoders and decoders for FFmpeg. Instead, we are using the built-in decoders that come with FFmpeg by default. This simplifies the build process and redistribution via Docker images.

Artifact Extraction

The ARTIFACT_EXTRACTION_POLICY property can now be assigned a value of NONE, VISUAL_TYPES_ONLY, ALL_TYPES, or ALL_DETECTIONS.
- With the VISUAL_TYPES_ONLY or ALL_TYPES policy, artifacts will be extracted according to the ARTIFACT_EXTRACTION_POLICY* properties. With the NONE and ALL_DETECTIONS policies, those settings are ignored.
- Note that previously NONE, VISUAL_EXEMPLARS_ONLY, EXEMPLARS_ONLY, ALL_VISUAL_DETECTIONS, and ALL_DETECTIONS were supported.
The following ARTIFACT_EXTRACTION_POLICY* properties are now supported:
- ARTIFACT_EXTRACTION_POLICY_EXEMPLAR_FRAME_PLUS: Extract the exemplar frame from the track, plus this many frames before and after the exemplar.
- ARTIFACT_EXTRACTION_POLICY_FIRST_FRAME: If true, extract the first frame from the track.
- ARTIFACT_EXTRACTION_POLICY_MIDDLE_FRAME: If true, extract the frame with a detection that is closest to the middle frame from the track.
- ARTIFACT_EXTRACTION_POLICY_LAST_FRAME: If true, extract the last frame from the track.
- ARTIFACT_EXTRACTION_POLICY_TOP_CONFIDENCE_COUNT: Sort the detections in a track by confidence and then extract this many detections, starting with those which have the highest confidence.
- ARTIFACT_EXTRACTION_POLICY_CROPPING: If true, an artifact will be extracted for each detection in each frame that is selected according to the other ARTIFACT_EXTRACTION_POLICY* properties. The extracted artifact will be cropped to the width and height of the detection bounding box, and the artifact will be rotated according to the detection ROTATION property. If false, the artifact extraction behavior is unchanged from the previous release: the entire frame will be extracted without any rotation.
For clarity, OUTPUT_EXEMPLARS_ONLY has been renamed to OUTPUT_ARTIFACTS_AND_EXEMPLARS_ONLY. Extracted artifacts will always be reported in the JSON output object.
The mpf.output.objects.exemplars.only system property has been renamed to mpf.output.objects.artifacts.and.exemplars.only. It works the same as before with the exception that if an artifact is extracted for a detection then that detection will always be represented in the JSON output object, whether it's an exemplar or not.
The mpf.output.objects.last.stage.only system property has been renamed to mpf.output.objects.last.task.only. It works the same as before with the exception that when set to true artifact extraction is skipped for all tasks but the last task.

REST Endpoints

Modified [GET] /rest/info. Now returns output like {"version": "4.1.0", "dockerEnabled": true}.
Added the following REST endpoints for getting, removing, and creating actions, tasks, and pipelines. Refer to the REST API for more information:
- [GET] /rest/actions, [GET] /rest/tasks, [GET] /rest/pipelines
- [DELETE] /rest/actions, [DELETE] /rest/tasks, [DELETE] /rest/pipelines
- [POST] /rest/actions , [POST] /rest/tasks, [POST] /rest/pipelines
All of the endpoints above are new with the exception of [GET] /rest/pipelines. The endpoint has changed since the last version of OpenMPF. Some fields in the response JSON have been removed and renamed. Also, it now returns a collection of tasks for each pipelines. Refer to the REST API.
[GET] /rest/algorithms can be used to get information about algorithms. Note that algorithms are tied to registered components, so to remove an algorithm you must unregister the associated component. To add an algorithm, start the associated component's Docker container so it self-registers with the Workflow Manager.

Incomplete Actions, Tasks, and Pipelines

The previous version of OpenMPF would generate an error when attempting to register a component that included actions, tasks, or pipelines that depend on algorithms, actions, or tasks that are not yet registered with the Workflow Manager. This required components to be registered in a specific order. Also, when unregistering a component, it required the components which depend on it to be unregistered. These dependency checks are no longer enforced.
In general, the Workflow Manager now appropriately handles incomplete actions, tasks, and pipelines by checking if all of the elements are defined before executing a job, and then preserving that information in memory until the job is complete. This allows components to be registered and removed in an arbitrary order without affecting the state of other components, actions, tasks, or pipelines. This also allows actions and tasks to be removed using the new REST endpoints and then re-added at a later time while still preserving the elements that depend on them.
Note that unregistering a component while a job is running will cause it to stall. Please ensure that no jobs are using a component before unregistering it.

Python Arbitrary Rotation

The Python MPFVideoCapture and MPFImageReader tools now support ROTATION values other than 0, 90, 180, and 270 degrees. Users can now specify a clockwise ROTATION job property in the range [0, 360). Values outside that range will be normalized to that range. Floating point values are accepted. This is similar to the existing support for C++ arbitrary rotation.

OpenCV Deep Neural Networks (DNN) Detection Component

This new component replaces the old CaffeDetection component. It supports the same GoogLeNet and Yahoo Not Suitable For Work (NSFW) models as the old component, but removes support for the Rezafuad vehicle color detection model in favor of a custom TensorFlow vehicle color detection model. In our tests, the new model has proven to be more generalizable and provide more accurate results on never-before-seen test data. Refer to the README.

Azure Cognitive Services (ACS) Speech Detection Component

This new component utilizes the Azure Cognitive Services Batch Transcription REST endpoint to transcribe speech from audio and video files. Refer to the README.

Tesseract OCR Text Detection Component

Text tagging has been simplified to only support regular expression searches. Whole keyword searches are a subset of regular expression searches, and are therefore still supported. Also, the text-tags.json file format has been updated to allow for specifying case-sensitive regular expression searches.
Additionally, the TRIGGER_WORDS and TRIGGER_WORDS_OFFSET detection properties are now supported, which list the OCR'd words that resulted in adding a TAG to the detection, and the character offset of those words within the OCR'd TEXT, respectively.
Key changes to tagging output and text-tags.json format are outlined below. Refer to the README for more information:
- Regex patterns should now be entered in the format {"pattern": "regex_pattern"}. Users can add and toggle the "caseSensitive" regex flag for each pattern.
  - For example: {"pattern": "(\\b)bus(\\b)", "caseSensitive": true} enables case-sensitive regex pattern matching.
  - By default, each regex pattern, including those in the legacy format, will be case-insensitive.
- As part of the text tagging update, the TAGS outputs are now separated by semicolons ; rather than commas , to be consistent with the delimiters for TRIGGER_WORDS and TRIGGER_WORDS_OFFSET output patterns.
  - Because semicolons can be part of the trigger word itself, those semicolons will be encapsulated in brackets.
    - For example, detected trigger with a ; in the OCR'd TEXT is reported as TRIGGER_WORDS=detected trigger with a [;]; some other trigger.
  - Commas are now used to group each set of TRIGGER_WORDS_OFFSET with its respective TRIGGER_WORDS output. Both TAGS and TRIGGER_WORDS are separated by semicolons only.
    - For example: TRIGGER_WORDS=trigger1; trigger2, TRIGGER_WORDS_OFFSET=0-5, 6-10; 12-15, means that trigger1 occurs twice in the text at the index ranges 0-5 and 6-10, and trigger2 occurs at index range 12-15.
- Regex tagging now follows the C++ ECMAS format ( see examples here) after resolving JSON string conversion for regex tags.
- As a result the regex patterns \b and \p in the text tagging file must now be written as \\b and \\p, respectively, to match the format of other regex character patterns (ex. \\d, \\w, \\s, etc.).
The MAX_PARALLEL_SCRIPT_THREADS and MAX_PARALLEL_PAGE_THREADS properties are now supported. When processing images, the first property is used to determine how many threads to run in parallel. Each thread performs OCR using a different language or script model. When processing PDFs, the second property is used to determine how many threads to run in parallel. Each thread performs OCR on a different page of the PDF.
The ENABLE_OSD_FALLBACK property is now supported. If enabled, an additional round of OSD is performed when the first round fails to generate script predictions that are above the OSD score and confidence thresholds. In the second pass, the component will run OSD on multiple copies of the input text image to get an improved prediction score and OSD_FALLBACK_OCCURRED detection property will be set to true.
If any OSD-detected models are missing, the new MISSING_LANGUAGE_MODELS detection property will list the missing models.

Tika Text Detection Component

The Tika text detection component now supports text tagging in the same way as the Tesseract component. Refer to the README.

Other Improvements

Simplified component descriptor.json files by moving the specification of common properties, such as CONFIDENCE_THRESHOLD, FRAME_INTERVAL, MIN_SEGMENT_LENGTH, etc., to a single workflow-properties.json file. Now when the Workflow Manager is updated to support new features, the component descriptor.json file will not need to be updated.
Updated the Sphinx component to return TRANSCRIPT instead of TRANSCRIPTION, which is grammatically correct.
Whitespace is now trimmed from property names when jobs are submitted via the REST API.
The Darknet Docker image now includes the YOLOv3 model weights.
The C++ and Python ModelsIniParser now allows users to specify optional fields.
When a job completion callback fails, but otherwise the job is successful, the final state of the job will be COMPLETE_WITH_WARNINGS.

Bug Fixes

[#772] Can now create a custom pipeline with long action names using the Pipelines 2 UI.
[#812] Now properly setting the start and stop index for elements in the detectionProcessingErrors collection in the JSON output object. Errors reported for each job segment will now appear in the collection.
[#941] Tesseract component no longer segfaults when handling corrupt media.
[#1005] Fixed a bug that caused a NullPointerException when attempting to get output object JSON via REST before a job completes.
[#1035] The search bar in the Job Status UI can once again for used to search for job id.
[#1104] Fixed C++/Python component executor memory leaks.
[#1108] Fixed a bug when handling frames and detections that are horizontally flipped. This affected both markup and feed-forward behaviors.
[#1119] Fixed Tesseract component memory leaks and uninitialized read issues.

Known Issues

[#1028] Media inspection fails to handle Apple-optimized PNGs with the CgBI data chunk before the IHDR chunk.
[#1109] We made the search bar in the Job Status UI more efficient by shifting it to a database query, but in doing so introduced a bug where the search operates on UTC time instead of local system time.
[#1010] mpf.output.objects.enabled does not behave as expected for batch jobs. A user would expect it to control whether the JSON output object is generated, but it's generated regardless of that setting.
[#1032] Jobs fail on corrupt QuickTime videos. For these videos, the OpenCV-reported frame count is more than twice the actual frame count.
[#1106] When a job ends in ERROR the job status UI does not show an End Date.

OpenMPF 4.1.x

4.1.14: June 2020

Bug Fixes

[#1120] The node-manager Docker image now correctly installs CUDA libraries so that GPU-enabled components on that image can run on the GPU.
[#1064] Fixed memory leaks in the Darknet component for various network types, and when using GPU resources. This bug covers everything not addressed by #1062.

4.1.13: June 2020

Updates

Updated the OpenCV build and media inspection process to properly handle webp images.

4.1.12: May 2020

Updates

Updated JDK from jdk-8u181-linux-x64.rpm to jdk-8u251-linux-x64.rpm.

4.1.11: May 2020

Tesseract OCR Text Detection Component

Added INVALID_MIN_IMAGE_SIZE job property to filter out images with extremely low width or height.
Updated image rescaling behavior to account for image dimension limits.
Fixed handling of nullptr returns from Tesseract API OCR calls.

4.1.8: May 2020

Azure Cognitive Services (ACS) OCR Component

This new component utilizes the ACS OCR REST endpoint to extract text from images and videos. Refer to the README.

4.1.6: April 2020

Updates

Now silently discarding ActiveMQ DLQ "Suppressing duplicate delivery on connection" messages in addition to "duplicate from store" messages.

4.1.5: March 2020

Bug Fixes

[#1062] Fixed a memory leak in the Darknet component that occurred when running jobs on CPU resources with the Tiny YOLO model.

Known Issues

[#1064] The Darknet component has memory leaks for various network types, and potentially when using GPU resources. This bug covers everything not addressed by #1062.

4.1.4: March 2020

Updates

Updated from Hibernate 5.0.8 to 5.4.12 to support schema-based multitenancy. This allows multiple instances of OpenMPF to use the same PostgreSQL database as long as each instance connects to the database as a separate user, and the database is configured appropriately. This also required updating Tomcat from 7.0.72 to 7.0.76.

JSON Output Object

Updated the Workflow Manager to include an outputobjecturi in GET callbacks, and outputObjectUri in POST callbacks, when jobs complete. This URI specifies a file path, or path on the object storage server, depending on where the JSON output object is located.

Interoperability Package

Updated JsonCallbackBody.java to contain an outputObjectUri field.

4.1.3: February 2020

Features

Added support for DETECTION_PADDING_X and DETECTION_PADDING_Y optional job properties. The value can be a percentage or whole-number pixel value. When positive, each detection region in each track will be expanded. When negative, the region will shrink. If the detection region is shrunk to nothing, the shrunk dimension(s) will be set to a value of 1 pixel and the SHRUNK_TO_NOTHING detection property will be set to true.
Added support for DISTANCE_CONFIDENCE_WEIGHT_FACTOR and SIZE_CONFIDENCE_WEIGHT_FACTOR SuBSENSE algorithm properties. Increasing the value of the first property will generate detection confidence values that favor being closer to the center frame of a track. Increasing the value of the second property will generate detection confidence values that favor large detection regions.

4.1.1: January 2020

Bug Fixes

[#1016] Fixed a bug that caused a deadlock situation when the media inspection process failed quickly when processing many jobs using a pipeline with more than one stage.

4.1.0: July 2019

Documentation

Updated the C++ Batch Component API to describe the ROTATION detection property. See the C++ Arbitrary Rotation section below.
Updated the REST API with new component registration REST endpoints. See the Component Registration REST Endpoints section below.
Added a README for the EAST text region detection component. See the EAST Text Region Detection Component section below.
Updated the Tesseract OCR text detection component README . See the Tesseract OCR Text Detection Component section below.
Updated the openmpf-docker repo README and SWARM guide to describe the new streamlined approach to using docker-compose config. See the Docker Deployment section below.
Fixed the description of MIN_SEGMENT_LENGTH and associated examples in the User Guide for issue #891.
Updated the Java Batch Component API with information on how to use Log4j2. Related to resolving issue #855.
Updated the Install Guide to point to the Docker README.
Transformed the Build Guide into a Development Environment Guide.

C++ Arbitrary Rotation

The C++ MPFVideoCapture and MPFImageReader tools now support ROTATION values other than 0, 90, 180, and 270 degrees. Users can now specify a clockwise ROTATION job property in the range [0, 360). Values outside that range will be normalized to that range. Floating point values are accepted.
When using those tools to read frame data, they will automatically correct for rotation so that the returned frame is horizontally oriented toward the normal 3 o'clock position.
- When FEED_FORWARD_TYPE=REGION, these tools will look for a ROTATION detection property in the feed-forward detections and automatically correct for rotation. For example, a detection property of ROTATION=90 represents that the region is rotated 90 degrees counter clockwise, and therefore must be rotated 90 degrees clockwise to correct for it.
- When FEED_FORWARD_TYPE=SUPERSET_REGION, these tools will properly account for the ROTATION detection property associated with each feed-forward detection when calculating the bounding box that encapsulates all of those regions.
- When FEED_FORWARD_TYPE=FRAME, these tools will rotate the frame according to the ROTATION job property. It's important to note that for rotations other than 0, 90, 180, and 270 degrees the rotated frame dimensions will be larger than the original frame dimensions. This is because the frame needs to be expanded to encapsulate the entirety of the original rotated frame region. Black pixels are used to fill the empty space near the edges of the original frame.
The Markup component now places a colored dot at the upper-left corner of each detection region so that users can determine the rotation of the region relative to the entire frame.

Component Registration REST Endpoints

Added a [POST] /rest/components/registerUnmanaged endpoint so that components running as separate Docker containers can self-register with the Workflow Manager.
- Since these components are not managed by the Node Manager, they are considered unmanaged OpenMPF components. These components are not displayed in Nodes web UI and are tagged as unmanaged in the Component Registration web UI where they can only be removed.
- Note that components uploaded to the Component Registration web UI as .tar.gz files are considered managed components.
Added a [DELETE] /rest/components/{componentName} endpoint that can be used to remove managed and unmanaged components.

Python Component Executor Docker Image

Component developers can now use a Python component executor Docker image to write a Python component for OpenMPF that can be encapsulated within a Docker container. This isolates the build and execution environment from the rest of OpenMPF. For more information, see the README.
Components developed with this image are not managed by the Node Manager; rather, they self-register with the Workflow Manager and their lifetime is determined by their own Docker container.

Docker Deployment

Streamlined single-host docker-compose up deployments and multi-host docker stack deploy swarm deployments. Now users are instructed to create a single docker-compose.yml file for both types of deployments.
Removed the docker-generate-compose-files.sh script in favor of allowing users the flexibility of combining multiple docker-compose.*.yml files together using docker-compose config. See the Generate docker-compose.yml section of the README.
Components based on the Python component executor Docker image can now be defined and configured directly in docker-compose.yml.
OpenMPF Docker images now make use of Docker labels.

EAST Text Region Detection Component

This new component uses the Efficient and Accurate Scene Text (EAST) detection model to detect text regions in images and videos. It reports their location, angle of rotation, and text type (STRUCTURED or UNSTRUCTURED), and supports a variety of settings to control the behavior of merging text regions into larger regions. It does not perform OCR on the text or track detections across video frames. Thus, each video track is at most one detection long. For more information, see the README.
Optionally, this component can be built as a Docker image using the Python component executor Docker image, allowing it to exist apart from the Node Manager image.

Tesseract OCR Text Detection Component

Updated to support reading tessdata *.traineddata files at a specified MODELS_DIR_PATH. This allows users to install new *.traineddata files post deployment.
Updated to optionally perform Tesseract Orientation and Script Detection (OSD). When enabled, the component will attempt to use the orientation results of OSD to automatically rotate the image, as well as perform OCR using the scripts detected by OSD.
Updated to optionally rotate a feed-forward text region 180 degrees to account for upside-down text.
Now supports the following preprocessing properties for both structured and unstructured text:
- Text sharpening
- Text rescaling
- Otsu image thresholding
- Adaptive thresholding
- Histogram equalization
- Adaptive histogram equalization (also known as Contrast Limited Adaptive Histogram Equalization (CLAHE))
Will use the TEXT_TYPE detection property in feed-forward regions provided by the EAST component to determine which preprocessing steps to perform.
For more information on these new features, see the README.
Removed gibberish and string filters since they only worked on English text.

ActiveMQ Profiles

The ActiveMQ Docker image now supports custom profiles. The container selects an activemq.xml and env file to use at runtime based on the value of the ACTIVE_MQ_PROFILE environment variable. Among others, these files contain configuration settings for Java heap space and component queue memory limits.
This release only supports a default profile setting, as defined by activemq-default.xml and env.default; however, developers are free to add other activemq-<profile>.xml and env.<profile> files to the ActiveMQ Docker image to suit their needs.

Disabled ActiveMQ Prefetch

Disabled ActiveMQ prefetching on all component queues. Previously, a prefetch value of one was resulting in situations where one component service could be dispatched two sub-jobs, thereby starving other available component services which could process one of those sub-jobs in parallel.

Search Region Percentages

In addition to using exact pixel values, users can now use percentages for the following properties when specifying search regions for C++ and Python components:
- SEARCH_REGION_TOP_LEFT_X_DETECTION
- SEARCH_REGION_TOP_LEFT_Y_DETECTION
- SEARCH_REGION_BOTTOM_RIGHT_X_DETECTION
- SEARCH_REGION_BOTTOM_RIGHT_Y_DETECTION
For example, setting SEARCH_REGION_TOP_LEFT_X_DETECTION=50% will result in components only processing the right half of an image or video.
Optionally, users can specify exact pixel values of some of these properties and percentages for others.

Other Improvements

Increased the number of ActiveMQ maxConcurrentConsumers for the MPF.COMPLETED_DETECTIONS queue from 30 to 60.
The Create Job web UI now only displays the content of the $MPF_HOME/share/remote-media directory instead of all of $MPF_HOME/share, which prevents the Workflow Manager from indexing generated JSON output files, artifacts, and markup. Indexing the latter resulted in Java heap space issues for large scale production systems. This is a mitigation for issue #897.
The Job Status web UI now makes proper use of pagination in SQL/Hibernate through the Workflow Manager to avoid retrieving the entire jobs table, which was inefficient.
The Workflow Manager will now silently discard all duplicate messages in the ActiveMQ Dead Letter Queue (DLQ), regardless of destination. Previously, only messages destined for component sub-job request queues were discarded.

Bug Fixes

[#891] Fixed a bug where the Workflow Manager media segmenter generated short segments that were minimally MIN_SEGMENT_LENGTH+1 in size instead of MIN_SEGMENT_LENGTH.
[#745] In environments where thousands of jobs are processed, users have observed that, on occasion, pending sub-job messages in ActiveMQ queues are not processed until a new job is created. This seems to have been resolved by disabling ActiveMQ prefetch behavior on component queues.
[#855] A logback circular reference suppressed exception no longer throws a StackOverflowError. This was resolved by transitioning the Workflow Manager and Java components from the Logback framework to Log4j2.

Known Issues

[#897] OpenMPF will attempt to index files located in $MPF_HOME/share as soon as the webapp is started by Tomcat. This is so that those files can be listed in a directory tree in the Create Job web UI. The main problem is that once a file gets indexed it's never removed from the cache, even if the file is manually deleted, resulting in a memory leak.

Late Additions: November 2019

User names, roles, and passwords can now be set by using an optional user.properties file. This allows administrators to override the default OpenMPF users that come preconfigured, which may be a security risk. Refer to the "Configure Users" section of the openmpf-docker README for more information.

Late Additions: December 2019

Transitioned from using a mySQL persistent database to PostgreSQL to support users that use an external PostgreSQL database in the cloud.
Updated the EAST component to support a TEMPORARY_PADDING and FINAL_PADDING property. The first property determines how much padding is added to detections during the non-maximum suppression or merging step. This padding is effectively removed from the final detections. The second property is used to control the final amount of padding on the output regions. Refer to the README.

OpenMPF 4.0.x

4.0.0: February 2019

Documentation

Added an Object Storage Guide with information on how to configure OpenMPF to work with a custom NGINX object storage server, and how to run jobs that use an S3 object storage server. Note that the system properties for the custom NGINX object storage server have changed since the last release.

Upgrade to Tesseract 4.0

Both the Tesseract OCR Text Detection Component and OpenALPR License Plate Detection Components have been updated to use the new version of Tesseract.
Additionally, Leptonica has been upgraded from 1.72 to 1.75.

Docker Deployment

The Docker images now use the yum package manager to install ImageMagick6 from a public RPM repository instead of downloading the RPMs directly from imagemagick.org. This resolves an issue with the OpenMPF Docker build where RPMs on imagemagick.org were no longer available.

Tesseract OCR Text Detection Component

Updated to allow the user to set a TESSERACT_OEM property in order to select an OCR engine mode (OEM).
"script/Latin" can now be specified as the TESSERACT_LANGUAGE. When selected, Tesseract will select all Latin characters, which can be from different Latin languages.

Ceph S3 Object Storage

Added support for downloading files from, and uploading files to, an S3 object storage server. The following job properties can be provided: S3_ACCESS_KEY, S3_SECRET_KEY, S3_RESULTS_BUCKET, S3_UPLOAD_ONLY.
At this time, only support for Ceph object storage has been tested. However, the Workflow Manager uses the AWS SDK for Java to communicate with the object store, so it is possible that other S3-compatible storage solutions may work as well.

ISO-8601 Timestamps

All timestamps in the JSON output object, and streaming video callbacks, are now in the ISO-8601 format (e.g. " 2018-12-19T12:12:59.995-05:00"). This new format includes the time zone, which makes it possible to compare timestamps generated between systems in different time zones.
This change does not affect the track and detection start and stop offset times, which are still reported in milliseconds since the start of the video.

Reduced Redis Usage

The Workflow Manager has been refactored to reduce usage of the Redis in-memory database. In general, Redis is not necessary for storing job information and only resulted in introducing potential delays in accessing that data over the network stack.
Now, only track and detection data is stored in Redis for batch jobs. This reduces the amount of memory the Workflow Manager requires of the Java Virtual Machine. Compared to the other job information, track and detection data can potentially be relatively much larger. In the future, we plan to store frame data in Redis for streaming jobs as well.

Caffe Vehicle Color Estimation

The Caffe Component models.ini file has been updated with a "vehicle_color" section with links for downloading the Reza Fuad Rachmadi's Vehicle Color Recognition Using Convolutional Neural Network model files.
The following pipelines have been added. These require the above model files to be placed in $MPF_HOME/share/models/CaffeDetection:
- CAFFE REZAFUAD VEHICLE COLOR DETECTION PIPELINE
- CAFFE REZAFUAD VEHICLE COLOR DETECTION (WITH FF REGION FROM TINY YOLO VEHICLE DETECTOR) PIPELINE
- CAFFE REZAFUAD VEHICLE COLOR DETECTION (WITH FF REGION FROM YOLO VEHICLE DETECTOR) PIPELINE

Track Merging and Minimum Track Length

The following system properties now have "video" in their names:
- detection.video.track.merging.enabled
- detection.video.track.min.gap
- detection.video.track.min.length
- detection.video.track.overlap.threshold
The above properties can be overridden by the following job properties, respectively. These have not been renamed since the last release:
- MERGE_TRACKS
- MIN_GAP_BETWEEN_TRACKS
- MIN_TRACK_LENGTH
- MIN_OVERLAP
These system and job properties now only apply to video media. This resolves an issue where users had set detection.track.min.length=5, which resulted in dropping all image media tracks. By design, each image track can only contain a single detection.

Bug Fixes

Fixed a bug where the Docker entrypoint scripts appended properties to the end of $MPF_HOME/share/config/mpf-custom.properties every time the Docker deployment was restarted, resulting in entries like detection.segment.target.length=5000,5000,5000.
Upgrading to Tesseract 4 fixes a bug where, when specifying TESSERACT_LANGUAGE, if one of the languages is Arabic, then Arabic must be specified last. Arabic can now be specified first, for example: ara+eng.
Fixed a bug where the minimum track length property was being applied to image tracks. Now it's only applied to video tracks.
Fixed a bug where ImageMagick6 installation failed while building Docker images.

OpenMPF 3.0.x

3.0.0: December 2018

NOTE: The Build Guide and Install Guide are outdated. The old process for manually configuring a Build VM, using it to build an OpenMPF package, and installing that package, is deprecated in favor of Docker containers. Please refer to the openmpf-docker README.

NOTE: Do not attempt to register or unregister a component through the Nodes UI in a Docker deployment. It may appear to succeed, but the changes will not affect the child Node Manager containers, only the Workflow Manager container. Also, do not attempt to use the mpf command line tools in a Docker deployment.

Documentation

Added a README , SWARM guide, and CONTRIBUTING guide for Docker deployment.
Updated the User Guide with information on how track properties and track confidence are handled when merging tracks.
Added README files for new components. Refer to the component sections below.

Docker Support

OpenMPF can now be built and distributed as 5 Docker images: openmpf_workflow_manager, openmpf_node_manager, openmpf_active_mq, mysql_database, and redis.
These images can be deployed on a single host using docker-compose up.
They can also be deployed across multiple hosts in a Docker swarm cluster using docker stack deploy.
GPU support is enabled through the NVIDIA Docker runtime.
Both HTTP and HTTPS deployments are supported.

JSON Output Object

Added a trackProperties field at the track level that works in much the same way as the detectionProperties field at the detection level. Both are maps that contain zero or more key-value pairs. The component APIs have always supported the ability to return track-level properties, but they were never represented in the JSON output object, until now.
Similarly, added a track confidence field. The component APIs always supported setting it, but the value was never used in the JSON output object, until now.
Added jobErrors andjobWarnings fields. The jobErrors field will mention that there are items in detectionProcessingErrors fields.
The offset, startOffset, and stopOffset fields have been removed in favor of the existing offsetFrame , startOffsetFrame, and stopOffsetFrame fields, respectively. They were redundant and deprecated.
Added a mpf.output.objects.exemplars.only system property, and OUTPUT_EXEMPLARS_ONLY job property, that can be set to reduce the size of the JSON output object by only recording the track exemplars instead of all of the detections in each track.
Added a mpf.output.objects.last.stage.only system property, and OUTPUT_LAST_STAGE_ONLY job property, that can be set to reduce the size of the JSON output object by only recording the detections for the last non-markup stage of a pipeline.

Darknet Component

The Darknet component can now support processing streaming video.
In batch mode, video frames are prefetched, decoded, and stored in a buffer using a separate thread from the one that performs the detection. The size of the prefetch buffer can be configured by setting FRAME_QUEUE_CAPACITY.
The Darknet component can now perform basic tracking and generate video tracks with multiple detections. Both the default detection mode and preprocessor detection mode are supported.
The Darknet component has been updated to support the full and tiny YOLOv3 models. The YOLOv2 models are no longer supported.

Tesseract OCR Text Detection Component

This new component extracts text found in an image and reports it as a single-detection track.
PDF documents can also be processed with one track detection per page.
Users may set the language of each track using the TESSERACT_LANGUAGE property as well as adjust other image preprocessing properties for text extraction.
Refer to the README.

OpenCV Scene Change Detection Component

This new component detects and segments a given video by scenes. Each scene change is detected using histogram comparison, edge comparison, brightness (fade outs), and overall hue/saturation/value differences between adjacent frames.
Users can toggle each type of of scene change detection technique as well as threshold properties for each detection method.
Refer to the README.

Tika Text Detection Component

This new component extracts text contained in documents and performs language detection. 71 languages and most document formats (.txt, .pptx, .docx, .doc, .pdf, etc.) are supported.
Refer to the README.

Tika Image Detection Component

This new component extracts images embedded in document formats (.pdf, .ppt, .doc) and stores them on disk in a specified directory.
Refer to the README.

Track-Level Properties and Confidence

Refer to the addition of track-level properties and confidence in the JSON Output Object section.
Components have been updated to return meaningful track-level properties. Caffe and Darknet include CLASSIFICATION, OALPR includes the exemplar TEXT, and Sphinx includes the TRANSCRIPTION.
The Workflow Manager will now populate the track-level confidence. It is the same as the exemplar confidence, which is the max of all of the track detections.

Custom NGINX HTTP Object Storage

Added http.object.storage.* system properties for configuring an optional custom NGINX object storage server on which to store generated detection artifacts, JSON output objects, and markup files.
When a file cannot be uploaded to the server, the Workflow Manager will fall back to storing it in $MPF_HOME/share, which is the default behavior when an object storage server is not specified.
If and when a failure occurs, the JSON output object will contain a descriptive message in the jobWarnings field, and, if appropriate, the markupResult.message field. If the job completes without other issues, the final status will be COMPLETE_WITH_WARNINGS.
The NGINX storage server runs custom server-side code which we can make available upon request. In the future, we plan to support more common storage server solutions, such as Amazon S3.

ActiveMQ

The MPF_OUTPUT queue is no longer supported and has been removed. Job producers can specify a callback URL when creating a job so that they are alerted when the job is complete. Users observed heap space issues with ActiveMQ after running thousands of jobs without consuming messages from the MPF_OUTPUT queue.
The Workflow Manager will now silently discard duplicate sub-job request messages in the ActiveMQ Dead Letter Queue ( DLQ). This fixes a bug where the Workflow Manager would prematurely terminate jobs corresponding to the duplicate messages. It's assumed that ActiveMQ will only place a duplicate message in the DLQ if the original message, or another duplicate, can be delivered.

Node Auto-Configuration

Added the node.auto.config.enabled, node.auto.unconfig.enabled, and node.auto.config.num.services.per.component system properties for automatically managing the configuration of services when nodes join and leave the OpenMPF cluster.
Docker will assign a a hostname with a randomly-generated id to containers in a swarm deployment. The above properties allow the Workflow Manager to automatically discover and configure services on child Node Manager components, which is convenient since the hostname of those containers cannot be known in advance, and new containers with new hostnames are created when the swarm is restarted.

Job Status Web UI

Added the web.broadcast.job.status.enabled and web.job.polling.interval system properties that can be used to configure if the Workflow Manager automatically broadcasts updates to the Job Status web UI. By default, the broadcasts are enabled.
In a production environment that processes hundreds of jobs or more at the same time, this behavior can result in overloading the web UI, causing it to slow down and freeze up. To prevent this, set web.broadcast.job.status.enabled to false. If web.job.polling.interval is set to a non-zero value, the web UI will poll for updates at that interval (specified in milliseconds).
To disable broadcasts and polling, set web.broadcast.job.status.enabled to false and web.job.polling.interval to a zero or negative value. Users will then need to manually refresh the Job Status web page using their web browser.

Other Improvements

Now using variable-length text fields in the mySQL database for string data that may exceed 255 characters.
Updated the MPFImageReader tool to use OpenCV video capture behind the scenes to support reading data from HTTP URLs.
Python components can now include pre-built wheel files in the plugin package.
We now use a Jenkinsfile Groovy script for our Jenkins build process. This allows us to use revision control for our continuous integration process and share that process with the open source community.
Added remote.media.download.retries and remote.media.download.sleep system properties that can be used to configure how the Workflow Manager will attempt to retry downloading remote media if it encounters a problem.
Artifact extraction now uses MPFVideoCapture, which employs various fallback strategies for extracting frames in cases where a video is not well-formed or corrupted. For components that use MPFVideoCapture, this enables better consistency between the frames they process and the artifacts that are later extracted.

Bug Fixes

Jobs now properly end in ERROR if an invalid media URL is provided or there is a problem accessing remote media.
Jobs now end in COMPLETE_WITH_ERRORS when a detection splitter error occurs due to missing system properties.
Components can now include their own version of the Google Protobuf library. It will not conflict with the version used by the rest of OpenMPF.
The Java component executor now sets the proper job id in the job name instead of using the ActiveMQ message request id.
The Java component executor now sets the run directory using setRunDirectory().
Actions can now be properly added using an "extras" component. An extras component only includes a descriptor.json file and declares Actions, Tasks, and Pipelines using other component algorithms.
Refer to the items listed in the ActiveMQ section.
Refer to the addition of track-level properties and confidence in the JSON Output Object section.

Known Issues

[#745] In environments where thousands of jobs are processed, users have observed that, on occasion, pending sub-job messages in ActiveMQ queues are not processed until a new job is created. The reason is currently unknown.
[#544] Image artifacts retain some permissions from source files available on the local host. This can result in some of the image artifacts having executable permissions.
[#604] The Sphinx component cannot be unregistered because $MPF_HOME/plugins/SphinxSpeechDetection/lib is owned by root on a deployment machine.
[#623] The Nodes UI does not work correctly when [POST] /rest/nodes/config is used at the same time. This is because the UI's state is not automatically updated to reflect changes made through the REST endpoint.
[#783] The Tesseract OCR Text Detection Component has a known issue because it uses Tesseract 3. If a combination of languages is specified using TESSERACT_LANGUAGE, and one of the languages is Arabic, then Arabic must be specified last. For example, for English and Arabic, eng+ara will work, but ara+eng will not.
[#784] Sometimes services do not start on OpenMPF nodes, and those services cannot be started through the Nodes web UI. This is not a Docker-specific problem, but it has been observed in a Docker swarm deployment when auto-configuration is enabled. The workaround is to restart the Docker swarm deployment, or remove the entire node in the Nodes UI and add it again.

OpenMPF 2.1.x

2.1.0: June 2018

NOTE: If building this release on a machine used to build a previous version of OpenMPF, then please run sudo pip install --upgrade pip to update to at least pip 10.0.1. If not, the OpenMPF build script will fail to properly download .whl files for Python modules.

Documentation

Added the Python Batch Component API.
Added the Node Guide.
Added the GPU Support Guide.
Updated the Install Guide with an "(Optional) Install the NVIDIA CUDA Toolkit" section.
Renamed Admin Manual to Admin Guide for consistency.

Python Batch Component API

Developers can now write batch components in Python using the mpf_component_api module.
Dependencies can be specified in a setup.py file. OpenMPF will automatically download the .whl files using pip at build time.
When deployed, a virtualenv is created for the Python component so that it runs in a sandbox isolated from the rest of the system.
OpenMPF ImageReader and VideoCapture tools are provided in the mpf_component_util module.
Example Python components are provided for reference.

Spare Nodes

Spare nodes can join and leave an OpenMPF cluster while the Workflow Manager is running. You can create a spare node by cloning an existing OpenMPF child node. Refer to the Node Guide.
Note that changes made using the Component Registration web page only affect core nodes, not spare nodes. Core nodes are those configured during the OpenMPF installation process.
Added mpf list-nodes command to list the core nodes and available spare nodes.
OpenMPF now uses the JGroups FILE_PING protocol for peer discovery instead of TCPPING. This means that the list of OpenMPF nodes no longer needs to be fully specified when the Workflow Manager starts. Instead, the Workflow Manager, and Node Manager process on each node, use the files in $MPF_HOME/share/nodes to determine which nodes are currently available.
Updated JGroups from 3.6.4. to 4.0.11.
The environment variables specified in /etc/profile.d/mpf.sh have been simplified. Of note, ALL_MPF_NODES has been replaced by CORE_MPF_NODES.

Default Detection System Properties

The detection properties that specify the default values when creating new jobs can now be updated at runtime without restarting the Workflow Manager. Changing these properties will only have an effect on new jobs, not jobs that are currently running.
These default detection system properties are separated from the general system properties in the Properties web page. The latter still require the Workflow Manager to be restarted for changes to take effect.
The Apache Commons Configuration library is now used to read and write properties files. When defining a property value using an environment variable in the Properties web page, or $MPF_HOME/config/mpf-custom.properties, be sure to prepend the variable name with env:. For example:

detection.models.dir.path=${env:MPF_HOME}/models/

Alternatively, you can define system properties using other system properties:

detection.models.dir.path=${mpf.share.path}/models/

Adaptive Frame Interval

The FRAME_RATE_CAP property can be used to set a threshold on the maximum number of frames to process within one second of the native video time. This property takes precedence over the user-provided / pipeline-provided value for FRAME_INTERVAL. When the FRAME_RATE_CAP property is specified, an internal frame interval value is calculated as follows:

calcFrameInterval = max(1, floor(mediaNativeFPS / frameRateCapProp));

FRAME_RATE_CAP may be disabled by setting it <= 0. FRAME_INTERVAL can be disabled in the same way.
If FRAME_RATE_CAP is disabled, then FRAME_INTERVAL will be used instead.
If both FRAME_RATE_CAP and FRAME_INTERVAL are disabled, then a value of 1 will be used for FRAME_INTERVAL.

Darknet Component

This release includes a component that uses the Darknet neural network framework to perform detection and classification of objects using trained models.
Pipelines for the Tiny YOLO and YOLOv2 models are provided. Due to its large size, the YOLOv2 weights file must be downloaded separately and placed in $MPF_HOME/share/models/DarknetDetection in order to use the YOLOv2 pipelines. Refer to DarknetDetection/plugin-files/models/models.ini for more information.
This component supports a preprocessor mode and default mode of operation. If preprocessor mode is enabled, and multiple Darknet detections in a frame share the same classification, then those are merged into a single detection where the region corresponds to the superset region that encapsulates all of the original detections, and the confidence value is the probability that at least one of the original detections is a true positive. If disabled, multiple Darknet detections in a frame are not merged together.
Detections are not tracked across frames. One track is generated per detection.
This component supports an optional CLASS_WHITELIST_FILE property. When provided, only detections with class names listed in the file will be generated.
This component can be compiled with GPU support if the NVIDIA CUDA Toolkit is installed on the build machine. Refer to the GPU Support Guide. If the toolkit is not found, then the component will compile with CPU support only.
To run on a GPU, set the CUDA_DEVICE_ID job property, or set the detection.cuda.device.id system property, >= 0.
When CUDA_DEVICE_ID >= 0, you can set the FALLBACK_TO_CPU_WHEN_GPU_PROBLEM job property, or the detection.use.cpu.when.gpu.problem system property, to TRUE if you want to run the component logic on the CPU instead of the GPU when a GPU problem is detected.

Models Directory

The$MPF_HOME/share/models directory is now used by the Darknet and Caffe components to store model files and associated files, such as classification names files, weights files, etc. This allows users to more easily add model files post-deployment. Instead of copying the model files to $MPF_HOME/plugins/<component-name>/models directory on each node in the OpenMPF cluster, they only need to copy them to the shared directory once.
To add new models to the Darknet and Caffe component, add an entry to the respective <component-name>/plugin-files/models/models.ini file.

Packaging and Deployment

Python components are packaged with their respective dependencies as .whl files. This can be automated by providing a setup.py file. An example OpenCV Python component is provided that demonstrates how the component is packaged and deployed with the opencv-python module. When deployed, a virtualenv is created for the component with the .whl files installed in it.
When deploying OpenMPF, LD_LIBRARY_PATH is no longer set system-wide. Refer to Known Issues.

Web User Interface

Updated the Nodes page to distinguish between core nodes and spare nodes, and to show when a node is online or offline.
Updated the Component Registration page to list the core nodes as a reminder that changes will not affect spare nodes.
Updated the Properties page to separate the default detection properties from the general system properties.

Bug Fixes

Custom Action, task, and pipeline names can now contain "(" and ")" characters again.
Detection location elements for audio tracks and generic tracks in a JSON output object will now have a y value of 0 instead of 1.
Streaming health report and summary report timestamps have been corrected to represent hours in the 0-23 range instead of 1-24.
Single-frame .gif files are now segmented properly and no longer result in a NullPointerException.
LD_LIBRARY_PATH is now set at the process level for Tomcat, the Node Manager, and component services, instead of at the system level in /etc/profile.d/mpf.sh. Also, deployments no longer create /etc/ld.so.conf.d/mpf.conf. This better isolates OpenMPF from the rest of the system and prevents issues, such as being unable to use SSH, when system libraries are not compatible with OpenMPF libraries. The latter situation may occur when running yum update on the system, which can make OpenMPF unusable until a new deployment package with compatible libraries is installed.
The Workflow Manager will no longer generate an "Error retrieving the SingleJobInfo model" line in the log if someone is viewing the Job Status page when a job submitted through the REST API is in progress.

Known Issues

When multiple component services of the same type on the same node log to the same file at the same time, sometimes log lines will not be captured in the log file. The logging frameworks (log4j and log4cxx) do not support that usage. This problem happens more frequently on systems running many component services at the same time.
The following exception was observed:

com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: data_uri

Further debugging is necessary to determine the reason why that message was missing that field. The situation is not easily reproducible. It may occur when ActiveMQ and / or the system is under heavy load and sends duplicate messages in attempt to ensure message delivery. Some of those messages seem to end up in the dead letter queue (DLQ). For now, we've improved the way we handle messages in the DLQ. If OpenMPF can process a message successfully, the job is marked as COMPLETED_WITH_ERRORS, and the message is moved from ActiveMQ.DLQ to MPF.DLQ_PROCESSED_MESSAGES. If OpenMPF cannot process a message successfully, it is moved from ActiveMQ.DLQ to MPF.DLQ_INVALID_MESSAGES.

The mpf stop command will stop the Workflow Manager, which will in turn send commands to all of the available nodes to stop all running component services. If a service is processing a sub-job when the quit command is received, that service process will not terminate until that sub-job is completely processed. Thus, the service may put a sub-job response on the ActiveMQ response queue after the Workflow Manager has terminated. That will not cause a problem because the queues are flushed the next time the Workflow Manager starts; however, there will be a problem if the service finishes processing the sub-job after the Workflow Manager is restarted. At that time, the Workflow Manager will have no knowledge of the old job and will in turn generate warnings in the log about how the job id is "not known to the system" and/or "not found as a batch or a streaming job". These can be safely ignored. Often, if these messages appear in the log, then C++ services were running after stopping the Workflow Manager. To address this, you may wish to run sudo killall amq_detection_component after running mpf stop.

OpenMPF 2.0.x

2.0.0: February 2018

NOTE: Components built for previous releases of OpenMPF are not compatible with OpenMPF 2.0.0 due to Batch Component API changes to support generic detections, and changes made to the format of the descriptor.json file to support stream processing.

NOTE: This release contains basic support for processing video streams. Currently, the only way to make use of that functionality is through the REST API. Streaming jobs and services cannot be created or monitored through the web UI. Only the SuBSENSE component has been updated to support streaming. Only single-stage pipelines are supported at this time.

Documentation

Updated documents to distinguish the batch component APIs from the streaming component API.
Added the C++ Streaming Component API.
Updated the C++ Batch Component API to describe support for generic detections.
Updated the REST API with endpoints for streaming jobs.

Support for Generic Detections

C++ and Java components can now declare support for the UNKNOWN data type. The respective batch APIs have been updated with a function that will enable a component to process an MPFGenericJob, which represents a piece of media that is not a video, image, or audio file.
Note that these API changes make OpenMPF R2.0.0 incompatible with components built for previous releases of OpenMPF. Specifically, the new component executor will not be able to load the component logic library.

C++ Batch Component API

Added the following function to support generic detections:
- MPFDetectionError GetDetections(const MPFGenericJob &job, vector<MPFGenericTrack> &tracks)

Java Batch Component API

Added the following method to support generic detections:
- List<MPFGenericTrack> getDetections(MPFGenericJob job)

Streaming REST API

Added the following REST endpoints for streaming jobs:
- [GET] /rest/streaming/jobs: Returns a list of streaming job ids.
- [POST] /rest/streaming/jobs: Creates and submits a streaming job. Users can register for health report and summary report callbacks.
- [GET] /rest/streaming/jobs/{id}: Gets information about a streaming job.
- [POST] /rest/streaming/jobs/{id}/cancel: Cancels a streaming job.

Workflow Manager

Updated to support generic detections.
Updated Redis to store information about streaming jobs.
Added controllers for streaming job REST endpoints.
Added ability to generate health reports and segment summary reports for streaming jobs.
Improved code flow between the Workflow Manager and master Node Manager to support streaming jobs.
Added ActiveMQ queues to enable the C++ Streaming Component Executor to send reports and job status to the Workflow Manager.

Node Manager

Updated the master Node Manager and child Node Managers to spawn component services on demand to handle streaming jobs, cancel those jobs, and to monitor the status of those processes.
Using .ini files to represent streaming job properties and enable better communication between a child Node Manager and C++ Streaming Component Executor.

C++ Streaming Component API

Developed the C++ Streaming Component API with the following functions:
- MPFStreamingDetectionComponent(const MPFStreamingVideoJob &job): Constructor that takes a streaming video job.
- string GetDetectionType(): Returns the type of detection (i.e. "FACE").
- void BeginSegment(const VideoSegmentInfo &segment_info): Indicates the beginning of a new video segment.
- bool ProcessFrame(const cv::Mat &frame, int frame_number): Processes a single frame for the current video segment.
- vector<MPFVideoTrack> EndSegment(): Indicates the end of the current video segment.
Updated the C++ Hello World component to support streaming jobs.

C++ Streaming Component Executor

Developed the C++ Streaming Component Executor to load a streaming component logic library, read frames from a video stream, and exercise the component logic through the C++ Streaming Component API.
When the C++ Streaming Component Executor cannot read a frame from the stream, it will sleep for at least 1 millisecond, doubling the amount of sleep time per attempt until it reaches the stallTimeout value specified when the job was created. While stalled, the job status will be STALLED. After the timeout is exceeded, the job will be TERMINATED.
The C++ Streaming Component Executor supports FRAME_INTERVAL, as well as rotation, horizontal flipping, and cropping (region of interest) properties. Does not support USE_KEY_FRAMES.

Interoperability Package

Added the following Java classes to the interoperability package to simplify third party integration:
- JsonHealthReportCollection: Represents the JSON content of a health report callback. Contains one or more JsonHealthReport objects.
- JsonSegmentSummaryReport: Represents the JSON content of a summary report callback. Content is similar to the JSON output object used for batch processing.

SuBSENSE Component

The SuBSENSE component now supports both batch processing and stream processing.
Each video segment will be processed independently of the rest. In other words, tracks will be generated on a segment-by-segment basis and tracks will not carry over between segments.
Note that the last frame in the previous segment will be used to determine if there is motion in the first frame of the next segment.

Packaging and Deployment

Updated descriptor.json fields to allow components to support batch and/or streaming jobs. Components that use the old descriptor.json file format cannot be registered through the web UI.
Batch component logic and streaming component logic are compiled into separate libraries.
The mySQL streaming_job_request table has been updated with the following fields, which are used to populate the JSON health reports:
- status_detail: (Optional) A user-friendly description of the current job status.
- activity_frame_id: The frame id associated with the last job activity. Activity is defined as the start of a new track for the current segment.
- activity_timestamp: The timestamp associated with the last job activity.

Web User Interface

Added column names to the table that appears when the user clicks in the Media button associated with a job on the Job Status page. Now descriptive comments are provided when table cells are empty.

Bug Fixes

Upgraded Tika to 1.17 to resolve an issue with improper indentation in a Python file (rotation.py) that resulted in generating at least one error message per image processed. When processing a large number of images, this would generate may error messages, causing the Automatic Bug Reporting Tool daemon (abrtd) process to run at 100% CPU. Once in that state, that process would stay there, essentially wasting on CPU core. This resulted in some of the Jenkins virtual machines we used for testing to become unresponsive.

Known Issues

OpenCV 3.3.0 cv::imread() does not properly decode some TIFF images that have EXIF orientation metadata. It can handle images that are flipped horizontally, but not vertically. It also has issues with rotated images. Since most components rely on that function to read image data, those components may silently fail to generate detections for those kinds of images.
Using single quotes, apsotrophes, or double quotes in the name of an algorithm, action, task, or pipeline configured on an existing OpenMPF system will result in a failure to perform an OpenMPF upgrade on that system. Specifically, the step where pre-existing custom actions, tasks, and pipelines are carried over to the upgraded version of OpenMPF will fail. Please do not use those special characters while naming those elements. If this has been done already, then those elements should be manually renamed in the XML files prior to an upgrade attempt.
OpenMPF uses OpenCV, which uses FFmpeg, to connect to video streams. If a proxy and/or firewall prevents the network connection from succeeding, then OpenCV, or the underlying FFmpeg library, will segfault. This causes the C++ Streaming Component Executor process to fail. In turn, the job status will be set to ERROR with a status detail message of "Unexpected error. See logs for details". In this case, the logs will not contain any useful information. You can identify a segfault by the following line in the node-manager log:

2018-02-15 16:01:21,814 INFO [pool-3-thread-4] o.m.m.nms.streaming.StreamingProcess - Process: Component exited with exit code 139

To determine if FFmpeg can connect to the stream or not, run ffmpeg -i <stream-uri> in a terminal window. Here's an example when it's successful:

[mpf@localhost bin]$ ffmpeg -i rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov
ffmpeg version n3.3.3-1-ge51e07c Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4)
  configuration: --prefix=/apps/install --extra-cflags=-I/apps/install/include --extra-ldflags=-L/apps/install/lib --bindir=/apps/install/bin --enable-gpl --enable-nonfree --enable-libtheora --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-version3 --enable-shared --disable-libsoxr --enable-avresample
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
[rtsp @ 0x1924240] UDP timeout, retrying with TCP
Input #0, rtsp, from 'rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov':
  Metadata:
    title           : BigBuckBunny_115k.mov
  Duration: 00:09:56.48, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 12000 Hz, stereo, fltp
    Stream #0:1: Video: h264 (Constrained Baseline), yuv420p(progressive), 240x160, 24 fps, 24 tbr, 90k tbn, 48 tbc
At least one output file must be specified

Here's an example when it's not successful, so there may be network issues:

[mpf@localhost bin]$ ffmpeg -i rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov
ffmpeg version n3.3.3-1-ge51e07c Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4)
  configuration: --prefix=/apps/install --extra-cflags=-I/apps/install/include --extra-ldflags=-L/apps/install/lib --bindir=/apps/install/bin --enable-gpl --enable-nonfree --enable-libtheora --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-version3 --enable-shared --disable-libsoxr --enable-avresample
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
[tcp @ 0x171c300] Connection to tcp://184.72.239.149:554?timeout=0 failed: Invalid argument
rtsp://184.72.239.149/vod/mp4:BigBuckBunny_115k.mov: Invalid argument

Tika 1.17 does not come pre-packaged with support for some embedded image formats in PDF files, possibly to avoid patent issues. OpenMPF does not handle embedded images in PDFs, so that's not a problem. Tika will print out the following warnings, which can be safely ignored:

Jan 22, 2018 11:02:15 AM org.apache.tika.config.InitializableProblemHandler$3 handleInitializableProblem
WARNING: JBIG2ImageReader not loaded. jbig2 files will be ignored
See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for optional dependencies.
TIFFImageWriter not loaded. tiff files will not be processed
See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for optional dependencies.
J2KImageReader not loaded. JPEG2000 files will not be processed.
See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for optional dependencies.

OpenMPF 1.0.x

1.0.0: October 2017

Documentation

Updated the Build Guide with instructions for installing the latest JDK, latest JRE, FFmpeg 3.3.3, new codecs, and OpenCV 3.3.
Added an Acknowledgements section that provides information on third party dependencies leveraged by the OpenMPF.
Added a Feed Forward Guide that explains feed forward processing and how to use it.
Added missing requirements checklist content to the Install Guide.
Updated the README at the top level of each of the primary repositories to help with user navigation and provide general information.

Upgrade to FFmpeg 3.3.3 and OpenCV 3.3

Updated core framework from FFmpeg 2.6.3 to FFmpeg 3.3.3.
Added the following FFmpeg codecs: x256, VP9, AAC, Opus, Speex.
Updated core framework and components from OpenCV 3.2 to OpenCV 3.3. No longer building with opencv_contrib.

Feed Forward Behavior

Updated the Workflow Manager (WFM) and all video components to optionally perform feed forward processing for batch jobs. This allows tracks to be passed forward from one pipeline stage to the next. Components in the next stage will only process the frames associated with the detections in those tracks. This differs from the default segmenting behavior, which does not preserve detection regions or track information between stages.
To enable this behavior, the optional FEED_FORWARD_TYPE property must be set to FRAME, SUPERSET_REGION, or REGION. If set to FRAME then the components in the next stage will process the whole frame region associated with each detection in the track passed forward. If set to SUPERSET_REGION then the components in the next stage will determine the bounding box that encapsulates all of the detection regions in the track, and only process the pixel data within that superset region. If set to REGION then the components in the next stage will process the region associated with each detection in the track passed forward, which may vary in size and position from frame to frame.
The optional FEED_FORWARD_TOP_CONFIDENCE_COUNT property can be set to a number to limit the number of detections passed forward in a track. For example, if set to "5", then only the top 5 detections in the track will be passed forward and processed by the next stage. The top detections are defined as those with the highest confidence values, or if the confidence values are the same, those with the lowest frame index.
Note that setting the feed forward properties has no effect on the first pipeline stage because there is no prior stage that can pass tracks to it.

Caffe Component

Updated the Caffe component to process images in the BGR color space instead of the RGB color space. This addresses a bug found in OpenCV. Refer to the Bug Fixes section below.
Added support for processing videos.
Added support for an optional ACTIVATION_LAYER_LIST property. For each network layer specified in the list, the detectionProperties map in the JSON output object will contain one entry. The value is an encoded string of the JSON representation of an OpenCV matrix of the activation values for that layer. The activation values are obtained after the Caffe network has processed the frame data.
Added support for an optional SPECTRAL_HASH_FILE_LIST property. For each JSON file specified in the list, the detectionProperties map in the JSON output object will contain one entry. The value is a string of 0's and 1's representing the spectral hash calculated using the information in the spectral hash JSON file. The spectral hash is calculated using activation values after the Caffe network has processed the frame data.
Added a pipeline to showcase the above two features for the GoogLeNet Caffe model.
Removed the TRANSPOSE property from the Caffe component since it was not necessary.
Added red, green, and blue mean subtraction values to the GoogLeNet pipeline.

Use Key Frames

Added support for an optional USE_KEY_FRAMES property to each video component. When true the component will only look at key frames (I-frames) from the input video. Can be used in conjunction with FRAME_INTERVAL. For example, when USE_KEY_FRAMES is true, and FRAME_INTERVAL is set to "2", then every other key frame will be processed.

MPFVideoCapture and MPFImageReader Tools

Updated the MPFVideoCapture and MPFImageReader tools to handle feed forward properties.
Updated the MPFVideoCapture tool to handle FRAME_INTERVAL and USE_KEY_FRAMES properties.
Updated all existing components to leverage these tools as much as possible.
We encourage component developers to use these tools to automatically take care of common frame grabbing and frame manipulation behaviors, and not to reinvent the wheel.

Dead Letter Queue

If for some reason a sub-job request that should have gone to a component ends up on the ActiveMQ Dead Letter Queue ( DLQ), then the WFM will now process that failed request so that the job can complete. The ActiveMQ management page will now show that ActiveMQ.DLQ has 1 consumer. It will also show unconsumed messages in MPF.PROCESSED_DLQ_MESSAGES. Those are left for auditing purposes. The "Message Detail" for these shows the string representation of the original job request protobuf message.

Upgrade Path

Removed the Release 0.8 to Release 0.9 upgrade path in the deployment scripts.
Added support for a Release 0.9 to Release 1.0.0 upgrade path, and a Release 0.10.0 to Release 1.0.0 upgrade path.

Markup

Bounding boxes are now drawn along the interpolated path between detection regions whenever there are one or more frames in a track which do not have detections associated with them.
For each track, the color of the bounding box is now a randomly selected hue in the HSV color space. The colors are evenly distributed using the golden ratio.

Bug Fixes

Fixed a bug in OpenCV where the Caffe example code was processing images in the RGB color space instead of the BGR color space. Updated the OpenMPF Caffe component accordingly.
Fixed a bug in the OpenCV person detection component that caused bounding boxes to be too large for detections near the edge of a frame.
Resubmitting jobs now properly carries over configured job properties.
Fixed a bug in the build order of the OpenMPF project so that test modules that the WFM depends on are built before the WFM itself.
The Markup component draws bounding boxes between detections when a FRAME_INTERVAL is specified. This is so that the bounding box in the marked-up video appears in every frame. Fixed a bug where the bounding boxes drawn on non-detection frames appeared to stand still rather than move along the interpolated path between detection regions.
Fixed a bug on the OALPR license plate detection component where it was not properly handling the SEARCH_REGION_* properties.
Support for the MIN_GAP_BETWEEN_SEGMENTS property was not implemented properly. When the gap between two segments is less than this property value then the segments should be merged; otherwise, the segments should remain separate. In some cases, the exact opposite was happening. This bug has been fixed.

Known Issues

Because of the number of additional ActiveMQ messages involved, enabling feed forward for low resolution video may take longer than the non-feed-forward behavior.

OpenMPF 0.x.x

0.10.0: July 2017

WARNING: There is no longer a DEFAULT CAFFE ACTION, DEFAULT CAFFE TASK, or DEFAULT CAFFE PIPELINE. There is now a CAFFE GOOGLENET DETECTION PIPELINE and CAFFE YAHOO NSFW DETECTION PIPELINE, which each have a respective action and task.

NOTE: MPFImageReader has been re-enabled in this version of OpenMPF since we upgraded to OpenCV 3.2, which addressed the known issues with imread(), auto-orientation, and jpeg files in OpenCV 3.1.

Documentation

Added a Contributor Guide that provides guidelines for contributing to the OpenMPF codebase.
Updated the Java Batch Component API with links to the example Java components.
Updated the Build Guide with instructions for OpenCV 3.2.

Upgrade to OpenCV 3.2

Updated core framework and components from OpenCV 3.1 to OpenCV 3.2.

Support for Animated gifs

All gifs are now treated as videos. Each gif will be handled as an MPFVideoJob.
Unanimated gifs are treated as 1-frame videos.
The WFM Media Inspector now populates the media_properties map with a FRAME_COUNT entry (in addition to the DURATION and FPS entries).

Caffe Component

Added support for the Yahoo Not Suitable for Work (NSFW) Caffe model for explicit material detection.
Updated the Caffe component to support the OpenCV 3.2 Deep Neural Network (DNN) module.

Future Support for Streaming Video

NOTE: At this time, OpenMPF does not support streaming video. This section details what's being / has been done so far to prepare for that feature.

The codebase is being updated / refactored to support both the current "batch" job functionality and new "streaming" job functionality.
- batch job: complete video files are written to disk before they are processed
- streaming job: video frames are read from a streaming endpoint (such as RTSP) and processed in near real time
The REST API is being updated with endpoints for streaming jobs:
- [POST] /rest/streaming/jobs: Creates and submits a streaming job
- [POST] /rest/streaming/jobs/{id}/cancel: Cancels a streaming job
- [GET] /rest/streaming/jobs/{id}: Gets information about a streaming job
The Redis and mySQL databases are being updated to support streaming video jobs.
- A batch job will never have the same id as a streaming job. The integer ids will always be unique.

Bug Fixes

The MOG and SuBSENSE component services could segfault and terminate if the USE_MOTION_TRACKING property was set to “1” and a detection was found close to the edge of the frame. Specifically, this would only happen if the video had a width and/or height dimension that was not an exact power of two.
- The reason was because the code downsamples each frame by a power of two and rounds the value of the width and height up to the nearest integer. Later on when upscaling detection rectangles back to a size that’s relative to the original image, the resized rectangle sometimes extended beyond the bounds of the original frame.

Known Issues

If a job is submitted through the REST API, and a user to logged into the web UI and looking at the job status page, the WFM may generate "Error retrieving the SingleJobInfo model for the job with id" messages.
- This is because the job status is only added to the HTTP session object if the job is submitted through the web UI. When the UI queries the job status it inspects this object.
- This message does not appear if job status is obtained using the [GET] /rest/jobs/{id} endpoint.
The [GET] /rest/jobs/stats endpoint aggregates information about all of the jobs ever run on the system. If thousands of jobs have been run, this call could take minutes to complete. The code should be improved to execute a direct mySQL query.

0.9.0: April 2017

WARNING: MPFImageReader has been disabled in this version of OpenMPF. Component developers should use MPFVideoCapture instead. This affects components developed against previous versions of OpenMPF and components developed against this version of OpenMPF. Please refer to the Known Issues section for more information.

WARNING: The OALPR Text Detection Component has been renamed to OALPR License Plate Text Detection Component. This affects the name of the component package and the name of the actions, tasks, and pipelines. When upgrading from R0.8 to R0.9, if the old OALPR Text Detection Component is installed in R0.8 then you will be prompted to install it again at the end of the upgrade path script. We recommend declining this prompt because the old component will conflict with the new component.

WARNING: Action, task, and pipeline names that started with MOTION DETECTION PREPROCESSOR have been renamed MOG MOTION DETECTION PREPROCESSOR. Similarly, WITH MOTION PREPROCESSOR has changed to WITH MOG MOTION PREPROCESSOR.

Documentation

Updated the REST API to reflect job properties, algorithm-specific properties, and media-specific properties.
Streamlined the C++ Batch Component API document for clarity and simplicity.
Completed the Java Batch Component API document.
Updated the Admin Guide and User Guide to reflect web UI changes.
Updated the Build Guide with instructions for GitHub repositories.

Workflow Manager

Added support for job properties, which will override pre-defined pipeline properties.
Added support for algorithm-specific properties, which will apply to a single stage of the pipeline and will override job properties and pre-defined pipeline properties.
Added support for media-specific properties, which will apply to a single piece and media and will override job properties, algorithm-specific properties, and pre-defined pipeline properties.
Components can now be automatically registered and installed when the web application starts in Tomcat.

Web User Interface

The "Close All" button on pop-up notifications now dismisses all notifications from the queue, not just the visible ones.
Job completion notifications now only appear for jobs created during the current login session instead of all jobs.
The ROTATION, HORIZONTAL_FLIP, and SEARCH_REGION_* properties can be set using the web interface when creating a job. Once files are selected for a job, these properties can be set individually or by groups of files.
The Node and Process Status page has been merged into the Node Configuration page for simplicity and ease of use.
The Media Markup results page has been merged into the Job Status page for simplicity and ease of use.
The File Manager UI has been improved to handle large numbers of files and symbolic links.
The side navigation menu is now replaced by a top navigation bar.

REST API

Added an optional jobProperties object to the /rest/jobs/ request which contains String key-value pairs which override the pipeline's pre-configured job properties.
Added an optional algorithmProperties object to the /rest/jobs/ request which can be used to configure properties for specific algorithms in the pipeline. These properties override the pipeline's pre-configured job properties. They also override the values in the jobProperties object.
Updated the /rest/jobs/ request to add more detail to media, replacing a list of mediaUri Strings with a list of media objects, each of which contains a mediaUri and an optional mediaProperties map. The mediaProperties map can be used to configure properties for the specific piece of media. These properties override the pipeline's pre-configured job properties, values in the jobProperties object, and values in the algorithmProperties object.
Streamlined the actions, tasks, and pipelines endpoints that are used by the web UI.

Flipping, Rotation, and Region of Interest

The ROTATION, HORIZONTAL_FLIP, and SEARCH_REGION_* properties will no longer appear in the detectionProperties map in the JSON detection output object. When applied to an algorithm these properties now appear in the pipeline.stages.actions.properties element. When applied to a piece of media these properties will now appear in the the media.mediaProperties element.
The OpenMPF now supports multiple regions of interest in a single media file. Each region will produce tracks separately, and the tracks for each region will be listed in the JSON output as if from a separate media file.

Component API

Java Batch Component API is functionally complete for third-party development, with the exception of Component Adapter and frame transformation utilities classes.
Re-architected the Java Batch Component API to use a more traditional Java method structure of returning track lists and throwing exceptions (rather than modifying input track lists and returning statuses), and encapsulating job properties into MPFJob objects:
- List<MPFVideoTrack> getDetections(MPFVideoJob job) throws MPFComponentDetectionError
- List<MPFAudioTrack> getDetections(MPFAudioJob job) throws MPFComponentDetectionError
- List<MPFImageLocation> getDetections(MPFImageJob job) throws MPFComponentDetectionError
Created examples for the Java Batch Component API.
Reorganized the Java and C++ component source code to enable component development without the OpenMPF core, which will simplify component development and streamline the code base.

JSON Output Objects

The JSON output object for the job now contains a jobProperties map which contains all properties defined for the job in the job request. For example, if the job request specifies a CONFIDENCE_THRESHOLD of then the jobProperties map in the output will also list a CONFIDENCE_THRESHOLD of 5.
The JSON output object for the job now contains a algorithmProperties element which contains all algorithm-specific properties defined for the job in the job request. For example, if the job request specifies a FRAME_INTERVAL of 2 for FACECV then the algorithmProperties element in the output will contain an entry for "FACECV" and that entry will list a FRAME_INTERVAL of 2.
Each JSON media output object now contains a mediaProperties map which contains all media-specific properties defined by the job request. For example, if the job request specifies a ROTATION of 90 degrees for a single piece of media then the mediaProperties map for that piece of piece will list a ROTATION of 90.
The content of JSON output objects are now organized by detection type (e.g. MOTION, FACE, PERSON, TEXT, etc.) rather than action type.

Caffe Component

Added support for flip, rotation, and cropping to regions of interest.
Added support for returning multiple classifications per detection based on user-defined settings. The classification list is in order of decreasing confidence value.

New Pipelines

New SuBSENSE motion preprocessor pipelines have been added to components that perform detection on video.

Packaging and Deployment

Actions.xml, Algorithms.xml, nodeManagerConfig.xml, nodeServicesPalette.json, Pipelines.xml, and Tasks.xml are no longer stored within the Workflow Manager WAR file. They are now stored under $MPF_HOME/data. This makes it easier to upgrade the Workflow Manager and makes it easier for users to access these files.
Each component can now be optionally installed and registered during deployment. Components not registered are set to the UPLOADED state. They can then be removed or registered through the Component Registration page.
Java components are now packaged as tar.gz files instead of RPMs, bringing them into alignment with C++ components.
OpenMPF R0.9 can be installed over OpenMPF R0.8. The deployment scripts will determine that an upgrade should take place.
- After the upgrade, user-defined actions, tasks, and pipelines will have "CUSTOM" prepended to their name.
- The job_request table in the mySQL database will have a new "output_object_version" column. This column will have "1.0" for jobs created using OpenMPF R0.8 and "2.0" for jobs created using OpenMPF R0.9. The JSON output object schema has changed between these versions.
Reorganized source code repositories so that component SDKs can be downloaded separately from the OpenMPF core and so that components are grouped by license and maturity. Build scripts have been created to streamline and simplify the build process across the various repositories.

Upgrade to OpenCV 3.1

The OpenMPF software has been ported to use OpenCV 3.1, including all of the C++ detection components and the markup component. For the OpenALPR license plate detection component, the versions of the openalpr, tesseract, and leptonica libraries were also upgraded to openalpr-2.3.0, tesseract-3.0.4, and leptonica-1.7.2. For the SuBSENSE motion component, the version of the SuBSENSE library was upgraded to use the code found at this location: https://bitbucket.org/pierre_luc_st_charles/subsense/src.

Bug Fixes

MOG motion detection always detected motion in frame 0 of a video. Because motion can only be detected between two adjacent frames, frame 1 is now the first frame in which motion can be detected.
MOG motion detection never detected motion in the first frame of a video segment (other than the first video segment because of the frame 0 bug described above). Now, motion is detected using the first frame before the start of a segment, rather than the first frame of the segment.
The above bugs were also present in SuBSENSE motion detection and have been fixed.
SuBSENSE motion detection generated tracks where the frame numbers were off by one. Corrected the frame index logic.
Very large video files caused an out of memory error in the system during Workflow Manager media inspection.
A job would fail when processing images with an invalid metadata tag for the camera flash setting.
Users were permitted to select invalid file types using the File Manager UI.

Known Issues

MPFImageReader does not work reliably with the current release version of OpenCV 3.1: In OpenCV 3.1, new functionality was introduced to interpret EXIF information when reading jpeg files.
There are two issues with this new functionality that impact our ability to use the OpenCV imread() function with MPFImageReader:
- First, because of a bug in the OpenCV code, reading a jpeg file that contains exif information could cause it to hang. (See https://github.com/opencv/opencv/issues/6665.)
- Second, it is not possible to tell the imread()function to ignore the EXIF data, so the image it returns is automatically rotated. (See https://github.com/opencv/opencv/issues/6348.) This results in the MPFImageReader applying a second rotation to the image due to the EXIF information.
To address these issues, we developed the following workarounds:
- Created a version of the MPFVideoCapture that works with an MPFImageJob. The new MPFVideoCapture can pull frames from both video files and images. MPFVideoCapture leverages cv::VideoCapture, which does not have the two issues described above.
- Disabled the use of MPFImageReader to prevent new users from trying to develop code leveraging this previous functionality.