AIR_Sample_Exchange

Overview

The AIR Evidence Creator will submit samples AI Results (SR objects) produced by its system and the associated DICOM images. The goal of this test is to prepare Consumer actors so they are not surprised during Connectathon.

In order to facilitate this testing, Creators....please submit your samples 2-3 weeks before the usual preparatory test deadlines. 

Sample  Creator(s) Consumer(s)

DICOM SR(s)

DICOM Segmentation, Parametric Map or KOS IOD(s)

DICOM image(s)

AIR Evidence Creator

 AIR Image Mgr, Image Display, Imaging Doc Consumer

 

Instructions for Evidence Creators:

FIRST, prepare the sample(s)-->

Evidence Creator system must provide a sample that includes both DIOM SRs and image objects.  Although it is likely that an Acquisition Modality (ie not Evidence Creator) will be the source of images for which AIR SR objects are created, we ask the Evidence Creator to submit them together as a 'sample'

The AI result SR objects(s) Evidence Creator should be representative of what the system would created during expected clinical use when an AI algorithm is run.

  1. You will upload sample SR object(s) into Gazelle Test Management under Testing-->Sample exchange in the "AIR" entry.   If your Evidence Creator implements more than one AI algorithm (i.e. produces different AI result SRs depending on the algoritm), then the Evidence  Creator shall create separate Sample entries for each AI algorithm)
  2. In the sample entry, identify the IOD and primitives for the sample.  You will see "attributes" you can select; choose all that apply to this sample:
    • IOD - Comprehensive 3D SR Storage (1.2.840.10008.5.1.4.1.1.88.34)
    • IOD - Segmentation Storage (1.2.840.10008.5.1.4.1.1.66.4)
    • IOD - Parametric Map Storage (1.2.840.10008.5.1.4.1.1.30)
    • IOD - Key Object Selection Document Storage (1.2.840.10008.5.1.4.1.1.88.59)
    • Primitive - Qualitative Findings
    • Primitive - Measurements
    • Primitive - Locations
    • Primitive - Regions
    • Primitive - Tracking Identifiers
    • Primitive - Parametric Maps

    3. Validate the SR using EVSClient (green arrow in on the sample page).  You must select the DICOM validator from Pixelmed.  During the Connectathon, we will be looking for a validation result with no errors (warnings are OK).

    4. Evidence Creator of the AI result SR object should also upload the images of the study to which the SR pertains.

    5. If you have this ability, render the objects you produced as a reference. The goal during Connectathon is for an Image Display partner to examine your rendered images and SR objects and compare that to what their software produces. Perform a screen capture and/or save as JPEG or other reasonable format.  Upload the screenshot onto the sample page in Gazelle Test Management.

    Repeat Steps 1 - 5 for each AI algorithm and resulting AI sample.

    Finally, create a short txt file indicating you have completed the upload step. Upload '''that''' txt file into Gazelle Test Management as the result file for this preparatory test.

    There is no specific evaluation for this test.  Feedback comes as your partner Image Display and Image Manager partners test with your images in their lab.

     

    Instructions for Consumers:

    1. Find sample SR objects and images uploaded by other vendors in Gazelle Test Management under Testing-> Sample exchange.    On the Samples available for rendering tab under the AIR entry, you willl find samples and screen captures submitted your Evidence Creator test partners.  This page will evolve as vendors add samples, so be patient. 
    2. Retrieve the DICOM or zip files created by the other vendors.  Refer to these help pages for details on this task.
    3. Examine/import/render them so that you are confident your software understands the content.
    4. When you are finished, create a .txt or other file that lists the samples you tested with, and a brief comment on your result (eg "ok", or "could not render"). Upload that txt file into Gazelle Test Management as the result file for this preparatory test.
    5. If you find issues with the images, send an email to the Connectathon Technical Manager.
    6. The evaluation of this test is performed by examining the text you provided to make sure you made a good faith effort to review the sample images. If you find problems with some samples, we will give your feedback to the Evidence Creator test partner.
    7. The goal is no surprises.

    Finally, for both Creators & Consumers

    1. In Gazelle Test Management, find the entry for this test instance, and change the status to "Verified by Vendor".  This is a signal to the Technical Manager that you have completed this task.