This article provides information about the different file processing queues
Introduction
All processing database queues and folders are automatically monitored and Xpertise are notified if there are problems, i.e. if files remain in queues or folders for longer than a configured time or if processing errors are logged. As a result, problems are generally addressed by Xpertise staff if and when they occur, without the need for you to contact us. If you think that there are processing problems that we have missed, then create a support ticket. But read this article before you do.
Monitoring the queues
To find out how many files are in the processing queues and for how long, go to Logs in back office and click on the Processing tab. This view provides a brief overview of files that are queued to be processed. Note that this function is available in Infradox version 32.9 or later.
Inserts data read
If new files enter the ingestion process, metadata is read and inserted into a processing database table. Metadata may be read from the uploaded files, from sidecar files (i.e. a single file with metadata is uploaded to accompany each master file), or from bulk metadata files. How the data is read depends on how the system is configured for you. Please always follow the instructions that were given to you.
Once the metadata is in this queue, the system will start generating thumbnails and when this is done (to prevent files from appearing online before there are thumbnails and previews), the data is moved to the “Inserts waiting” database table.
If files do not appear in this queue within a few minutes:
The service that scans the upload folders is most likely not running. If this service is not running then the server software may be undergoing maintenance. In case of errors, Xpertise are automatically notified. Please wait for this service to start again.
If files remain in this queue for a long time:
- If your ingestion process requires you to upload separate poster files (e.g. for video) then make sure that you have uploaded those files too.
- If your ingestion process is configured to read sidecar files, files remain in the queue until the sidecar files is uploaded.
- If your ingestion process is configured to read bulk metadata files, files remain in the queue until the data could be read from the bulk file. Make sure you upload the file and make sure that the contents of the file are correct and that there’s metadata for all the uploaded master files.
- If you are sure that you have done all of the above, then any of the following problems may have occurred:
– there were processing errors, possibly caused by corrupt files,
– master file storage locations may be unavailable (e.g. storage is full or off line),
– file processing services may have stopped.
Xpertise will have been automatically notified of such problems. You may create a support ticket if you need help.
Inserts waiting
Files in this queue table are waiting to be processed or are currently being processed. Processing consists of a number of steps, among them supplier matching, processing data rules (that you can configure in backoffice) and updating the full text index. When this process is complete, files may already appear online but additional processing steps may need to be completed.
ISPA updates waiting
The metadata of both new and changed files require a number of processing steps that will not affect the full text index. Depending on your configuration, these steps may involve extracting unique words, creating canonical URL’s, generating similar files data, automatic creation/updating of sets (galleries), localisation, controlled vocabulary processing and so on. Files in this queue are waiting to be processed and processing has not yet begun.
If files remain in this queue for a long time:
If the column “ISPA currently processing” has no files, and there are files in the “ISPA updates waiting” queue for a long time, then the ISPA processing service may have stopped. If this service has been stopped manually this may be because of maintenance and processing will not continue automatically. If the service has stopped because of an exception, then the service will automatically start again within a few minutes. If there are unrecoverable errors then Xpertise needs to check the error logs for you.
If there are many files in this queue or the “currently processing” queue:
There may be a lot of files waiting to be processed if you, your agents or contributors have uploaded many files. Files that were updated with the batch edit function are also in the processing queues, just as files for which you have requested reprocessing from Supplier management. The time needed to executed certain steps (such as processing similars) will increase as the number of online files grows. You can request certain processing steps to be disabled, for example if you don’t want to use the similar files function.
If there were a lot of entries waiting and that number suddenly dropped
This may happen if orphan entries were automatically removed. E.g. there may have been entries queued for files that have already been deleted before processing started or finished.
ISPA currently processing
This column shows the number of files for which processing of the steps described above has started, but not all steps have been completed. Note that ISPA processes a maximum number of entries and it will wait for all entries currently being processed to complete. Only then will it start processing the next block of entries. The default block size is 100 entries. If you’re website is on virtual private (dedicated) web server you may request a larger block size to be configured to improve processing throughput. Also note that some steps are executed at the same time but some steps need to wait for previous steps to complete.
Updates and Deletes waiting
These columns will show the number of files that were already online and that are now waiting to be deleted or for the data to be reprocessed (i.e. after changing the metadata). If you update the metadata of files, these files may also need to be reprocessed by ISPA as well (see ISPA updates waiting). For example to remove references to unique words, similars data, galleries, lightboxes, carts and so on.
Processing folders
One or more folders will be listed, but only if the there are errors or possible problems as described below.
Error folders
The processing folders are monitored to detect problems with files in the ingestion workflow. There may be files in error folders. This may happen for example if an uploaded file is corrupt. If there are files that have been moved to an error folder, then you’ll see this on the Processing page. Support is automatically notified of such conditions, so you normally you will not have to report this. Files will remain in error folders until the problem has been addressed by Support.
Possible problem folders
If there are files in the processing folders on the server that have been waiting to be processed longer than a configured number of minutes, then these folders will also display on the Processing page. Note that files may be longer than usual in the processing folders if many new files were uploaded. This doesn’t necessarily indicate a problem. The folder reported gets updated after the configured number of minutes, so it may be that folders appear on your overview although the files were in fact processed already. Support is automatically notified if files remain in the processing folders for longer than usual, and support will address any problems, you may – but you don’t have to – create a support ticket.
Most recent processing jobs
At the bottom of the page you’ll find a list of the most recent processing jobs (no more than 15). The job log keeps track of processing of new files, updates and deletes. The right-most column shows the status of the job.
