productAI

Batch Processing

When the user needs to process large amount of images but does not require realtime responses, it is recommended to use ProductAI Batch Processing.
The advantages are:

  • No need to call API mutiple times, just upload all the dataset through Console/SDK/API in one operation.
  • The ProductAI backend leverages batch-optimizing strategy, users will experience a much shorter total process time.
  • The ProductAI backend resources will be scheduled with an on-demand scheme, let’s save the Earth together.

Supported Services

Be noticed, currently part of the ProductAI services can be scheduled in Batch Processing. Here are the supported services:

User can fetch supported service list through SDK/API

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)
api = cli.get_batch_api()
resp = api.get_services()
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->listBatchServices();
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initilizing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Fetch supported service list
GetSupportServicesRequest request = new GetSupportServicesRequest();
GetSupportServicesResponse response = client.getResponse(request);
String[] serviceIds = response.getSupportServiceIds();
// String json=response.getResponseJsonString();
using MalongTech.ProductAI.Core;
using MalongTech.ProductAI.API.Entity;
using MalongTech.ProductAI.API.Entity.Batch;

IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new GetSupportServicesRequest();
var response = client.GetResponse(request);
curl -X GET \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
https://api.productai.com/batch/_1000001/services

Request

N/a

Response

Success, will return a HTTP 200 status code

{
"data": [
"_0000159",
...
],
"is_error": 0,
"request_id": "3c0a99a8-e634-11e7-9fe1-ee0ab1643ad8"
}
FieldTypeDescription
dataArrayContains IDs of supported service
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of this request, this can be leveraged in product issue debugging with ProductAI support team

Error Code

HTTP status code is used by SDK/API to notify the client whether current API call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess

Create Task

Create a Batch Processing task through uploading a csv file

Note: at least two unique image urls should be filled in this csv file

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# create batch processing task context
api = cli.get_batch_api()

# submit a set of data to create a new task
urls = [
['https://xxx.jpg'],
['https://yyy.jpg']
]
resp = api.prepare(SERVICE_ID, urls)

# or submit a csv file, each line is an url
with open('batch.csv') as f:
resp = api.prepare_by_file(SERVICE_ID, f)
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');

// submit a set of data to create a new task
$task = $product_ai->prepareBatchTask(SERVICE_ID, [
'https://xxx.jpg',
'https://yyy.jpg',
]);

// or submit a csv file, each line is an url
$task = $product_ai->prepareBatchTask(SERVICE_ID, $image_urls='batch.csv')
import java.io.File;
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initilizing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Create Batch Processing task
CreateTaskByFileRequest request = new CreateTaskByFileRequest(SERVICE_ID, new File("batch.csv"));
CreateTaskByFileResponse response = client.getResponse(request);

// Get task Id
String taskId=response.getTaskInfo().getTaskId();
IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new CreateTaskByFileRequest
{
ServiceId=SERVICE_ID,
CsvFile=new System.IO.FileInfo(@".\batch.csv")
};

var response = client.GetResponse(request);
curl -X POST \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
-F "service_id=SERVICE_ID" \
-F "urls=@batch.csv" \
https://api.productai.com/batch/_1000001/task/prepare

Request

ArgumentTypeDescriptionRequiredLimitation
service_idStringService ID which is in use by current Batch Processing taskYesN/a
urlsArrayarray of urls for batch processingYesat least two unique image urls

Response

Return Http Status Code 200, When Success

{
"data": {
"estimated_processing_time": 1,
"estimated_waiting_time": 0,
"state": "RECEIVED",
"task_id": "6b166b74-fd00-4079-b4ee-9fd54261713b",
"total": 2
},
"is_error": 0,
"request_id": "56320a68-e6f3-11e7-9fe1-ee0ab1643ad8"
}
FieldTypeDescription
dataDictionaryContains error details,refer to data fields
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of this request, this can be leveraged in product issue debugging with ProductAI support team

data fiels

FieldTypeDescription
estimated_processing_timeIntegerEstimated processing time
estimated_waiting_timeIntegerEstimated waiting time
stateStringcurrent task status
task_idStringThe unique ID of this task, all the following options on this task are depending on it
totalIntegerTotal amount of images pending on process

Task status

StatusDescription
RECEIVEDTask received
PENDINGTask pending
STARTEDTask started
SUCCESSTask processed successfully
REVOKEDTask revoked
FAILURETask processed with failure

Return Http Status Code Non-200, When Failed

{
"data": {
"error_code": 400005,
"message": "1 URLs are duplicated or verification failed.",
"urls_duplicated": {},
"urls_verify_failed": {
"0": "h"
}
},
"is_error": 1,
"request_id": "d60031a6-e6ef-11e7-8326-9e19da6007e9"
}
FieldTypeDescription
dataDictionaryContains detailed infomation,refer to data fields
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of current api call, this can be leveraged in product issue debugging with ProductAI support team

data fields

FieldTypeDescription
error_codeIntegerDetailed error code
messageStringError message
urls_duplicatedArrayContains all the duplicated urls
urls_verify_failedArrayContains all the invalid urls

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess
400400001Missing service_id
400400002Missing urls
400400003url parameter is not encoded in utf-8
400400004url array is empty
400400005Invalid or duplicated url(s) in url array
400400007Service not exist for service_id
400400008Batch processing is not supported for service_id
400400009Quota exceeded. There is a daily batch processing quota for each account
400400010There is a running batch processing task with the same service of service_id

Start Task

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing task API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.apply(task_id)
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->applyBatchTask(task_id);
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Launch batch processing task
StartTaskRequest request = new StartTaskRequest(task_id);
StartTaskResponse response = client.getResponse(request);
// String json=response.getResponseJsonString();
TaskDetailInfo taskInfo = response.getTaskInfo();
IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new StartTaskRequest
{
TaskId=task_id
};

var response = client.GetResponse(request);
curl -X POST \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
-F "task_id=TASK_ID" \
https://api.productai.com/batch/_1000001/task/apply

Request

ArgumentTypeDescriptionRequiredLimitation
task_idStringBatch processing task IDYesN/a

Response

Same as Create Task

Error Code

HTTP status code is used by API/SDK to notify client whether current api call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess
400400009Quota exceeded. There is a daily quota for batch processing of each account
400400010There is a running batch processing task with the same service of service_id
400400011Task of task_id does not exist

Query Task

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.get_task_info(task_id)
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->getBatchTaskInfo(task_id);
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Query status of batch processing task
PollTaskStatusRequest request = new PollTaskStatusRequest(task_id);
PollTaskStatusResponse response = client.getResponse(request);
//String json=response.getResponseJsonString();
TaskDetailInfo taskInfo = response.getTaskInfo();
IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new PollTaskStatusRequest
{
TaskId=task_id
};

var response = client.GetResponse(request);
curl -X GET \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
https://api.productai.com/batch/_1000001/task/info/TASK_ID

Request

ArgumentTypeDescriptionRequiredLimitation
task_idStringBatch processing task IDYesN/a

Response

{
"data": {
"csv": 'http://xxx.csv',
"download_urls": [
"http://kkk.zip"
],
"received_time": "2017-12-22T08:37:26Z",
"service_id": "_0000159",
"state": "SUCCESS",
"success": 2,
"task_id": "6b166b74-fd00-4079-b4ee-9fd54261713b",
"total": 2
},
"is_error": 0,
"request_id": "772c5f32-e88a-11e7-8326-9e19da6007e9"
}
FieldTypeDescription
dataDictionaryRefer to data fields
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of this api call, it can be used in issue debugging with ProductAI support team

data fields

  • Fixed fields for any state of response
FieldTypeDescription
csvStringData csv file url for batch processing
received_timeStringTask accepted time
service_idStringID of service which is used by batch processing
stateStringBatch processing status,refer to task status
task_idStringTask ID
totalIntegerData amount of batch processing task
  • Response fields when state is SUCCESS
FieldTypeDescription
download_urlsArrayResult file urls after batch processing is done
successIntegerSuccessfully processed data amount of batch processing
  • Response fields when state is PENDING
FieldTypeDescription
estimated_processing_timeIntegerEstimated processing time
estimated_waiting_timeIntegerEstimated waiting time
  • Response fields when state is STARTED
FieldTypeDescription
estimated_processing_timeIntegerEstimated processing time

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess
400400011Task of task_id does not exist

Cancel Task

A task can be revoked only if this task is in PENDING status, it cannot be revoked once it is started

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.revoke(task_id)
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->revokeBatchTask($task_id=task_id);
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Cancel batch processing task:
CancelTaskRequest request = new CancelTaskRequest(task_id);
CancelTaskResponse response = client.getResponse(request);
TaskDetailInfo taskInfo = response.getTaskInfo();
IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new CancelTaskRequest
{
TaskId=task_id
};

var response = client.GetResponse(request);
curl -X POST \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
https://api.productai.com/batch/_1000001/task/revoke/TASK_ID

Request

ArgumentTypeDescriptionRequiredLimitation
task_idStringBatch processing task IDYesN/a

Response

{
"data": {
"code": 200001,
"args": ['6b166b74-fd00-4079-b4ee-9fd54261713b', ]
},
"is_error": 0,
"request_id": "772c5f32-e88a-11e7-8326-9e19da6007e9"
}
FieldTypeDescription
dataDictionaryRefer to data fields
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of this api call, it can be used in issue debugging with ProductAI support team

data fields

FieldTypeDescription
codeIntegerAlways return 200001 if canceling is success
argsArrayAlways return task ID

Error Code

HTTP status code is used by API/SDK to notify client whether current api call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess
400400011Task of task_id does not exist
400400012Task of task_id is not in PENDING status

List Tasks

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Return task(s) which was(were) launched within [start, end)
from datetime import datetime, timedelta
api.get_tasks(start=datetime.utcnow() - timedelta(days=7), end=datetime.utcnow())
use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->listBatchTasks($start=START, $end=END);
import java.util.HashMap;
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Query batch processing task
SearchTaskRequest request = new SearchTaskRequest("2018-01-01", "2018-01-10");
SearchTaskResponse response = client.getResponse(request);
HashMap<String, TaskDetailInfo> taskList = response.getTasks()
IProfile profile = new DefaultProfile
{
Version="1",
AccessKeyId=ACCESS_KEY_ID,
SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new SearchTaskRequest
{
StartDate=null,
EndDate=DateTime.Now.AddDays(1).Date
};

var response = client.GetResponse(request);
curl -X GET \
-H 'x-ca-version: 1.0' \
-H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
https://api.productai.com/batch/_1000001/tasks?start=2017-12-01&end=2017-12-31

Request

ArgumentTypeDescriptionRequiredLimitation
startDateTimeQuery with task accepted time (which is the time when prepare function is invoked), query start time. By default, it will return from the first record if this field is not providedNoTime format %Y-%m-%dT%H:%M:%SZ
endDateTimeQuery with task accepted time (which is the time when prepare function is invoked), query end time. By default it will return records until the last one if this field is not providedNoTime format %Y-%m-%dT%H:%M:%SZ

Response

{
"data": {
"5b6569dc-fe8c-40b3-b96b-7ae7e851fdca": {
...
},
"6b166b74-fd00-4079-b4ee-9fd54261713b": {
...
}
},
"is_error": 0,
"request_id": "693509c0-e88e-11e7-a6de-ee6487ba2bab"
}
FieldTypeDescription
dataDictionaryContains all the tasks, key is task ID, value is task details, format please refer to Query task - data fields
is_errorIntegersuccess is 0; failure is 1
request_idStringThe unique ID of this api call, it can be used in issue debugging with ProductAI support team

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Codeerror_codeDescription
200N/aSuccess