Batch Processing

When the user needs to process large amount of images but does not require realtime responses, it is recommended to use ProductAI Batch Processing.
The advantages are:

No need to call API mutiple times, just upload all the dataset through Console/SDK/API in one operation.
The ProductAI backend leverages batch-optimizing strategy, users will experience a much shorter total process time.
The ProductAI backend resources will be scheduled with an on-demand scheme, let’s save the Earth together.

Notes: Only support one task with same service id simultaneously. Please submit another task after one finish.

Supported Services

Be noticed, currently part of the ProductAI services can be scheduled in Batch Processing. Here are the supported services:

User can fetch supported service list through SDK/API

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)
api = cli.get_batch_api()
resp = api.get_services()

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->listBatchServices();

import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initilizing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Fetch supported service list
GetSupportServicesRequest request = new GetSupportServicesRequest();
GetSupportServicesResponse response = client.getResponse(request);
String[] serviceIds = response.getSupportServiceIds();
// String json=response.getResponseJsonString();

using MalongTech.ProductAI.Core;
using MalongTech.ProductAI.API.Entity;
using MalongTech.ProductAI.API.Entity.Batch;

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new GetSupportServicesRequest();
var response = client.GetResponse(request);

curl -X GET \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  https://api.productai.com/batch/_1000001/services

Request

N/a

Response

Success, will return a HTTP 200 status code

{
  "data": [
    "_0000159",
    ...
  ],
  "is_error": 0,
  "request_id": "3c0a99a8-e634-11e7-9fe1-ee0ab1643ad8"
}

Field	Type	Description
data	Array	Contains IDs of supported service
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of this request, this can be leveraged in product issue debugging with ProductAI support team

Error Code

HTTP status code is used by SDK/API to notify the client whether current API call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success

Create Task

Create a Batch Processing task through uploading a csv file

Note: at least two unique image urls should be filled in this csv file

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# create batch processing task context
api = cli.get_batch_api()

# submit a set of data to create a new task
urls = [
    ['https://xxx.jpg'],
    ['https://yyy.jpg']
]
resp = api.prepare(SERVICE_ID, urls)

# or submit a csv file, each line is an url
with open('batch.csv') as f:
  resp = api.prepare_by_file(SERVICE_ID, f)

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');

// submit a set of data to create a new task
$task = $product_ai->prepareBatchTask(SERVICE_ID, [
    'https://xxx.jpg',
    'https://yyy.jpg',
]);

// or submit a csv file, each line is an url
$task = $product_ai->prepareBatchTask(SERVICE_ID, $image_urls='batch.csv')

import java.io.File;
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initilizing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Create Batch Processing task
CreateTaskByFileRequest request = new CreateTaskByFileRequest(SERVICE_ID, new File("batch.csv"));
CreateTaskByFileResponse response = client.getResponse(request);

// Get task Id
String taskId=response.getTaskInfo().getTaskId();

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new CreateTaskByFileRequest
{
    ServiceId=SERVICE_ID,
    CsvFile=new System.IO.FileInfo(@".\batch.csv")
};

var response = client.GetResponse(request);

curl -X POST \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  -F "service_id=SERVICE_ID" \
  -F "urls=@batch.csv" \
  https://api.productai.com/batch/_1000001/task/prepare

Request

Argument	Type	Description	Required	Limitation
service_id	String	Service ID which is in use by current Batch Processing task	Yes	N/a
urls	Array	array of urls for batch processing	Yes	at least two unique image urls

Response

Return Http Status Code 200, When Success

{
  "data": {
    "estimated_processing_time": 1,
    "estimated_waiting_time": 0,
    "state": "RECEIVED",
    "task_id": "6b166b74-fd00-4079-b4ee-9fd54261713b",
    "total": 2
  },
  "is_error": 0,
  "request_id": "56320a68-e6f3-11e7-9fe1-ee0ab1643ad8"
}

Field	Type	Description
data	Dictionary	Contains error details, refer to data fields
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of this request, this can be leveraged in product issue debugging with ProductAI support team

data fiels

Field	Type	Description
estimated_processing_time	Integer	Estimated processing time
estimated_waiting_time	Integer	Estimated waiting time
state	String	current task status
task_id	String	The unique ID of this task, all the following options on this task are depending on it
total	Integer	Total amount of images pending on process

Task status

Status	Description
RECEIVED	Task received
PENDING	Task pending
STARTED	Task started
SUCCESS	Task processed successfully
REVOKED	Task revoked
FAILURE	Task processed with failure

Return Http Status Code Non-200, When Failed

{
  "data": {
    "error_code": 400005,
    "message": "1 URLs are duplicated or verification failed.",
    "urls_duplicated": {},
    "urls_verify_failed": {
      "0": "h"
    }
  },
  "is_error": 1,
  "request_id": "d60031a6-e6ef-11e7-8326-9e19da6007e9"
}

Field	Type	Description
data	Dictionary	Contains detailed infomation, refer to data fields
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of current api call, this can be leveraged in product issue debugging with ProductAI support team

data fields

Field	Type	Description
error_code	Integer	Detailed error code
message	String	Error message
urls_duplicated	Array	Contains all the duplicated urls
urls_verify_failed	Array	Contains all the invalid urls

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success
400	400001	Missing service_id
400	400002	Missing urls
400	400003	url parameter is not encoded in utf-8
400	400004	url array is empty
400	400005	Invalid or duplicated url(s) in url array
400	400007	Service not exist for service_id
400	400008	Batch processing is not supported for service_id
400	400009	Quota exceeded. There is a daily batch processing quota for each account
400	400010	There is a running batch processing task with the same service of service_id

Start Task

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing task API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.apply(task_id)

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->applyBatchTask(task_id);

import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Launch batch processing task
StartTaskRequest request = new StartTaskRequest(task_id);
StartTaskResponse response = client.getResponse(request);
// String json=response.getResponseJsonString();
TaskDetailInfo taskInfo = response.getTaskInfo();

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new StartTaskRequest
{
    TaskId=task_id
};

var response = client.GetResponse(request);

curl -X POST \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  -F "task_id=TASK_ID" \
  https://api.productai.com/batch/_1000001/task/apply

Request

Argument	Type	Description	Required	Limitation
task_id	String	Batch processing task ID	Yes	N/a

Response

Same as Create Task

Error Code

HTTP status code is used by API/SDK to notify client whether current api call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success
400	400009	Quota exceeded. There is a daily quota for batch processing of each account
400	400010	There is a running batch processing task with the same service of service_id
400	400011	Task of task_id does not exist

Query Task

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.get_task_info(task_id)

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->getBatchTaskInfo(task_id);

import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Query status of batch processing task
PollTaskStatusRequest request = new PollTaskStatusRequest(task_id);
PollTaskStatusResponse response = client.getResponse(request);
//String json=response.getResponseJsonString();
TaskDetailInfo taskInfo = response.getTaskInfo();

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new PollTaskStatusRequest
{
    TaskId=task_id
};

var response = client.GetResponse(request);

curl -X GET \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  https://api.productai.com/batch/_1000001/task/info/TASK_ID

Request

Argument	Type	Description	Required	Limitation
task_id	String	Batch processing task ID	Yes	N/a

Response

{
  "data": {
    "csv": 'http://xxx.csv',
    "download_urls": [
      "http://kkk.zip"
    ],
    "received_time": "2017-12-22T08:37:26Z",
    "service_id": "_0000159",
    "state": "SUCCESS",
    "success": 2,
    "task_id": "6b166b74-fd00-4079-b4ee-9fd54261713b",
    "total": 2
  },
  "is_error": 0,
  "request_id": "772c5f32-e88a-11e7-8326-9e19da6007e9"
}

Field	Type	Description
data	Dictionary	Refer to data fields
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of this api call, it can be used in issue debugging with ProductAI support team

data fields

Fixed fields for any state of response

Field	Type	Description
csv	String	Data csv file url for batch processing
received_time	String	Task accepted time
service_id	String	ID of service which is used by batch processing
state	String	Batch processing status, refer to task status
task_id	String	Task ID
total	Integer	Data amount of batch processing task

Response fields when state is SUCCESS

Field	Type	Description
download_urls	Array	Result file urls after batch processing is done
success	Integer	Successfully processed data amount of batch processing

Response fields when state is PENDING

Field	Type	Description
estimated_processing_time	Integer	Estimated processing time
estimated_waiting_time	Integer	Estimated waiting time

Response fields when state is STARTED

Field	Type	Description
estimated_processing_time	Integer	Estimated processing time

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success
400	400011	Task of task_id does not exist

Cancel Task

A task can be revoked only if this task is in PENDING status, it cannot be revoked once it is started

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Launch task by task_id which is returned through prepare function
api.revoke(task_id)

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->revokeBatchTask($task_id=task_id);

import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Cancel batch processing task: 
CancelTaskRequest request = new CancelTaskRequest(task_id);
CancelTaskResponse response = client.getResponse(request);
TaskDetailInfo taskInfo = response.getTaskInfo();

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new CancelTaskRequest
{
    TaskId=task_id
};

var response = client.GetResponse(request);

curl -X POST \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  https://api.productai.com/batch/_1000001/task/revoke/TASK_ID

Request

Argument	Type	Description	Required	Limitation
task_id	String	Batch processing task ID	Yes	N/a

Response

{
  "data": {
    "code": 200001,
    "args": ['6b166b74-fd00-4079-b4ee-9fd54261713b', ]
  },
  "is_error": 0,
  "request_id": "772c5f32-e88a-11e7-8326-9e19da6007e9"
}

Field	Type	Description
data	Dictionary	Refer to data fields
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of this api call, it can be used in issue debugging with ProductAI support team

data fields

Field	Type	Description
code	Integer	Always return 200001 if canceling is success
args	Array	Always return task ID

Error Code

HTTP status code is used by API/SDK to notify client whether current api call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success
400	400011	Task of task_id does not exist
400	400012	Task of task_id is not in PENDING status

List Tasks

Python

PHP

Java

C#

cURL

from productai import Client

cli = Client(ACCESS_KEY_ID, SECRET_KEY)

# Create batch processing API context
api = cli.get_batch_api()

# Return task(s) which was(were) launched within [start, end)
from datetime import datetime, timedelta
api.get_tasks(start=datetime.utcnow() - timedelta(days=7), end=datetime.utcnow())

use ProductAI;

$product_ai = new ProductAI\API(ACCESS_KEY_ID, SECRET_KEY, 'zh-Hans');
$task = $product_ai->listBatchTasks($start=START, $end=END);

import java.util.HashMap;
import cn.productai.api.core.*;
import cn.productai.api.core.enums.*;
import cn.productai.api.pai.entity.batch.*;

// Set API initializing parameters
IProfile profile = new DefaultProfile();
profile.setAccessKeyId(ACCESS_KEY_ID);
profile.setSecretKey(SECRET_KEY);
profile.setVersion("1");

IWebClient client = new DefaultProductAIClient(profile);

// Query batch processing task
SearchTaskRequest request = new SearchTaskRequest("2018-01-01", "2018-01-10");
SearchTaskResponse response = client.getResponse(request);
HashMap<String, TaskDetailInfo> taskList = response.getTasks()

IProfile profile = new DefaultProfile
{
    Version="1",
    AccessKeyId=ACCESS_KEY_ID,
    SecretKey=SECRET_KEY
};
var client = new DefaultProductAIClient(profile);

var request = new SearchTaskRequest
{
    StartDate=null,
    EndDate=DateTime.Now.AddDays(1).Date
};

var response = client.GetResponse(request);

curl -X GET \
  -H 'x-ca-version: 1.0' \
  -H 'x-ca-accesskeyid: ACCESS_KEY_ID' \
  https://api.productai.com/batch/_1000001/tasks?start=2017-12-01&end=2017-12-31

Request

Argument	Type	Description	Required	Limitation
start	DateTime	Query with task accepted time (which is the time when prepare function is invoked), query start time. By default, it will return from the first record if this field is not provided	No	Time format %Y-%m-%dT%H:%M:%SZ
end	DateTime	Query with task accepted time (which is the time when prepare function is invoked), query end time. By default it will return records until the last one if this field is not provided	No	Time format %Y-%m-%dT%H:%M:%SZ

Response

{
  "data": {
    "5b6569dc-fe8c-40b3-b96b-7ae7e851fdca": {
      ...
    },
    "6b166b74-fd00-4079-b4ee-9fd54261713b": {
      ...
    }
  },
  "is_error": 0,
  "request_id": "693509c0-e88e-11e7-a6de-ee6487ba2bab"
}

Field	Type	Description
data	Dictionary	Contains all the tasks, key is task ID, value is task details, format please refer to Query task - data fields
is_error	Integer	success is 0; failure is 1
request_id	String	The unique ID of this api call, it can be used in issue debugging with ProductAI support team

Error Code

HTTP status code is used by SDK/API to notify client whether current api call is success or fail

HTTP Status Code	error_code	Description
200	N/a	Success

productAI

Supported Services

Request

Response

Error Code

Create Task

Request

Response

Return Http Status Code 200, When Success

Return Http Status Code Non-200, When Failed

Error Code

Start Task

Request

Response

Error Code

Query Task

Request

Response

Error Code

Cancel Task

Request

Response

Error Code

List Tasks

Request

Response

Error Code