easy_vision.python.dataset_tools

easy_vision.python.dataset_tools.custom_generator_example

class easy_vision.python.dataset_tools.custom_generator_example.CustomGenerator(data_config, start_cls_id=0)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

classmethod create_class(name)

easy_vision.python.dataset_tools.data_reader

class easy_vision.python.dataset_tools.data_reader.DataReader(oss_config=None)

Bases: object

a wrapper class that could read data from local path, http url(with retry in case of failure) and oss path.

HTTP_MAX_NUM_IMG_READ_TRY = 10

__init__(oss_config=None)

Parameters:

• error_callback – a callback func, accept two params, first one is error url, second one is error message
• oss_config – string, path to .osscredentials if need to read images from oss

read(file_path)

depends on file_path format, choose one of the reading methods

Parameters:file_path – path to the file

Return: numpy array, [HxWxC], dtype=uint8

easy_vision.python.dataset_tools.dataset_info

class easy_vision.python.dataset_tools.dataset_info.ClassificationDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.DatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

add_global(global_info)

add global info for a dataset

class easy_vision.python.dataset_tools.dataset_info.DataInfoFieldType

Bases: object

CONST = 'const'

COUNT = 'count'

FILE = 'file'

STATISTIC = 'statistic'

class easy_vision.python.dataset_tools.dataset_info.DataInfoFields

Bases: object

aspect_ratio = 'aspect_ratio'

bbox_area = 'bbox_area'

bbox_aspect_ratio = 'bbox_aspect_ratio'

char_dict_path = 'char_dict_path'

label_map_path = 'label_map_path'

num_classes = 'num_classes'

num_images = 'num_images'

num_keypoints = 'num_keypoints'

class easy_vision.python.dataset_tools.dataset_info.DatasetInfo(fields_info=None)

Bases: object

__init__(fields_info=None)

x.__init__(…) initializes x; see help(type(x)) for signature

add_const_field(name, value)

add one dataset const field info, like num_classes etc.

add_count_field(name)

add one dataset count field info, like num_images etc.

add_file_field(name, value)

add one dataset file field info, like label_map_path etc.

add_global(global_info)

add global info for a dataset

add_single(single_info)

add single image and label info into dataset, single_info keys is same as tfrecord keys.

add_statistic_field(name, value)

add one dataset statistic field info, like aspect_ratio etc.

dump(info_path)

dump info to json file

get(check_file=False)

get field info dict

load(info_path_pattern)

load dataset info from json files :param info_path_pattern: str or list, glob info file pattern

update(fields_info)

use another dataset fields_info update this dataset info

update_const_field(name, field_info)

update one dataset const field info, like num_classes etc.

update_count_field(name, field_info)

add one dataset count field info, like num_images etc.

update_file_field(name, field_info)

update one dataset file field info, like label_map_path etc.

update_statistic_field(name, field_info)

update one dataset statistic field info, like aspect_ratio etc.

class easy_vision.python.dataset_tools.dataset_info.DetectionDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.ClassificationDatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

add_single(single_info)

add single image and label info into dataset, single_image_info keys is same as tfrecord keys.

class easy_vision.python.dataset_tools.dataset_info.SegmentationDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.ClassificationDatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

class easy_vision.python.dataset_tools.dataset_info.TextDetectionDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.DetectionDatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

add_global(global_info)

add global info for a dataset

class easy_vision.python.dataset_tools.dataset_info.TextEnd2EndDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.DetectionDatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

add_global(global_info)

add global info for a dataset

class easy_vision.python.dataset_tools.dataset_info.TextRecognitionDatasetInfo

Bases: easy_vision.python.dataset_tools.dataset_info.DatasetInfo

__init__()

x.__init__(…) initializes x; see help(type(x)) for signature

add_global(global_info)

add global info for a dataset

easy_vision.python.dataset_tools.pai_converter

class easy_vision.python.dataset_tools.pai_converter.PaiConverter(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_converter.MultiProcConverter

__init__(data_config)

Parameters:data_config – DataConfig instance

classmethod create_class(name)

split_train_test(input_path, output_dir, test_ratio)

split into train test file with test_ratio

Parameters:

• input_path – string, file to split
• output_dir – output directory to save splited files
• test_ratio – number of test data / number of total data

Returns:

train data file path test_input_path: test data file path

Return type:

train_input_path

class easy_vision.python.dataset_tools.pai_converter.PaiRow(row, task_id=None)

Bases: easy_vision.python.dataset_tools.tf_record_converter.DataRow

__init__(row, task_id=None)

get_anws()

get anws string from anws field

Returns:anws, json object, depends on task

get_img_url()

get img url from row

Returns:url, string, path of the image

static get_task_ids(row)

get task ids in the data-row

Parameters:row – json dict Returns:task ids in the data row

iter_objs()

iter_polygons()

easy_vision.python.dataset_tools.qince_converter

class easy_vision.python.dataset_tools.qince_converter.QinceConverter(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_converter.MultiProcConverter

implement methods for qince format # check input file has any error _check_lines(self, file_path) # load from easy_vision.python.input_path, wrapped with QinceRow, push into self._input_queue _load_input(self, input_path) # save error data in self._remark_map to file_name for correction. _save_for_fix(self, file_name)

CREATE_TASK_URL = 'https://qince.taobao.com/label/createCutout.htm?template_type=103&device=2'

__init__(data_config)

Parameters:data_config – DataConfig instance

classmethod create_class(name)

split_train_test(input_path, output_dir, test_ratio)

split into train test file with test_ratio

Parameters:

• input_path – string, file to split
• output_dir – output directory to save splited files
• test_ratio – number of test data / number of total data

Returns:

train data file path test_input_path: test data file path

Return type:

train_input_path

class easy_vision.python.dataset_tools.qince_converter.QinceRow(row, separator=None)

Bases: easy_vision.python.dataset_tools.tf_record_converter.DataRow

parse an qince row to get image url and anwser 0.数据ID 1.原始数据 2.答案 3.打标人 4.打标时间 5.题目类型 6.是否质检 7.质检人 The following methods of DataRow are reimplemented:

iter_objs(self) get_cls(self)

ANWS = 2

CHECK = 6

CHECK_PERSON = 7

DRECTION_MAP = {u'\u5e95\u90e8\u671d\u4e0a': 2, u'\u5e95\u90e8\u671d\u4e0b': 0, u'\u5e95\u90e8\u671d\u53f3': 3, u'\u5e95\u90e8\u671d\u5de6': 1}

ID = 0

PERSON = 3

QUESTION_TYPE = 5

RAW_DATA = 1

TIME = 4

URL_PATTERN = '^(https?|oss|file)://[-A-Za-z0-9+&@#/%?=~_|!:,.;]+[-A-Za-z0-9+&@#/%=~_|]$'

__init__(row, separator=None)

Parameters:row – generate by csv.reader, array of columns in csv file

get_anws()

get anws string from anws field

Returns:anws, json object, depends on task

get_cls()

get_img_url()

get img url from raw_data

Returns:url, string, path of the image

get_mask_url()

get_text()

iter_objs()

iter_text_det_objs()

iter_text_e2e_objs()

easy_vision.python.dataset_tools.tf_record_converter

class easy_vision.python.dataset_tools.tf_record_converter.DataRow(row)

Bases: object

Base class for extract tf record information from input. One subclass is QinceRow. all subclass must implement:

get_img_url(self) get_anws(self)

for classification tasks, one more method:

get_cls(self)

for detection tasks, one more method:

iter_objs(self)

for other tasks, to be extend.

__init__(row)

Parameters:a tuple of str, input line split in Converter._load_input (row) –

get_anws()

must be implemented for all tasks if data correction is necessary. :returns:

any type data if you want to use easy-vision generators , an json object

should be returned

get_cls()

must be implemented for CLASSIFICATION task :returns: a list of string, class_name, or class_name + separator + class_description

get_img_url()

must be implemented for all tasks

get_mask_url()

must be implemented for SEGMENTATION task :returns: string, mask url

get_text()

must be implemented for TEXT_RECOGNITION task when use text_recognition style data :returns: string, text content

get_url_data(url)

get binary data read from url :param url: data url

Returns:binary data of url

iter_objs()

must be implemented for DETECTION task :returns: an iterator, enumerate it to get class_name, bounding box

label could be class_name or class_name + separator + class_description boundingbox is an array of numbers [left_x, top_y, right_x, bottom_y]

iter_objs_with_texts()

must be implemented for TEXT_END2END task, or TEXT_RECOGNITION task when use text_end2end style data :returns: an iterator, enumerate it to get class_name, corners, text_content,

direction_id, difficult

iter_polygons()

must be implemented for POLYGON_SEGMENTATION task :returns: an iterator, enumerate it to get class_name, polygon

polygon is an array of numbers [x, y]

set_url_data(url, data)

set binary data read from url :param url: data url :param data: binary data of url

class easy_vision.python.dataset_tools.tf_record_converter.DictReduceType

Bases: enum.Enum

ADD = 0

UPDATE = 1

class easy_vision.python.dataset_tools.tf_record_converter.MultiProcConverter(data_config)

Bases: object

Base class for parallel converter for all kinds of input format:

such as qince format.

subclass must implement the following methods:

_check_lines(self, input_path) _load_input(self, input_path) _save_for_fix(self, file_name), optional if you want to save

error labeled data for fix.

split_train_test(self, input_path, output_dir, test_ratio)

__init__(data_config)

init a parallel convert object :param data_config: a DataConfig object file, specifies params

for tf record generation

Parameters:proc_num – number of process to parallel convert

classmethod create_class(name)

parallel_convert(input_path, output_prefix, output_label_path, output_chardict_path)

create object detection tf record using qince marked data :param input_path: .csv format, exported from qince :param output_prefix: output path to save tfrecord :param output_label_path: output path to save label_map, a

protos.string_int_label_map_pb2 text format file

Parameters:output_chardict_path – output path to save char_dict

Returns: None :raises: ValueError, if input_path does not exist

reduce_dicts(dicts, output_dir, dict_name, op=<DictReduceType.UPDATE: 1>)

gather dicts from all hosts

Parameters:

• dicts – dict type data
• output_dir – save_dir, string
• dict_name – dict name, an unique tag for this type of dict
• op – default to update, if ‘ADD’, add elementwise

Returns:

dict gathered from all hosts

split_train_test(input_path, output_dir, test_ratio)

split into train test file with test_ratio :param input_path: string, file to split :param output_dir: output directory to save splited files :param test_ratio: number of test data / number of total data

Returns:train data file path test_input_path: test data file path Return type:train_input_path

split_train_test_multi_worker(input_path, output_dir, test_ratio=0.1)

easy_vision.python.dataset_tools.tf_record_generator

class easy_vision.python.dataset_tools.tf_record_generator.ClassificationGenerator(data_config, start_cls_id=0)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=0)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.DetectionGenerator(data_config, start_cls_id=1)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=1)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.SegmentationGenerator(data_config, start_cls_id=0, use_polygon=False)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=0, use_polygon=False)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator(data_config, start_cls_id=0)

Bases: object

Base converter class for all tasks: classification, detection and so on. sub class must implement:

_to_tf_example(self, img, anws, url)

ERROR_CLASS = -2

IGNORE_CLASS = -3

INVALID_CLASS = -1

__init__(data_config, start_cls_id=0)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

generate(row)

convert data row to serialized tf.Example

Parameters:row – instance of DataRow subclass Returns:a list of string, a list of serialized tf.Example

init_resource()

init resource used by generator

save_char_dict(save_path)

save_label_proto(save_path)

set_dataset(dataset_name)

set_max_image_size(max_image_size)

class easy_vision.python.dataset_tools.tf_record_generator.TextDetectionGenerator(data_config, start_cls_id=1)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=1)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.TextEnd2EndGenerator(data_config, start_cls_id=1)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=1)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.TextRecognitionGenerator(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.TextRectificationGenerator(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

class easy_vision.python.dataset_tools.tf_record_generator.VideoClassificationGenerator(data_config, start_cls_id=0)

Bases: easy_vision.python.dataset_tools.tf_record_generator.TFRecordGenerator

__init__(data_config, start_cls_id=0)

init from data_config

Parameters:

• DataConfig instance (data_config,) –
• encode classes from start_cls_id, for classification (start_cls_id,) – tasks, usually start from 0, for detection tasks, usually start from 1

classmethod create_class(name)

generate(row)

convert data row to serialized tf.Example

Parameters:row – instance of DataRow subclass Returns:a list of string, a list of serialized tf.Example

init_resource()

init resource used by generator

easy_vision.python.dataset_tools.tf_record_generator.build_tfrecord_generator(data_config)

build tfrecord converter from data_config :param config_path: string, path of the qince config file

Returns:instance of subclass of TFRecordConverter

easy_vision.python.dataset_tools.tf_record_writer

class easy_vision.python.dataset_tools.tf_record_writer.SingleWriter(input_que, output_prefix, line_num, gen_proc_num)

__init__(input_que, output_prefix, line_num, gen_proc_num)

join()

num_good_sam

write()

class easy_vision.python.dataset_tools.tf_record_writer.SplitWriter(input_que, output_prefix, line_num, gen_proc_num, record_size=64, thread_num=1)

Split the record into multiple parts, each contain record_size samples.

__init__(input_que, output_prefix, line_num, gen_proc_num, record_size=64, thread_num=1)

any_fail()

join()

num_good_sam

write()

if any writer failed, return False

class easy_vision.python.dataset_tools.tf_record_writer.SplitWriterThread(thread_id)

__init__(thread_id)

end()

end by write None into the queue

fail()

isopen()

join()

start()

write(data)

easy_vision.python.dataset_tools.video_converter

class easy_vision.python.dataset_tools.video_converter.VideoClassificationConverter(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_converter.MultiProcConverter

implement methods for converting a text line file into tfrecord required by image segmentation models.

__init__(data_config)

Parameters:data_config – DataConfig instance

classmethod create_class(name)

split_train_test(input_path, output_dir, test_ratio=0.1)

class easy_vision.python.dataset_tools.video_converter.VideoClassificationRow(row)

Bases: easy_vision.python.dataset_tools.tf_record_converter.DataRow

parse an image_path mask_path line to get image url and anwser The following methods of DataRow are reimplemented:

get_mask

__init__(row)

Parameters:row – generate by csv.reader, array of columns in csv file

get_anws()

get anws string from anws field

Returns:anws, json object, depends on task

get_cls()

get class info from answer

Returns:classes, list object, a list of classes

get_img_url()

get img url from raw_data

Returns:url, string, path of the image

easy_vision.python.dataset_tools.voc_converter

class easy_vision.python.dataset_tools.voc_converter.VocConverter(data_config)

Bases: easy_vision.python.dataset_tools.tf_record_converter.MultiProcConverter

implement methods for voc format

__init__(data_config)

Parameters:data_config – DataConfig instance

classmethod create_class(name)

split_train_test(input_path, output_dir, test_ratio=0.1)

class easy_vision.python.dataset_tools.voc_converter.VocDataRow(row)

Bases: easy_vision.python.dataset_tools.tf_record_converter.DataRow

parse image name to get image url and anwser img_path xml_path

__init__(row)

Parameters:row – string, image_name

get_anws()

get anws string from anws field :returns: string, anwser path name

get_img_url()

get img url from raw_data :returns: url, string, path of the image

iter_objs()

get obj bounding box and label

Returns:string bnd(np.array): dtype=np.float32, size=4 Return type:label