FMEmbedding

[kuzma-long]

您好，关于特征embedding，我想知道FeatureType中的'token'、'float'、'token_seq'是分别如何做embedding的，如果是评论数据是要分词吗？

[Sherry-XLL]

@kuzma-long 您好！

RecBole 支持四种数据格式，'token'、'token_seq'、'float'、'float_seq'。

feat_type	Explanations	Examples
token	single discrete feature	user_id, age
token_seq	discrete features sequence	review
float	single continuous feature	rating, timestamp
float_seq	continuous feature sequence	vector

一个 token 可以直接作为一个 ID 用于特征 embedding，float 是浮点式的具体数据，一般直接用于数值的相关使用，无需 embedding；而 float_seq 通常用于加载预训练好的 embedding 向量。

由于 token_seq 类的数据本身就是以分隔符分割的 token 序列，无论是 token 还是 token_seq，RecBole 都会将字段映射为连续的 ID，可以通过 field2id_token 和 field2token_id 进行 ID 的相互转换：

field2id_token (dict): Dict mapping feature name (str) to a :class:`np.ndarray`, which stores the original token
    of this feature. For example, if ``test`` is token-like feature, ``token_a`` is remapped to 1, ``token_b``
    is remapped to 2. Then ``field2id_token['test'] = ['[PAD]', 'token_a', 'token_b']``. (Note that 0 is
    always PADDING for token-like features.)

field2token_id (dict): Dict mapping feature name (str) to a dict, which stores the token remap table
    of this feature. For example, if ``test`` is token-like feature, ``token_a`` is remapped to 1, ``token_b``
    is remapped to 2. Then ``field2token_id['test'] = {'[PAD]': 0, 'token_a': 1, 'token_b': 2}``.
    (Note that 0 is always PADDING for token-like features.)

也就是说，若为 token_seq 类的评论数据，RecBole 会将评论序列中的所有 token 都映射为统一的内部 ID，然后通过 rnn_utils.pad_sequence 填充到统一的长度。

以 ml-100k.item 为例，电影的 item_id 和发行年份 release_year 为 token 数据，电影名称 movie_title 和电影类型 class 为 token_seq 数据，原始的原子文件为：

RecBole/dataset/ml-100k/ml-100k.item

Lines 1 to 6 in 896155c

	item_id:token movie_title:token_seq release_year:token class:token_seq
	1 Toy Story 1995 Animation Children's Comedy
	2 GoldenEye 1995 Action Adventure Thriller
	3 Four Rooms 1995 Thriller
	4 Get Shorty 1995 Action Comedy Drama
	5 Copycat 1995 Crime Drama Thriller

经过内部映射转换为 Interaction 数据格式之后，每一列特征都变为了 torch.int64 类型的 Tensor 序列：

The batch_size of interaction: 1683
    item_id, torch.Size([1683]), cpu, torch.int64
    movie_title, torch.Size([1683, 15]), cpu, torch.int64
    release_year, torch.Size([1683]), cpu, torch.int64
    class, torch.Size([1683, 6]), cpu, torch.int64

此时映射后的数据变为了：

item_id:token	movie_title:token_seq	release_year:token	genre:token_seq
25	[1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	1	[1, 2, 3, 0, 0, 0]
148	[3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	1	[4, 5, 6, 0, 0, 0]
234	[4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	1	[6, 0, 0, 0, 0, 0]
48	[6, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	1	[4, 3, 7, 0, 0, 0]
76	[8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]	1	[8, 7, 6, 0, 0, 0]

经过映射的预处理之后，原来的 token_seq 数据就变为了相同长度的 ID 序列，仍然对每个 ID 用一个 embedding 进行表示即可。

[kuzma-long]

感谢回复，解释的很详细，还想问一下token_seq对应的ID序列，每个ID做embedding之后，是做了平均池化吗还是concatenate？

[Sherry-XLL]

@kuzma-long 您好，如果是 context-aware 类型的模型的话，聚合 embedding 的方式可以自行选择池化的模式，比如最大池化 max、加和池化 sum 和平均池化 mean：

RecBole/recbole/model/abstract_recommender.py

Lines 302 to 339 in 896155c

	def embed_token_seq_fields(self, token_seq_fields, mode='mean'):
	"""Embed the token feature columns
	Args:
	token_seq_fields (torch.LongTensor): The input tensor. shape of [batch_size, seq_len]
	mode (str): How to aggregate the embedding of feature in this field. default=mean
	Returns:
	torch.FloatTensor: The result embedding tensor of token sequence columns.
	"""
	# input is a list of Tensor shape of [batch_size, seq_len]
	fields_result = []
	for i, token_seq_field in enumerate(token_seq_fields):
	embedding_table = self.token_seq_embedding_table[i]
	mask = token_seq_field != 0 # [batch_size, seq_len]
	mask = mask.float()
	value_cnt = torch.sum(mask, dim=1, keepdim=True) # [batch_size, 1]
	token_seq_embedding = embedding_table(token_seq_field) # [batch_size, seq_len, embed_dim]
	mask = mask.unsqueeze(2).expand_as(token_seq_embedding) # [batch_size, seq_len, embed_dim]
	if mode == 'max':
	masked_token_seq_embedding = token_seq_embedding - (1 - mask) * 1e9 # [batch_size, seq_len, embed_dim]
	result = torch.max(masked_token_seq_embedding, dim=1, keepdim=True) # [batch_size, 1, embed_dim]
	elif mode == 'sum':
	masked_token_seq_embedding = token_seq_embedding * mask.float()
	result = torch.sum(masked_token_seq_embedding, dim=1, keepdim=True) # [batch_size, 1, embed_dim]
	else:
	masked_token_seq_embedding = token_seq_embedding * mask.float()
	result = torch.sum(masked_token_seq_embedding, dim=1) # [batch_size, embed_dim]
	eps = torch.FloatTensor([1e-8]).to(self.device)
	result = torch.div(result, value_cnt + eps) # [batch_size, embed_dim]
	result = result.unsqueeze(1) # [batch_size, 1, embed_dim]
	fields_result.append(result)
	if len(fields_result) == 0:
	return None
	else:
	return torch.cat(fields_result, dim=1) # [batch_size, num_token_seq_field, embed_dim]